Vitamin D3 for Life

Live Enlightened

Research libraryVitamin D3 For Life
Study design and methods

Secondary relevance to vitamin D This paper mainly addresses another topic and also reports a vitamin D analysis, result or scientific synthesis.

This paper mainly addresses another topic and also reports a vitamin D analysis, result or scientific synthesis.

Why this paper is included

Supporting source evidence:

For example, in one case, vitamin D was listed on admission but not repeated in the discharge medication; the model labeled it as “unchanged,” which some raters considered a hallucination because no corresponding entry was present.

Source: Principal Findings. View original paper

Relevance classification does not establish study quality, causation or editorial approval.

Study design Study design awaiting verification.

Study design awaiting verification

The available source does not establish a sufficiently clear study design.

Meta-analysis unclear · Blinding unclear / not reported · Human research reported

How this paper was classified

Automatic source-based classification; awaiting editorial verification. This is not a quality score.

For each layout item, participants were able to view the original prompt and asked to evaluate whether the LLM-generated summary adhered exactly to the requested formatting and structure (eg, presence of specific section headings, use of bold formatting, bullet lists with a maximum number of items, or mandated section titles such as “medication”).

Source: Methods.

View supporting evidence in the original paper

What the study types mean
Systematic review
Uses explicit methods to identify, assess and summarise research on a defined question.
Scoping review
Maps the extent and characteristics of research on a topic.
Narrative review
Discusses existing research; systematic-review methods have not been established by this classification.
Review / evidence synthesis
Reviews existing research; the exact review methods need further assessment.
Randomised controlled trial
Assigns participants or groups to comparison interventions by chance.
Non-randomised intervention study
Examines an intervention without random allocation.
Cohort study
Examines exposures and outcomes over time within a defined group.
Case-control study
Compares people with an outcome against people without it, examining exposures.
Cross-sectional study
Examines a population at a particular point in time.
Observational study
Observes exposures and outcomes without assigning the exposure; the subtype is not established here.
Case report
Describes an individual case.
Case series
Describes a group of individual cases.
Secondary analysis
Analyses existing study data; this paper is not automatically another independent trial.
Laboratory study
Investigates biological processes in a laboratory setting.
Qualitative study
Examines experiences or meanings using qualitative methods.
Genetic instrumental-variable study
Uses genetic variation as an instrument to investigate relationships; this is not a randomised treatment trial.
Study protocol
Describes planned methods and is not evidence that the planned study has reported results.
Multiple reported designs
More than one design is explicitly reported; examine the supporting methods.
Study design awaiting verification
The available source does not establish a sufficiently clear study design.

Meta-analysis combines numerical results from studies. Blinding describes who was unaware of treatment allocation. Neither is a separate quality grade. Study design, vitamin D relevance and document format are independent filters.

Abstracts

Expert Evaluation and Consensus on GPT-4o Summaries of Clinical Letters: Validation and Results of the Framework and Implementation of AI Tools Project

Study design awaiting verification · JMIR Medical Informatics · 2026

Original publication title

Expert Evaluation and Consensus on GPT-4o Summaries of Clinical Letters: Validation and Results of the Framework and Implementation of AI Tools Project

Mieke Deschepper; Helga Rogge; Kirsten Colpaert

Copyright © Mieke Deschepper, Helga Rogge, Kirsten Colpaert. Originally published in JMIR Medical Informatics (https://medinform.jmir.org)

Original publication · Publisher record · CC BY 4.0

Source wording retained. Formatting, section navigation and display-title punctuation adapted for this research record.

Authors’ conclusion

Our framework offers a robust approach for evaluating LLM-generated discharge summaries, balancing usability and clinical relevance. Semantic integrity, especially regarding medication details, was identified as a key vulnerability. Perceived overall quality was driven by a positive response rate for content. High ICC values for global score, with lower item-level agreement lead toward the need for clearer, context-specific prompts and standardized evaluation criteria to reduce interrater variability. Human oversight and targeted automated checks for omissions and hallucinations are essential for safe clinical deployment.

An extract from the source publication. Read it in context.

At a glance

Abstract: What objectives does the paper state?

This study aimed to quantify expert agreement and interrater reliability on LLM summaries of discharge letters, identify frequent and clinically relevant errors, and evaluate practical implications for integrating LLMs into documentation workflows.

Source: Objective

Abstract: What methods does the paper describe?

Thirty expert-curated synthetic Dutch discharge letters were summarized. Thirty-one clinicians from Flemish care settings (1 university hospital, 2 private hospitals, and 2 general practice circles) evaluated the summaries. The evaluation framework consisted of 61 binary layout items assessing whether required sections and formatting were correctly present, 33 content items (correct or complete vs incorrect, subcategorizing missing, irrelevant, and hallucinated information), a 4-point global quality rating, and an open comment. Statistical analyses included descriptive statistics, mixed effects ordinal regression on the global score, consensus (agreement per question or letter) percentages, interrater reliability (Cohen κ, intraclass correlation coefficient [ICC], Fleiss κ, and prevalence index), and thematic synthesis of comments.

Source: Methods

Abstract: Results

Layout adherence was high (88%), especially in the conclusion section. The positive response rate for content was overall moderate (78%), with the best performance observed in the medical history section and the lowest performance observed in the medication section, which also showed the highest rate of hallucinations and the weakest interrater consensus. Across all sections, missing information was the most common error. Nearly 70% of global ratings were “good” or “very good.” Higher positive response rates for content predicted better global scores (β=.079; P<.001), while layout and participant specialty were not relevant to global scoring.

Consensus was high for the layout questions (median 96.8%, IQR 90.2%-100%) and somewhat lower for content (median 83.9%, IQR 67.7-96.8), with the lowest agreement in the medication section. Interrater agreement was moderate (median Cohen κ=0.36, IQR 0.29-0.43; range 0.07‐0.56), but overall reliability was high (ICC 0.945, 95% CI 0.942-0.948), indicating strong consistency at the global level despite interrater variability. The prevalence index demonstrated that high ICC values were partly driven by the strong prevalence of affirmative responses in layout items, while content items showed more balanced distributions and lower agreement.

Source: Results

Abstract: Conclusions

Our framework offers a robust approach for evaluating LLM-generated discharge summaries, balancing usability and clinical relevance. Semantic integrity, especially regarding medication details, was identified as a key vulnerability. Perceived overall quality was driven by a positive response rate for content. High ICC values for global score, with lower item-level agreement lead toward the need for clearer, context-specific prompts and standardized evaluation criteria to reduce interrater variability. Human oversight and targeted automated checks for omissions and hallucinations are essential for safe clinical deployment.

Source: Conclusions
Authors’ abstract

Abstract

Background

Large language models (LLMs) are increasingly used to summarize clinical documents; yet, automated metrics often inadequately capture clinical relevance and safety. In the initial phase of the “Framework and Implementation of AI Tools,” an expert-driven, cocreated evaluation methodology was established to assess LLM-generated discharge letter summaries, combining prompt design considerations with intuitive expert appraisal.

Objective

This study aimed to quantify expert agreement and interrater reliability on LLM summaries of discharge letters, identify frequent and clinically relevant errors, and evaluate practical implications for integrating LLMs into documentation workflows.

Methods

Thirty expert-curated synthetic Dutch discharge letters were summarized. Thirty-one clinicians from Flemish care settings (1 university hospital, 2 private hospitals, and 2 general practice circles) evaluated the summaries. The evaluation framework consisted of 61 binary layout items assessing whether required sections and formatting were correctly present, 33 content items (correct or complete vs incorrect, subcategorizing missing, irrelevant, and hallucinated information), a 4-point global quality rating, and an open comment. Statistical analyses included descriptive statistics, mixed effects ordinal regression on the global score, consensus (agreement per question or letter) percentages, interrater reliability (Cohen κ, intraclass correlation coefficient [ICC], Fleiss κ, and prevalence index), and thematic synthesis of comments.

Results

Layout adherence was high (88%), especially in the conclusion section. The positive response rate for content was overall moderate (78%), with the best performance observed in the medical history section and the lowest performance observed in the medication section, which also showed the highest rate of hallucinations and the weakest interrater consensus. Across all sections, missing information was the most common error. Nearly 70% of global ratings were “good” or “very good.” Higher positive response rates for content predicted better global scores (β=.079; P<.001), while layout and participant specialty were not relevant to global scoring.

Consensus was high for the layout questions (median 96.8%, IQR 90.2%-100%) and somewhat lower for content (median 83.9%, IQR 67.7-96.8), with the lowest agreement in the medication section. Interrater agreement was moderate (median Cohen κ=0.36, IQR 0.29-0.43; range 0.07‐0.56), but overall reliability was high (ICC 0.945, 95% CI 0.942-0.948), indicating strong consistency at the global level despite interrater variability. The prevalence index demonstrated that high ICC values were partly driven by the strong prevalence of affirmative responses in layout items, while content items showed more balanced distributions and lower agreement.

Conclusions

Our framework offers a robust approach for evaluating LLM-generated discharge summaries, balancing usability and clinical relevance. Semantic integrity, especially regarding medication details, was identified as a key vulnerability. Perceived overall quality was driven by a positive response rate for content. High ICC values for global score, with lower item-level agreement lead toward the need for clearer, context-specific prompts and standardized evaluation criteria to reduce interrater variability. Human oversight and targeted automated checks for omissions and hallucinations are essential for safe clinical deployment.

Publication and source details

Original title: Expert Evaluation and Consensus on GPT-4o Summaries of Clinical Letters: Validation and Results of the Framework and Implementation of AI Tools Project

Mieke Deschepper; Helga Rogge; Kirsten Colpaert

JMIR Medical Informatics · 2026

Mieke Deschepper, MSc, PhD, Data Science Institute, Ghent University Hospital, Corneel Heymanslaan 10, Ghent, Flanders, 9000, Belgium, +32 9 3321814; mieke.deschepper@uzgent.beCopyright © Mieke Deschepper, Helga Rogge, Kirsten Colpaert. Originally published in JMIR Medical Informatics (https://medinform.jmir.org)2026https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.

Repository JATS snapshot; version as supplied

Source retrieved: 2026-11-10T23:13:06.599668+00:00

DOI: 10.2196/90374 · PMC13160486