Qingyu Chen, PhD: Our lab does two things that are usually treated as separate: We build medical AI, and we try to study where it fails.

On the building side, we work with two main kinds of information medicine runs on. We develop models that read clinical records and the medical literature, models that analyze medical images to help diagnose disease and predict its course, and systems that combine both, so an AI can weigh a patient’s written history alongside their scans, the way a physician would.

But a model that performs well on a test is not the same as a model you can trust with a patient. So, we also study how these systems fail. They can state falsehoods with complete confidence. They can reach a right answer through faulty reasoning. And, as our recent work shows, they can memorize sensitive information from the data they were trained on.

Our goal is to develop medical AI that is genuinely useful, understand where it breaks down, and produce the evidence needed to know when it can—and cannot—be trusted.

Chen: Because medicine is inherently multimodal. A patient cannot be understood through a single paragraph or a single image. Clinical decisions often require bringing together a patient’s history, laboratory results, medical notes, and imaging findings.

Our work spans both sides of that. On the language side, we develop methods to help computers understand clinical records and biomedical literature. On the imaging side, much of our research focuses on medical images and specialties that depend heavily on them—ophthalmology in particular, where we work on diagnosing eye disease and predicting how it will progress. This is also why I hold a secondary appointment in ophthalmology.

What excites us most is bringing these areas together: systems that can integrate text, images, and other clinical information to form a more complete picture of a patient and better support medical decision making.

Progress here requires more than developing new models. One of the biggest barriers is the limited availability of medical data that is large enough, reliable enough, and free for researchers to share. To help address this, we recently developed MedPMC, a system that has assembled 11 million medical images paired with their accompanying text, drawn from research literature that is openly licensed for reuse—and that is designed to keep growing as new research is published. We’ve made the data, the tools used to assemble it, the tests for measuring performance, and the resulting models all publicly available, so that other institutions can develop, evaluate, reproduce, and adapt these systems rather than starting from scratch.

Chen: Memorization means that a model can recall or reproduce content it encountered during training. If a model has been trained on clinical guidelines, it may reproduce part of a guideline when you give it the opening of that passage. If it has had additional training on a set of medical exam questions, it may produce an answer choice that appeared in that training data—even after we removed that choice from the question.

That is different from simply saying that a model performs well. When a model answers a question correctly, there are several possible explanations. It may have acquired genuine medical knowledge. It may have learned a pattern of reasoning it can apply to problems it hasn’t seen before. Or it may simply recognize the question and reproduce something it previously saw during training. If all we check is whether the final answer is correct, we cannot tell these apart.

So, our goal was to look beyond accuracy and ask a different set of questions: How often does memorization occur? What types of content get memorized? How much can a model reproduce? Does what it memorized earlier survive further training? And what does all of this mean for using these systems in medicine?

Chen: We looked at the main stages a general-purpose model goes through on its way to becoming a medical one.

First, we examined models that had already undergone continued pretraining, in which a general-purpose model is trained further on large collections of medical text, including biomedical literature, clinical guidelines, and clinical notes. Second, we evaluated models that had been fine-tuned on the standard question-and-answer datasets that the field uses to test medical models. Third, we conducted a privacy-protected, HIPAA-compliant study using more than 13,000 medical records to fine-tune models for disease diagnosis. This was done in an isolated and secure computing environment. These records had already been collected in the course of care; the study did not recruit patients or change anyone’s treatment.

Across those settings, we examined both general-purpose models and models trained on medical data, 10 different datasets containing hundreds of thousands of records, and thousands of model responses that we manually reviewed.

The patterns differed depending on the training stage. Continued pretraining was more likely to produce long, word-for-word matches to source documents. Fine-tuning produced less long-form copying in some settings, but more memorization tied to the specific task the model was trained on. For example, after fine-tuning on medical question-and-answer datasets, models reproduced roughly 14% to 21% of the answer choices that had been removed from the question the model was shown.

We also found that memorization was persistent. Depending on the setting, as much as 87% of what a model memorized during continued pretraining was still present after it had been fine-tuned on a new medical task. Fine-tuning does not necessarily erase what a model previously memorized. It may preserve that content while adding new memorization specific to the task it was just trained on.

Chen: The clinical case study showed both the potential benefit and the risk of adapting these models to real-world medical data. Fine-tuning improved diagnostic performance; for one model, the correct diagnosis came up as its first choice 54.8% of the time, up from 48.6%. In some specialties the gains were larger than that—more than 10 percentage points in areas such as cardiology and nephrology, which deal with heart and kidney conditions.

Chen: No. One of the central findings of our study is that memorization in medicine is not a single behavior.

We identified three broad types. The first is beneficial memorization. A model may accurately retain biomedical concepts, clinical guidelines, the medical literature it has read, or specific medical knowledge tied to its task. That kind of memory may support factual accuracy and help the model perform medical tasks more effectively.

The second is uninformative memorization. Models sometimes reproduce document disclaimers, section headings, formatting instructions, or other boilerplate language. This adds little medical value and may indicate that the model is learning surface-level patterns rather than deeper medical understanding.

The third is harmful memorization. This includes reproducing quirks specific to a particular dataset, word-for-word passages from patient notes, protected health information, or other sensitive patient content. This form of memorization may create privacy risks and may also indicate that the model is leaning too heavily on its training data rather than generalizing to new cases.

The important question, then, is not simply whether a model memorizes; it is what the model memorizes, why it memorizes it, and whether that memory supports or undermines the medical use it is intended for.

Chen: One thing that stood out: memorization did not only show up late, after a model had been trained too long. It began early.

As we trained the models, we tracked their progress and compared three things: how much they were memorizing, how well they were learning by the usual measure, and how accurate their diagnoses were. Memorization began increasing relatively early, even while the standard measures still showed the model improving, and before its diagnoses had reached their peak accuracy.

That means traditional indicators researchers watch during training—such as whether the model keeps improving on held-out data, or the point at which they would normally stop training—are useful but may not be sufficient on their own. A model can appear to be learning effectively according to standard performance measures while simultaneously increasing its retention of training-specific content.

We also found two other patterns. Larger models and longer inputs were generally associated with more memorization. By contrast, changing common generation settings such as temperature—which controls how varied the model’s answers are—had relatively limited effects. This suggests that memorization is fundamentally connected to how a model is trained and what data it is exposed to, rather than being something that can simply be solved by adjusting how the model generates its answers after training.

Chen: Adapting an AI model to medicine does not simply make it “more medical.” It changes what the model knows, what it remembers, and what it may reproduce.

Some of that memory is valuable. We want models to retain accurate medical knowledge and clinical guidance. But we do not want them to rely on meaningless repetition, echo back the answers to test questions, or expose sensitive information from patient records.

Trustworthy medical AI therefore requires more than measuring whether a model gets the answer right. We need to understand how it got there, what it retained from training, and whether it will stay safe and reliable when used in a new setting.