Q&A: Multimodal foundation models in action: Applications across drug development

Raluca Verona, PhD: The biggest opportunities are moving beyond traditional, hypothesis-driven biomarker development and embracing reverse translation. Foundation models allow us to incorporate complex patterns from data like pathology images without a pre-specified hypothesis, leading to a deeper understanding of response and resistance. Secondly, by connecting multimodal data with clinical outcomes, we can generate new insights into disease biology. This creates a continuous learning loop that can lead to better targets and more effective novel biomarkers.

Amit Aggarwal, PhD: What excites me most is the ability to move clinical development from an empirical approach to a more predictive, mechanism-driven one. We can use multimodal AI to integrate spatial features, omics, and clinical context to create composite signatures. This helps us identify efficacy signals earlier, differentiate therapies more clearly, and generate better hypotheses for rational combinations, especially where single biomarkers are insufficient.

Raluca Verona, PhD: The most important thing is to start with the clinical or biological question you are trying to answer, not with the technology. Once you identify the need, you can build the right approach. A major challenge is evidence generation. A model developed on a large dataset must be validated on smaller, independent clinical cohorts. Planning for this validation early in the clinical development process is critical to ensure you are following a true clinical signal.

Amit Aggarwal, PhD: From a technical standpoint, the hurdle isn’t compute power, but cutting through the hype to build something clinically valuable. We face friction points like data completeness, where modalities may be missing or collected at different times. Pre-trained models can also overfit on small clinical trial cohorts, so planning your training and test cases is essential. Given the “black box” nature of some models, ensuring the signals are biologically meaningful is another key challenge. Success requires a subtractive approach: use the leanest modalities paired with the right model, grounded in the biological mechanism of interest.

DH Lee: Simplicity often wins, but the choice depends on the goal. Key factors include the complexity of the tumor biology you need to capture, the data available at both development and deployment, and the stage of the drug program. If a single modality is strongly associated with the biology and performs well, that is the best path. However, for complex biology, we recommend exploring multimodal models, especially in early-stage development. A valid concern is deployment. If only an H&E image is available in the clinic, a multimodal signature is not practical. In these cases, we can use a strategy to distill the complex multimodal signature into a simpler, single-modality signature that approximates the performance for deployment.

Amit Aggarwal, PhD: The mechanism of ADCs is a multi-step process, but patient selection has often relied only on target expression via IHC, which captures just one part of the biology. A multimodal AI approach is better suited for this complexity. We can use foundation models to extract biological features reflecting the multiple mechanisms of action from different data types. By training these models on rich, real-world data and benchmarking them against matched standard-of-care controls, we can develop a more precise patient selection approach. This moves beyond a single biomarker to incorporate information from different sources, helping to inform patient selection and differentiate assets.

Raluca Verona, PhD: This is a critical question. If the intended use is patient selection, you need to develop a specific assay, establish analytical parameters, and validate it clinically. This entire pipeline must be planned. Often, an algorithm is developed on a small, retrospective cohort. The first step is to test it on independent datasets, which can be a limitation. Beyond the algorithm itself, there is a whole infrastructure to consider, from tissue preparation and scanners to navigating the regulatory landscape. It requires a great deal of early planning, and as a field, we are all learning as we go.

Caitlin McWilliams: I agree completely. To translate these promising technologies into applications with clinical impact requires a broad set of expertise. You need to develop AI applications that are not just interesting but clinically relevant, with a clear intended use. Expertise in validation and navigating regulatory pathways is critical. It is also essential to align the AI development timelines with the clinical development timelines so that the drug and the diagnostic can be brought to market in a synergistic way. Finally, you must have the right deployment channels in place so physicians and patients can actually access the test.

Raluca Verona, PhD: As early as possible. For new programs, I see this being applied from the very beginning. It will likely become a requirement to have a line of sight into how you will implement an AI-driven strategy in the clinic. You need to do the work upfront to identify the right model, the validation needed, and the signal you are pursuing.

Amit Aggarwal, PhD: I would echo that. You have to keep the end in mind. For early-stage portfolios, there is more time to think through open questions and develop models that can be used for forward or reverse translation. This gives you more runway. Trying to add this to a late-stage program may not be conducive to the timeline.