Hyeji Jun, Yutaro Tanaka, Shreya Johri, Filipe LF Carvalho, Alexander C. Jordan, Chris Labaki, Matthew Nagy, Tess A. O’Meara, Theodora Pappa, Erica Maria Pimenta, Eddy Saad, David D Yang, Riaz Gillani, Alok K. Tewari, Brendan Reardon, Eliezer Van Allen
doi: https://doi.org/10.1101/2025.05.09.25327312
Abstract
The rapid expansion of molecularly informed therapies in oncology, coupled with evolving regulatory FDA approvals, poses a challenge for oncologists seeking to integrate precision cancer medicine into patient care. Large Language Models (LLMs) have demonstrated potential for clinical applications, but their reliance on general knowledge limits their ability to provide up-to-date and niche treatment recommendations.
To address this challenge, we developed a RAG-LLM workflow augmented with Molecular Oncology Almanac (MOAlmanac), a curated precision oncology knowledge resource, and evaluated this approach relative to alternative frameworks (i.e. LLM-only) in making biomarker-driven treatment recommendations using both unstructured and structured data. We evaluated performance across 234 therapy-biomarker relationships. Finally, we assessed real-world applicability of the workflow by testing it on actual queries from practicing oncologists.
While LLM-only achieved 62–75% accuracy in biomarker-driven treatment recommendations, RAG-LLM achieved 79–91% accuracy with an unstructured database and 94–95% accuracy with a structured database. In addition to accuracy, structured context augmentation significantly increased precision (49% to 80%) and F1-score (57% to 84%) compared to unstructured data augmentation. In queries provided by practicing oncologists, RAG-LLM achieved 81–90% accuracy.
These findings demonstrate that the RAG-LLM framework effectively delivers precise and reliable FDA-approved precision oncology therapy recommendations grounded in individualized clinical data, and highlight the importance of integrating a well-curated, structured knowledge base in this process. While our RAG-LLM approach significantly improved accuracy compared to standard LLMs, further efforts will enhance the generation of reliable responses for ambiguous or unsupported clinical scenarios.
Competing Interest Statement
RG has equity in Google, Microsoft, Amazon, Apple, Moderna, Pfizer, and Vertex Pharmaceuticals; his spouse is employed by Carrum Health. ES receives research funding from Genentech/imCORE and Oncohost. CL receives research funding from Genentech/imCORE. EMVA holds consulting roles with Enara Bio, Manifold Bio, Monte Rosa, Novartis Institute for Biomedical Research, Serinus Bio, and TracerBio; he previously held consulting roles with Tango Therapeutics, Invitae, Syapse, Janssen, Genome Medical, Genomic Life, and Riva Therapeutics; he receives research support from Novartis, Bristol-Myers Squibb, Sanofi, and NextPoint; he has equity in Tango Therapeutics, Genome Medical, Genomic Life, Enara Bio, Manifold Bio, Microsoft, Monte Rosa, Riva Therapeutics, Serinus Bio, Syapse, and TracerDx; he received travel reimbursement from Roche and Genentech; he has filed institutional patents on chromatin mutations and immunotherapy response, and methods for clinical interpretation, and provides intermittent legal consulting on patents for Foaley & Hoag. Other authors have no relevant disclosures.
Funding Statement
This work was supported by P50CA272390, DOD W81XWH-21-PCRP-DSA, DOD HT94252410415, and the Mark Foundation Emerging Leader Award.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Copyright
The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC 4.0 International license.
bioRxiv and medRxiv thank the following for their generous financial support:
The Chan Zuckerberg Initiative, Cold Spring Harbor Laboratory, the Sergey Brin Family Foundation, California Institute of Technology, Centre National de la Recherche Scientifique, Fred Hutchinson Cancer Center, Imperial College London, Massachusetts Institute of Technology, Stanford University, The University of Edinburgh, University of Washington, and Vrije Universiteit Amsterdam.