Celebrating One Year of MedARC

Key Takeaways:

• For the past year, MedARC has been at the forefront of neuroAI research with the release of MindEye, and is currently working on follow-up research and the development of foundation models for neuroscience, also having helped develop state-of-the-art foundation models for radiology in collaboration with Stanford.

• Founded a year ago by Dr. Tanishq Mathew Abraham, MedARC focuses on open and collaborative medical AI research now with the addition of two exceptional members to the MedARC team: Dr. Paul Scotti (Head of Neuroimaging + AI) and Dr. Griffin Adams (Head of Clinical NLP).

• MedARC is now venturing into clinical NLP research, aiming to train state-of-the-art language models for the medical domain and plan to share model weights, methodology, and results transparently.

• MedARC has partnerships with Answer.AI, Princeton University, and VALID AI initiative, to help us achieve MedARC’s research agenda.

A year ago, MedARC was founded with the vision of a novel, open, & collaborative approach to medical AI research. It was created to develop large-scale AI foundation models for medicine & build interdisciplinary teams to address clinical needs with special interest in applying the latest advances in generative AI to medical applications. The team has made significant progress in building towards this vision.

“The role of generative AI in medicine cannot be underestimated and the MedARC community has achieved amazing results in just a year” said Stability AI CEO, Emad Mostaque. “We look forward to continuing to support it to catalyse research and impact that benefits us all.”

Here are some highlights from the past year:

Adding Additional Skill Sets

MedARC is thrilled to announce the addition of two exceptional members to its team:

• Dr. Paul Scotti - formerly a postdoc at Princeton University, he is leading our neuroimaging + AI efforts since November 2023.

• Dr. Griffin Adams - a recent PhD graduate from Columbia University, he is now leading our clinical NLP efforts since January 2024.

MedARC was founded and continues to be led by Dr. Tanishq Mathew Abraham. He joined Stability AI full-time in Aug. 2023 as a Research Director. Notably, Jeremy Howard remains engaged with MedARC in an advisory capacity. The team is actively growing, reinforcing its commitment to open and collaborative medical AI research.

Research Projects

Over the past year, MedARC has initiated and completed several impactful research projects:

Neuroscience+AI

Led by Dr. Paul Scotti, MedARC is at the forefront of neuroAI research, collaborating with leading institutions such as Princeton University, ENS Paris, and the University of Minnesota. MedARC’s first project in this space was MindEye, a state-of-the-art fMRI-to-image reconstruction pipeline. That is, given fMRI scans that record the brain activity of a person, this pipeline is able to reconstruct the image the subject was looking at with a striking degree of accuracy, better than any previous method. To do so, MedARC leveraged recent advances in image representations (CLIP) and image generation (Stable Diffusion). This work was accepted at NeurIPS 2023 as a Spotlight presentation.

MindEye co-authors presenting the Spotlight poster at NeurIPS 2023

Soon, MedARC will be sharing their follow-up work that significantly improves on MindEye in reconstruction quality and can use significantly less data (a property that is crucial for enabling practical applications). Additionally, the team is currently working on developing foundation models for fMRI data to enable more generalizable and simpler pipelines for fMRI-to-image reconstruction with several other projects in the pipeline, so stay tuned!

Radiology foundation models

MedARC has additionally helped develop state-of-the-art foundation models for radiology, through collaborations with Stanford Center for Artificial Intelligence in Medicine & Imaging (AIMI). MedARC was an original contributor and supporter for RoentGen, a version of Stable Diffusion fine-tuned on chest X-rays. This work was one of the first to demonstrate the benefit of data generated by a Stable Diffusion model for improving downstream model performance.

More recently, MedARC has contributed to an additional Stanford AIMI project: CheXagent, an 8B vision-language model for radiology. This model beats previously-developed general- and medical-domain foundation models on a variety of chest X-ray interpretation tasks. This is also one of the team’s first projects focusing on vision-language models and they will further explore this space with the belief that multimodal models are necessary for practical applications of generative AI in medicine.

Clinical NLP

Led by Dr. Griffin Adams, MedARC's clinical NLP efforts are gaining momentum. The team aims to train state-of-the-art language models for the medical domain and plan to share model weights, methodology, and results transparently.

In order to work towards this goal, MedARC has spent significant time thinking about model evaluation and the current state of language model medical capabilities. In their recent blog post, they evaluated numerous open LLMs on a variety of medical benchmark tasks (which are added to lm-eval-harness to transparent and reproducible analysis), and were surprised to discover the strong medical capabilities of existing open LLMs, already beating the proprietary Med-PaLM v1 model.

Partnerships

MedARC has various academic and industry partnerships to help them achieve their research agenda, including Answer.AI for efficient NLP training and Princeton University for advancing neuroimaging + AI research. As a founding partner of the VALID AI initiative, MedARC is actively contributing to exploring the applications, pitfalls, and best practices of Generative AI in healthcare and research.

MedARC team lunch (Dec. 2023)

To stay updated on MedARC's projects and progress, follow us on Twitter. If you’re interested in contributing to our research projects, join our Discord server.

We look forward to another year of groundbreaking advancements, collaborations, and contributions to the field of open and collaborative medical AI research.