Facebook AI In Collaboration With NYU Introduce New Machine Learning (ML) Models To Predict COVID Patient’s Health Condition

Even after a year-long fight with the COVID pandemic, it’s still a challenge to predict a patient’s condition throughout treatment. In collaboration with NYU Langone Health’s Department of Radiology and Predictive Analytics Unit, Facebook AI developed three machine learning (ML) models to help healthcare providers and doctors predict how a patient’s condition may develop and plan accordingly. It will further help hospitals ensure they have the required resources for the patients’ care. These three ML models are as follow: 

1) A model that predicts a patient’s deterioration based on a single X-ray

2) A model that predicts a patient’s deterioration based on a sequence of X-rays 

3) A model that predicts the amount of supplemental oxygen (if any) a patient might need based on a single X-ray.

The collaboration has come up with a model, using sequential chest X-rays, that can predict up to 96 hours in advance if the patient requires some intensive care. 

Although we saw some progress with the previous methods, which used supervised training and single timeframe images, the process of labeling data is exceptionally time-intensive and limiting. The team chose to keep the above in mind to pre-train their Machine Learning system on two vast, public chest X-ray data sets: MIMIC-CXR-JPG and CheXpert. The above was done using a self-supervised learning technique called Momentum Contrast (MoCo). The above made it possible to use the large amounts of non-COVID chest X-ray data to train a neural network that can extract information from chest X-ray images. Then the MoCo model was fine-tuned using an extended version of the NYU COVID-19 data set.

MoCo relies on unsupervised learning using a contrastive loss function. It maps images to a latent space wherein similar images are mapped to vectors close together. These vectors are used as feature representations. Vectors allow users to train classifiers using a small number of labeled examples. 

After pre-training the MoCo model on MIMIC-CXR-JPG and CheXpert, the team used the pre-trained model to build classifiers. These classifiers could predict whether a COVID-19 patient’s condition is likely to deteriorate. Two kinds of classifiers were built to predict a patient’s health decline. The first model for predicting a patient’s deterioration is based on a single X-ray (similar to a previous study). The second model predicts a patient’s deterioration based on a sequence of X-rays by aggregating the image features via a Transformer model.


Building an ML model that can use a sequence of X-rays to predict health status mirrors how human radiologists work. The above method also accounts for the evolution of COVID infections over time.

According to William Moore, MD, a Professor of Radiology at NYU Langone Health, their team has successfully shown that serial chest radiographs can predict the need for escalation of care in patients with their model.  

Paper: https://arxiv.org/pdf/2101.04909.pdf

Github: https://github.com/facebookresearch/CovidPrognosis

Source: https://ai.facebook.com/blog/new-ai-research-to-help-predict-covid-19-resource-needs-from-a-series-of-x-rays/

Consultant Intern: He is Currently pursuing his Third year of B.Tech in Mechanical field from Indian Institute of Technology(IIT), Goa. He is motivated by his vision to bring remarkable changes in the society by his knowledge and experience. Being a ML enthusiast with keen interest in Robotics, he tries to be up to date with the latest advancements in Artificial Intelligence and deep learning.

[Announcing Gretel Navigator] Create, edit, and augment tabular data with the first compound AI system trusted by EY, Databricks, Google, and Microsoft