COVID-Net: Final Reflections

Thoughts and thanks as we transition the project to the University of Waterloo

“Sheldon, ping me when you have a moment — I have something you should see.”

I can still recall reading the Slack message above from our Chief Scientist and wincing. Not intentionally of course. Professor Wong, or Alex, as I’d called him since we started DarwinAI, is the most brilliant mind I’ve encountered in my professional career (recently recognized by Stanford as such). Problem was, I didn’t have a moment. Or a half-moment. It was March of 2020 and the world was on fire.

Of course that’s how it felt in the formative days of the pandemic. Rising infection rates, ominous warnings from abroad, and general ignorance of the virus and its societal implications spawned anxieties few of us will forget. My world consisted of a board prudently imploring us to take drastic measures to keep the business afloat and a childcare plan that collapsed overnight (daycare, what daycare?). I was overwhelmed.

But Alex persisted. A second message followed, then a third. And finally he dropped it on me: in the seven days since the provincial lockdown, he and the team had developed a neural network that could diagnose COVID-19 with uncanny accuracy.

I smiled. At the absurdity of it all or being humbled by a more powerful mind I’m not sure, but it was a small moment of levity amidst the global whirlwind: while our COO and I were taking steps to ensure the survival of DarwinAI, Alex and the team were taking steps to ensure the survival of the species.

We released COVID-Net on March 22, 2020. And today — 613 days later — we’re transitioning ownership of the project along with maintenance and support of the codebase to the wonderful team at the Vision and Image Processing Lab at the University of Waterloo.

While the story of how we rapidly developed COVID-Net is interesting and highlights key elements of our technology, in this post — our final on the topic — I’d like to reflect on our twenty-month journey around the initiative and the contributors and corporate partners that made it possible.

The response to our initial announcement of COVID-Net was overwhelming. As I wrote at the time, in determining “how to best deploy our skills in service of the present crisis” we “collaborated with researchers at the University of Waterloo’s VIP Lab to develop COVID-Net: a convolutional neural network for COVID-19 detection via chest radiography.” Moreover, we decided to fully open source the project “in hopes of developing a robust tool to assist health care professionals in combating the pandemic.”

Academics, researchers and developers were invited to peruse and enhance the codebase and did so in an impressive fashion and we were flooded with suggestions and updates from motivated individuals around the world. Word of the project also spread through substantial press coverage in those initial months, including pieces in MIT Tech, Forbes, Synced, VentureBeat, BetaKit, ZDNET and PC Magazine. Like us, folks in the AI community were grappling for a way to deploy their skills against the pandemic and COVID-Net seemed one straightforward way to do it.

But what exactly was the point behind an AI tool that could identify COVID given its small, but still non-zero, margin of error? After twenty months, it can be hard to recall the diagnostic challenges at onset of the crisis. Below is an extract from one of the many proposals we authored at the time:

At present, there are simply not enough materials and resources to meet the screening demands for COVID-19 using the standard Polymerase Chain Reaction (PCR) test. As has been widely documented, the PCR approach, though reliable, is not straightforward and requires specialized skills, laboratory equipment, and particular chemicals. With diminishing supplies and an increasing number of cases, patients are waiting days to get results.

This limited screening velocity presents challenges to both containing the pandemic and identifying those most affected by the virus. The proposed project is an important step in creating an alternative testing method that will allow assessment centers to triage patients more rapidly, increasing the rate of diagnosis and patient flow and enable hospitals to better utilize resources as the pandemic continues.

Initially, then, COVID-Net was intended to supplement the definitive but more arduous PCR test. Per an interview I gave two weeks after we released the source code “given the limitations around the classical test, the chemicals one needs, the lack of resources, the expertise you need to carry it out, the time it takes to around a result, this is a complementary tool that will give you a statistical answer in a very quick period of time.”

By providing a rapid, if probabilistic, method of diagnosis the argument went, we could triage vulnerable populations more effectively to help curtail the spread of the virus (e.g., “there is a 93% chance you’ve contracted COVID, self-isolate until we can administer a more robust test”). For marginalized and isolated communities with limited resources, COVID-Net proved a useful tool to this end and we received word the system had found its way to India, Malaysia, Italy, the United Kingdom, and even the Australian Outback where government officials were evaluating the pandemic’s propagation to the remotest parts of the country.

In light of the overwhelming — and unexpected — response to the initiative, we assigned a small, part-time team to work on the project with their counterparts at the University of Waterloo. The result was a steady stream of enhancements per the updates below:

1. March 29, 2020: COVID-Net: larger dataset, new models, and COVID-RiskNet
2. Apr 19, 2020: COVID-Net: More data, CT Scans and Risk Stratification
3. May 21, 2020: COVID-Net Update: More data, New models
4. May 25, 2020: Pandemics don’t wait: How we built COVID-Net in under 7 days
5. Jul 9, 2020: COVIDNET-CT: a new detection model for CT scans
6. Sept 21, 2020: COVIDNet-S: A neural network for grading COVID Severity
7. Jan 26, 2021: COVID-NET CT-2: An Evolutionary Leap
8. July, 2021: COVID-Net Clinical ICU: Predicting ICU Admission for COVID-19 Patients
9. Aug, 2021: COVID-Net US: A Highly Tailored Network for Ultrasound COVID-19 Screening

What sticks out the most during those initial months was the unspoken camaraderie that developed around the project. Amidst the frightening and fluid circumstances of the pandemic our contributors went above and beyond, showering COVID-Net with their time and expertise in ways that significantly improved the system.

Our corporate partners also stepped up with resources, support, and general goodwill. HPE, Intel, NVidia and ARM all generously provided hardware, software tools, and logistical support at various points in the project, resulting in an array of case studies that exemplified the system’s versatility:

1. NVidia: DarwinAI Achieves 96% Screening Accuracy for COVID-19 with Diverse CT Dataset
2. ARM: Embedded AI for Healthcare: How We Built COVID-Net for Embedded Devices
3. Intel: DarwinAI’s Deep Learning AI Screen Tool Helps Detect COVID-19
4. HPE: Transparent, Dynamic, and Democratic AI

In the months that followed, COVID-Net evolved from a narrow technical experiment into a robust clinical tool with broad application. The following diagram, extracted from our latest paper on the topic, illustrates the comprehensive and present state of the system for clinical decision support.

Figure 1: The current COVID-Net system for COVID-19 CXR decision support

As illustrated above, there are four parts to the COVID-Net system.

1. ) The COVIDx dataset — a large and diverse corpus of CXR images used to train the AI. At present, the repository comprises 16,560 images from 15,528 patients across 51 countries, making it the one of the largest datasets in open access form.
2. The COVID-Net model — the deep neural network responsible for case detection. The model currently differentiates between positive COVID-19 infections, pneumonia infections and normal patients. As of this writing, the model has an accuracy of 93.3% with a sensitivity of 91.0% and a positive predictive value of 98.9%.
3. The COVID-Net S model — a deep neural network responsible for scoring the lung disease severity for COVID-positive patients. The predicted scores can be used by clinicians and healthcare workers to obtain a better understanding of the patient’s disease stage and progression, which can then be used for individualized patient care decisions and treatment planning (e.g., oxygen therapy, ventilator use).
4. The COVID-NET User Interface — a report generation and case management interface that allows medical professionals to query, view and analyze patient scans in a streamlined fashion.

Figure 1 also illustrates the evolving clinical benefits of the system: far from a complementary diagnostic tool, COVID-Net is now used to grade disease severity and assist with patient treatment planning. The system supports numerous tasks around the clinical decision support workflow using automatic report generation features to assist clinicians with treatment decisions.

For example, a clinician using the system to assess a new patient would first evaluate their X-ray data using the COVID-Net classification model; in the case of a positive diagnosis, they’d then use the COVID-Net S model to assess disease severity and could plan for ICU admission if the diagnosis was severe enough.

Our efforts around COVID-Net culminated in a highly acclaimed paper in Nature that described the most innovative aspects of the system. Of particular importance was the way in which our Explainability technology could identify the virus’s repository markers and manifestations per the figure below.

Figure 2: COVID-Net highlighting critical factors for COVID-positive patients

The benefits of this feature — pinpointing the critical factors associated with COVID-19 — were twofold. First, it allowed clinicians to validate that the predictions made by COVID-Net aligned with their intuitive and expert understandings. Second, it unearthed new insights into the visual indicators behind COVID viral infections, which health professionals could integrate into their own diagnostic processes to improve screening velocity and accuracy.

In tandem, these benefits exemplify the power of Explainable AI (XAI): it enables the subject matter expert to validate the system is making the right decisions for the right reasons, while potentially enlightening that same individual on their very area of expertise. This iterative back-and-forth — the incremental sharpening of knowledge and understanding between human and machine — provides a compelling guidepost for our AI-filled future. That this theme was captured in Nature by way of COVID-Net seemed a worthwhile culmination of the project we started twelve months before.

Two strategic partnerships were key in advancing COVID-Net to the extent that we did.

First, Lockheed Martin, who we announced a strategic collaboration with shortly after the pandemic began, allocated significant funding to the project under Canada’s Industrial Benefits Policy (ITB) program. This capital allowed us to dedicate a full-time team to the project to coordinate research, conduct user testing, and build out the end-to-end system illustrated in Figure 1.

It also underscored the importance of the ITB policy to organizations like ours, in which aerospace and defense companies can deploy non-dilutive funding to Canadian startups in the interest of the common good to drive economic growth. Lorraine Ben, the CEO of Lockheed Martin Canada, emphasized this point in a joint article in the Canadian Defense Review:

“At Lockheed Martin Canada, the work we do is laying a foundation for decades to come in terms of innovation, job growth, STEM development, economic prosperity and security. Partnerships play a critical role in this effort, just like this exciting and timely initiative with DarwinAI. In a most challenging year, it is inspiring to see how Canada’s Industrial and Technological Benefits Policy drives enduring investments for Canadian industry and how it can accelerate solutions to the most complex challenges.”

Simply put, COVID-Net would be a very different — and likely very limited — beast were it not for Lockheed Martin and the Canadian government that approved the funding.

Our second integral partnership was our collaboration with Red Hat and Boston Children’s Hospital (BCH) to make the system accessible to clinicians and health care professionals. In the early days, COVID-Net was an unwieldy assortment of technical assets that had to be wired together and orchestrated to generate results. This was unsurprising given its academic roots, but daunting and of limited use to folks on the front line.

In cooperation with the Fetal Neonatal Neuroimaging and Developmental Science Center (FNNDSC) at BCH, our team developed the initial GUI that sat atop the hospital’s ChRIS system using Red Hat’s OpenShift framework. For our team, it was thrilling to work with the experts at a leading institution and understand the nuances of the virus from a diagnostic point of view. Clinicians from other institutions eventually joined the effort and helped transform our initial front-end into the slick and intuitive user interface depicted in Figure 3.

Figure 3: The COVID-Net User Interface

The screenshot above illustrates the key features of the COVID-Net system:

• Creating new predictive analyses (top)
• Finding and examining previously completed analyses
• Examining radiological images corresponding to an analysis using the COVID-Net UI radiology viewer (bottom)
• Generating a PDF report of a predictive analysis

In combination, these features allow a clinician to assess a new patient for COVID-19, review the patient’s medical images and compare them against previously been assessed patients, and share the results with other medical professionals. A minor part of the complex workflow behind COVID treatment, but one we’re proud to have simplified.

In reflecting on our company’s journey around COVID-Net, the sentiments are numerous: awe with its creators, who designed and developed the system so selflessly during a time of international crisis; gratitude to its contributors, who deployed their time and talents to enhance the project in ways too numerous to mention; appreciation for our corporate partners, whose resources and funding transformed an academic initiative into an enterprise-grade application; and finally, unmitigated respect for the front-line workers — clinicians, radiologists, nurses and other health care practitioners — whose cooperation gave us a small window into their on-the-ground efforts that continue to alleviate vast amounts of human suffering amidst these challenging times.

Today, COVID-Net has found a suitable home at the VIP Lab at the University of Waterloo. The latest project repository can be accessed here, and curious readers are encouraged to continue to experiment and build upon its solid foundation.

And, ever the entrepreneurs, our team has repurposed the core technology that enabled COVID-Net to commercial ends. Most notably, through our manufacturing solutions for quality inspection, which detects anomalies and defects for parts and goods with unerring precision, much like COVID-Net analyzes a CRX scan. Yet a hint of the humanitarian remains: one of our most exciting projects involves an inventive and forward-looking use-case around food security and insect agriculture (video here).

Much time has passed since the world was flipped on its head in March of 2020. The challenges and changes persist. But our work continues. The story goes on. And, to paraphrase a US president, an angel still rides in the whirlwind and directs this storm.

In Solidarity,
Sheldon Fernandez
CEO, DarwinAI
Nov 2021