Pandemics don’t wait: How we built COVID-Net in under 7 days

“Imagination is the highest form of Research” — Albert Einstein

On March 22nd our team announced the availability of COVID-Net (latest updates here), a neural network for COVID-19 detection using chest radiography (X-Rays). Our decision to open source the model, dataset and project source code, attracted an overwhelming response from researchers around the world and a significant amount of media coverage (here, here and here).

From the outset, we were clear that COVID-Net was intended as a complementary tool to assist clinicians in rapidly screening for the virus and not a replacement for the definitive but slower and more complex PCR test. One of the largest bottlenecks in triage and diagnosis is the need for experts to interpret radiography images.

Computer-aided diagnostic systems accordingly have the potential to save lives and more efficiently direct scarce medical resources. (To that end, we are co-authors of a just-released study of COVID-Net neural networks for COVID-19 lung disease severity assessment; we are deeply indebted to the good folks at NVidia for facilitating access to a powerful DGX2 that accelerated this analysis by an order of magnitude).

In this context, COVID-Net has been leveraged by researchers in Italy, Canada, Spain, Malaysia, the U.S and India, and a number of groups have extended our initial effort in impressive ways.

Yet beyond standard inquiries of how individuals could contribute to the project and leverage it responsibly, one question surfaced more than the others: how did we build such a high-performing and purpose-specific network so quickly?

As deep learning practitioners will attest, constructing a working neural network often requires — per Einstein’s declaration — imagination and creative intuition. It’s a laborious undertaking in which instinct is coupled with long trial-and-error cycles before something workable emerges. Yet as a wise writer said, creativity takes time and ideas simmer before they boil.

How, then, did we accelerate the development process behind the creation of COVID-Net?

Pandemics don’t wait

Such rapidity is especially important for fluid and fast-changing scenarios such as COVID-19 where data based on new cases emerges continuously and DNN architectures must be tailored to an ever-evolving knowledgebase. As such, we set a goal of ‘less than 7 days’ to develop and release COVID-Net to the global research community in hopes of quickly developing an important tool in combating the pandemic.

While similar efforts were underway in organizations throughout the world, they appeared to be private initiatives involving significantly larger teams with access to greater resources. This ZDNet article, for example, highlights a project at the Institute for Interdisciplinary Information Sciences at Tsinghua University in Beijing that involves a group of over 30 researchers.

By contrast, our team initially consisted of two individuals: Professor Alexander Wong, our Chief Scientist, and Linda Wang, one of our research students, who leveraged our platform to develop COVID-Net from scratch in under a week. An interesting byproduct of the endeavor is that it illustrated what could be achieved through breakthrough Explainable Artificial Intelligence (XAI) technology. Specifically, the team demonstrated that explainability:

  • Accelerates scaleable development beyond what humans and AI can accomplish independently
  • Provides greater transparency into model design and performance — for more trustworthy deep learning development and regulatory compliance

In pursuit of these objectives, the pair employed a human-machine collaborative design strategy in which they combined human-driven network design prototyping with machine-driven design exploration over five steps:

  1. Data collection
  2. Principled network design prototyping
  3. Machine-driven design exploration
  4. Validation via explainability
  5. Explaining and Understanding architecture

Rather than simply treating AI as a tool to be leveraged, this approach reimagines AI as a collaborator that learns from a developer’s needs and subsequently proposes multiple design approaches with different trade-offs in order to enable a rapid and iterative approach to model building.

In the following sections we outline the tasks our team undertook in the context of this strategy.

Step 1: Data collection

Image for post
Image for post

The COVIDx dataset consists of thousands of CXR images from public sources

While COVIDx has grown significantly since inception, it nevertheless exhibits data imbalances owing to the rarity of positive COVID-19 cases relative to other respiratory afflictions.

This problem is not uncommon, so model designers must remain vigilant and take steps to reduce impact of such biases.

Step 2: Principled network design prototyping

As our starting point, we leveraged residual architecture design principles as they enable reliable neural network architectures which are easy to train to high performance and which enable the construction of deeper architectures.

To help clinicians better triage as well as decide on the treatment strategies to employ, we designed our prototype to make one of three predictions (three-class softmax output):

  1. No infection (normal)
  2. Non-COVID-19 infection (e.g., non-COVID19 viral, bacterial, etc.)
  3. COVID-19 viral infection

From this point, the initial prototype needed to be converted into full-fledged deep neural networks.

Step 3: Machine-driven design exploration with Generative Synthesis

Rather than relying on a brute-force approach and exploring ad-hoc permutations, we instead combine GenSynth with our understanding of the domain requirements to promote a systematic and intelligent approach to design exploration.

Specifically, given data and human-defined requirements, the platform guides a design exploration that learns and identifies the optimal macro-and micro-architectures with which the final model can be constructed. Such a machine-driven approach enables greater flexibility than its human-driven counterpart while also ensuring the resulting network satisfies operational constraints.

Image for post
Image for post

DarwinAI’s GenSynth platform makes it easy for designers to explore and generate custom deep learning models tailored for data and task at hand

For COVID-Net, our operational requirements included greater than 80 percent sensitivity and positive predictive value (i.e., probability that a positive prediction is in fact correct). These parameters were chosen to enable the platform to strike an appropriate balance between accuracy and speed as a key consideration was designing a network that could run on different platforms — be it in the cloud or on an edge device (perhaps even the actual imaging device itself).

Using this information, GenSynth generated a number of different ‘ready-to-go’ models that met our design requirements subject to different characteristics and trade-offs.

From there, we were able to develop a detailed understanding of the design choices made by the platform, which in turn helped guide us as we explored and refined our model. Specifically, GenSynth not only helped us design new models, but identify key performance bottlenecks giving us greater transparency into the overall make-up and performance of the network itself. Such human-machine collaboration enables the creation of unique, tailored designs with different and — most importantly — understood trade-offs.

Image for post
Image for post

The GenSynth platform enabled informed design choices within generated models; this image shows different layers within different generated models, with red highlighting performance bottlenecks

Step 4: Validation via Explainability

The ‘black box’ problem that results from this limitation is a key reason why design audits are frequently omitted from DL workflows, as the alternatives to XAI-based assessments are cumbersome, time-consuming, and often involve scripts, interpretations and considerable manual effort. Moreover, they aren’t especially effective, most notably for unusual and non-intuitive cases, such as the example we’ll illustrate in a moment.

However, investing the time to audit your model can dramatically accelerate and simplify development: identifying the gaps in the design and the underlying factors behind them greatly increases your ability to design effectively and can avoid considerable pain and debugging down the road. Moreover, the insights gained through explainability can not only be used to generate better networks, but can also illustrate why they reach particular conclusions.

In the case of COVID-Net, using GenSynth to audit our design allowed us to:

  • Identify error scenarios
  • Understand the reliability of our model
  • Gain valuable insights into how to improve our model

All of these lessons contributed to building trust with users and ensuring long-term design efficacy.

Image for post
Image for post

A confusion matrix for COVID-Net generated by GenSynth

Starting from the confusion matrix above, the platform automatically groups different error scenarios so as to provide a quick high-level picture of how the network is performing. GenSynth further allows us to drill into the different error scenarios to pinpoint specific biases and gaps, as well as understand the critical factors behind model decisions as illustrated in the images below.

Image for post
Image for post

Example CXR images of COVID-19 cases from several different patients and their associated critical factors (highlighted in red) as identified by GenSynth

In addition to employing XAI for responsible and transparent design, the ability to interpret and obtain insight into how COVID-Net identifies viral infections is important for:

  • Increasing trust: By identifying the critical factors in the decision-making process, the predictions made by COVID-Net are made more transparent and trustworthy to clinicians, which can assist them in making faster yet more accurate assessments.
  • Revealing new insights: Highlighting critical factors can help clinicians unearth new insights into the key visual indicators behind COVID-19 viral infection, which they can appropriate into their own diagnostic processes to further improve screening accuracy

Example: When models make the right decision for the wrong reasons

Thus far, we’ve examined COVID-Net’s analysis against X-rays, but our team has been hard at work on COVID-Net-CT, which performs similar diagnostics using CT scans. In the course of auditing one of our earlier model designs we encountered an issue that would have been very difficult to identify in absence of XAI capabilities.

The figure below depicts 15 CT scans in which the model correctly detected COVID-19 infections. By highlighting the critical factors that led to the detections GenSynth revealed a startling phenomenon: the positive readings in this case weren’t not based on anomalies in the patients’ lungs, but rather the appearance of the bed of the CT scanner itself.

Identifying this false cue was crucial in improving the model; in this case, it compelled us to revisit how we created and processed our data and how we trained our models.

Image for post
Image for post

In these examples, GenSynth revealed that the critical decision factor (lighter gray areas) responsible for correct COVID-19 diagnoses based upon CT scans was the appearance of the bed of the CT scanner

Given the data-driven nature of deep learning, ‘right decision for the wrong reason’ scenarios are not uncommon, and can be extremely difficult to track and identify without an XAI-driven auditing strategy. As such, the value of explainability in improving the reliability of deep neural networks for clinical application cannot be understated.

Step 5: Explaining and understanding the COVID-Net architecture

COVID-Net exhibits an efficient micro-architecture design largely composed largely of 1x1 convolutional layers and depth-wise convolution layers. The heavy use of a projection-expansion-projection (PEPX) design pattern facilitates formidable performance efficiencies, while still maintaining strong COVID-19 sensitivity and PPV.

The model also includes selective long-range connectivity, which is fairly unusual as residual networks usually exhibit short-range connectivity. This feature is a result of a fundamental trade-off between performance and memory footprint. In the case of COVID-Net, being selective about employing long-range connectivity only where necessary minimizes the overall footprint in accordance with our human-directed operational parameters.

Image for post
Image for post

COVID-Net, in all its glory–the design employs a diverse collection of architectural traits that result in a high-performance model purpose-built for making accurate COVID-19 detections based upon chest X-ray images

Going forward

  • Accelerating scalable development beyond what humans and AI can accomplish independently
  • Providing greater transparency into model design and performance for more trustworthy deep learning development and regulatory compliance

Since releasing COVID-Net and delivering a webinar on the subject, we’ve benefitted from the support of the wider medical and AI community, including a significant increase in the number of X-ray images comprising the COVIDx dataset.

Going forward, we will continue to iterate on model improvements so as to:

  • Build upon COVID-Net to improve it as more data arrives
  • Design new neural network architectures for better sensitivity and PPV
  • Design new strategies for training COVID-Net to higher performance

As mentioned, our team is also working on COVID-Net-CT, a neural network tailored for COVID-19 detection via CT scans (we are likewise working on augmenting the COVIDx dataset with CT instances). For context, CTs provide superior levels of detail and clarity when compared to conventional X-Rays, and several studies suggest higher levels of sensitivity than viral testing.

Finally, our team is also hard at work on COVID-Net-Risk, a neural network tailored for COVID-19 risk stratification. The goal with COVID-Net-Risk is to provide greater insights into risk level and severity to assist caregivers with triage and treatment plans.

This article only scratches the surface of what is possible using the GenSynth platform.

Since the beginning, the goal at DarwinAI has been to unlock the potential of AI by accelerating deep learning through our unique technology. The construction of COVID-Net illustrates the promise of this approach.

Is there a deep learning project your team is looking to accelerate? If so, contact us and learn how we can help.

Likewise, questions about COVID-Net, contributing to the project, or GenSynth more generally can be directed here.

Stay tuned, the best is yet to come.

Written by

CEO, DarwinAI

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store