Transformer Models vs. Convolutional Neural Networks to Detect Structural Heart Murmurs

Posted in Published Research
Summary: Read about the performance comparison of a transformer-based machine learning model with an FDA-cleared Convolutional Neural Network (CNN) model for detecting structural heart murmurs, a common indicator of VHD. The transformer model, trained on unlabeled datasets and fine-tuned on labeled datasets, exhibited superior performance compared to the CNN model across multiple datasets.

 

Authors: George Mathew, Daniel Barbosa, John Prince, Caroline Currie, Eko Health

Background: Valvular Heart Disease (VHD) is a leading cause of mortality worldwide and cardiac murmurs are a common indicator of VHD. Yet standard of care diagnostic methods for identifying VHD related murmurs have proven highly variable amongst providers, often yielding a low detection rate. Adoption of novel technologies, such as digital stethoscopes and artificial intelligence, can serve to address these challenges.

Methods: This research compares the performance of a novel transformer-based machine learning model (pre-trained on unlabeled datasets and fine-tuned on labeled datasets) against an FDA cleared ResNet Convolutional Neural Network (CNN) model. Both models are validated on the same two datasets where all subjects received confirmatory echocardiograms and recordings were independently annotated by an expert panel. The pairing of heart sound labels and echocardiogram findings determine the ground truth for murmurs caused by VHD. Transformer operating point is determined by matching the sensitivity of the CNN model.

The Eko Test Dataset consists of 1893 recordings from 593 unique subjects (NCT04400513). The Eko Real World Evidence (RWE) Dataset consists of 2087 recordings from 368 unique subjects (NCT05459545).

Results: On the whole Eko Test Dataset, the CNN model achieved an AUC of 96.6% with a 93.8% sensitivity and 85.4% specificity. The transformer model achieved an AUC of 98.2% with a 93.8% sensitivity and 91.5% specificity.
On the adult sub-population of the Eko Test Dataset, the CNN model achieved an AUC of 98.5% with a 96.3% sensitivity and 88.9% specificity. The transformer model achieved an AUC of 99.2% with a 96.1% sensitivity and 94.9% specificity.
On the RWE dataset, the CNN model achieved an AUC of 96.3% with a 85.3% sensitivity and 94.4% specificity. The transformer model achieved an AUC of 98.1% with a 85.3% sensitivity and 97.1% specificity.

Conclusion: The demonstrated effectiveness of the transformer model, with its reduction in false positive rates while maintaining sensitivity, enhances the potential for early VHD detection without imposing additional burdens on healthcare systems.

View Research -->

MKT-0002951