Post Created by Max Burton. LinkedIn Profile
Technology buzzwords seem to come-and-go at an increasing rate, so trying to determine which trends will make lasting impacts can be a challenge. Despite its current buzzword state, deep learning (DL) is an immensely powerful tool and will continue to expand in applications, which brings us to the medical device world. Several deep learning applications being leveraged are diagnosing cancer[3], assessing stroke severity [4], and monitoring heart disease[5]. These assessments could then be leveraged using models into medical devices to produce a diagnosis, patient monitoring, and continuous improvement of patient care to respond appropriately to a patient’s changes. Despite these promises, one of the large challenges posing deep learning in healthcare is that of managing its implementation and regulations.
The world of regulations is slow to move, and for good reason. They are designed to keep patients safe and ensure full traceability is provided for a device’s safety and efficacy. Machine learning, on the other hand, is an anathema to the way regulations have historically been designed.
Regulations and quality systems, along with the testing, evaluation, inspection, and the validation of products, have historically been based off some form of ground truth. Even a computer simulation has a ground truth set of variables. DL algorithms are beyond the scope of this article but are a series of mathematical formulas iterated repeatedly to slowly better fit a data set. These fittings are a black box of re-fit equations and cannot provide an exact ‘ground truth.’ They also cannot provide “Why” a given result was spat out.
The ‘black box’ nature of machine learning poses a problem then for FDA, Notified Bodies, and other regulatory agencies. It also provides a challenge for a practitioner’s understanding of Why an automated diagnosis was made. As a result, it’s hard for regulators to currently adequately define validation of DL for medical applications. These systems are currently being developed, with FDA slated to release a pilot program to pre-certify machine learning programs by the end of 2018 [6]. These programs still have a significant line of development before they are fully realized.
The FDA is also working to tackle another challenge of machine learning: continuous learning. Scott Gottlieb, the 23rd Commissioner of FDA, is working to establish a regulatory framework to handle these changes [7]. With continuous learning, there is no longer a set point in time that can be inspected against. Yesterday’s neural net may be different from today’s, and a medical device company may have challenges producing versioning records for a regularly changing system.
Another challenge posing the regulatory system is that of standards requirements. DL requires algorithmic training, and regularly utilizes a training and evaluation data set. These sets can be of varying length and quality. These variables cannot be easily accounted for when they create a dynamic output model. For example, a training set of 1,000 training items and 200 evaluation items will produce different results than that of 100 training and 20 evaluation. This will further be impacted by the training algorithm and the quality of training items. Also, historical statistical models are not as adept for managing the type of output produced by DL models. As a consequence, new challenges for properly sampling and generating DL models are raised.
As it continues to crest the hype cycle and transition into stable technology, DL has numerous challenges it needs to overcome prior to maturing. The field is highly promising and the regulatory landscape to manage devices implementing deep learning models is rapidly changing. As shown by FDA’s pre-certify program, new thinking and creative solutions will need to be implemented to ensure deep learning is effectively applied and controlled, so that medical devices can reap the benefits of improved cancer detection, disease monitoring, and other novel medical applications.
[0] Hype Cycle
[1]Dance Now:
https://arxiv.org/pdf/1808.07371.pdf
[2]Waymo Expansion:
https://futurism.com/waymo-fiat-autonomous-vehicles
[3]Diagnosing cancer with DL:
[4]Assessing stroke severity:
https://pubs.rsna.org/doi/abs/10.1148/radiol.2018171567?journalCode=radiology
[5]Monitoring heart disease:
https://www.nature.com/articles/s41598-018-19457-0
[6]
https://www.fda.gov/newsevents/speeches/ucm605697.htm
[7]
https://twitter.com/SGottliebFDA/status/984122279918481409
[8]
https://www.healthcatalyst.com/fda-digital-health-guidance-impact-on-cds-and-software
[9]
https://aitrends.com/machine-learning/whats-inside-black-box-machine-learning/