Regulator loss functions and hierarchical modeling for safety decision making

Journal Medical Decision Making
Authors Hatfield LA, Baugh CM, Azzone V, Normand S-LT
Year Published 2017
Link to publication



Regulators must act to protect the public when evidence indicates safety problems with medical devices. This requires complex trade-offs among risks and benefits, which conventional safety surveillance methods do not incorporate.


To combine explicit regulator loss functions with statistical evidence on medical device safety signals to improve decision-making.


In the Hospital Cost and Utilization Project National Inpatient Sample, we select pediatric inpatient admissions and identify adverse medical device events (AMDEs). We fit hierarchical Bayesian models to the annual hospital-level AMDE rates, accounting for patient and hospital characteristics. These models produce expected AMDE rates (a safety target), against which we compare the observed rates in a test year to compute a safety signal. We specify a set of loss functions that quantify the costs and benefits of each action as a function of the safety signal. We integrate the loss functions over the posterior distribution of the safety signal to obtain the posterior (Bayes) risk; the preferred action has the smallest Bayes risk. Using simulation and an analysis of AMDE data, we compare our minimum-risk decisions to a conventional Z score approach for classifying safety signals.


The two rules produced different actions for nearly half of hospitals (45%). In the simulation, decisions that minimize Bayes risk outperform Z score-based decisions, even when the loss functions or hierarchical models are misspecified.


Our method is sensitive to the choice of loss functions; eliciting quantitative inputs to the loss functions from regulators is challenging.


A decision theoretic approach to acting on safety signals is potentially promising, but requires careful specification of loss functions in consultation with subject matter experts.