The DELTA (Data Extraction and Longitudinal Trend Analysis) System is designed to provide near real-time active safety surveillance of clinical (EHR) or clinical registry data during the course of evaluating a marketed medical device, medication, or therapeutic intervention. The system is compatible with a broad array of potential data sources and supports a variety of statistical methods, allowing for both unadjusted and risk-adjusted safety monitoring for prospective and retrospective analyses. The DELTA system architecture supports any number of simultaneous analyses limited only by the hardware capacity of the host system. DELTA also features flexible alerting mechanisms that can trigger notification (either through the system, the internet or via e-mail) when an observed event rate exceeds boundaries of risk-adjusted expectations for the event of interest, permitting analysts to monitor large numbers of simultaneous, prospective active surveillance studies. Additional analytic tools support subgroup analysis, triggered pre-specified sensitivity analysis, and drill-down functionality to permit signal verification and additional data exploration.

A typical cumulative adverse event-monitoring chart is shown here in which the red circles indicate a higher than expected adverse event rate for the studied medical device as compared with prospectively identified, propensity-matched control, population who received an alternative device.

DELTA-FigureSupported in part through MDEpiNet, FDA CDRH research grants, as well as private philanthropic foundation support, the DELTA projects are a part of the MDEpiNet Methodology Center of Harvard Medical School. The system has been validated in multiple medical device and medication safety surveillance domains and numerous data source environments from single hospital EHR systems, statewide clinical registries, national clinical registries, and distributed hospital networks.   In conjunction with the American College of Cardiology, current DELTA implementation projects include active surveillance of several National Cardiovascular Data Registries. In addition, a fully open-source version of DELTA will be released for academic and public health applications in the coming year.


Frederic Resnic, MD MSc – Lahey Hospital and Medical Center

Susan L. Robbins, BS – Lahey Hospital and Medical Center

Michael Matheny, MD MSc MPH – Vanderbilt University,

Sharon-Lise Normand, PhD – Harvard Medical School

Lucila Ohno-Machado, MD PhD – UCSD

Richard Cope – Boston Applied Analytics, Inc.

Sources of funding

NIH-NLM (R01 LM008142) “A System to Monitor Safety in Interventional Cariology”

FDA–HHSF (223200830058C) “Automated Medical Device Safety Monitoring”

FDA–DHHS (U01 FD004963) “Active Surveillance of Cardiovascular Devices: The Multi-Registry DELTA Network” Enhancing Active Surveillance of High Risk Medical Devices

Project Outputs

  1. Registries/data sources – Collaborative projects with the ACC NCDR national registries.
  2. Analytical systems
    DELTA3 Data Evaluation and Longitudinal Trend Analysis system version 3 as open-source tools to be distributed for use for academic and public health applications.
  3. Validated methods
    From linked analytic engine OCEANS Observational Cohort Event and Analysis Notification System
    Risk Adjusted Logistic Regression
    Risk Adjusted Propensity Score Matching
    Risk Adjusted Sequential Probability Ratio Testing
    Propensity matched prospective survival analysis (in development)
    Automated variable selection within Multivariate Logistic Regression toolkit.
  4. Open-source
    i. Actual programming code
    To be made available for academic and public health use through UCSD National Center for Biomedical Computing (NCBC) iDASH Program: Integrating Data for Analysis, Anonymization and Sharing



  1. Matheny ME, Arora N, Ohno-Machado L, Resnic FS. Rare adverse event monitoring of medical devices with the use of an automated surveillance tool. AMIA Annu Symp Proc. 2007 Oct 11:518-22.
  2. Matheny ME, Ohno-Machado L, Resnic FS. Risk-adjusted sequential probability ratio test control chart methods for monitoring operator and institutional mortality rates in interventional cardiology. Am Heart J. 2008 Jan;155(1):114-20.
  3. Matheny ME, Morrow DA, Ohno-Machado L, Cannon C, Sabatine MS, Resnic FS. Validation of an automated safety surveillance system with prospective randomized trial data. Med Decis Making 2008 Dec 2.
  4. Resnic FS, Gross TP, Marinac-Dabic D, Loyos-Berrios N, Donnelly S, Normand ST, Matheny M. Automated surveillance to detect post-procedure safety signals of approved medical devices: A study from the Massachusetts Angioplasty Registry.   2010 Nov 10; 304(18): 2019-2028.
  5. Vidi V, Matheny M, Donnelly S, Resnic FS. An Evaluation of a Distributed Medical Device Safety Surveillance System: The DELTA Network Study. Contemp Clin Trials. 2011 May; 32(3):309-17.
  6. Matheny M, Resnic FS, etal. Evaluation of an Automated Safety Surveillance System using Risk Adjusted Sequential Probability Ratio Testing. BMC Med Inform Decis Mak. 2011 Dec 14;11:75.
  7. Kumar A, Matheny ME, Ho KK, Yeh RW, Piemonte TC, Waldman H, Shah PB, Cope R, Normand SL, Donnelly S, Robbins S, Resnic FS.   The Data Extraction and Longitudinal Trend Analysis Network Study of Distributed Automated Postmarket Cardiovascular Device Safety Surveillance. Circ Cardiovasc Qual Outcomes. 2015 Jan;8(1)38-46.


“Automated Safety Surveillance and the MA Interventional Cardiology Pilot Study”/ Invited Lecturer, Medical Device Epidemiology Network Conference 2010.
Food and Drug Administration – Center for Device and Radiologic Health (CDRH), Silver Spring, MD

Automated Safety Surveillance and the MA Interventional Cardiology Pilot Study/ Featured Presenter, Medical Product Safety Roundtable Program (Webinar) 2010
Brookings Institute, Washington, D.C.

Automated Post-Market Safety Surveillance – The DELTA Surveillance Project/ Featured Presenter and Panel Member, Workshop on Public Health Effectiveness of the FDA 510(k) Clearance Process 2010
Institute of Medicine, Washington, D.C.

Improving the Post-Market Safety of Medical Devices in the United States/ Invited Witness and Panel Member 2011

United States Senate Special Committee on Aging, Hearing on Medical Device Approval Process and Safety Surveillance, Washington, D.C.

Automated Safety Surveillance and DELTA Program/ Invited Speaker and Panel Member, Medical Device Epidemiology Network Conference 2011
Food and Drug Administration – Center for Device and Radiologic Health (CDRH), Silver Spring, MD

Medical Device Safety Surveillance – Lessons from the DELTA Program/ Research Grand Rounds 2011
Duke Clinical Research Institute, Durham, North Carolina

Current Landscape of Post-Market Safety Surveillance for Medications and Devices in the United States/ Invited Speaker 2011
American Heart Association, Annual Scientific Meeting, Orlando, Florida

Automated Surveillance – Is it Feasible?
Invited Speaker and Panel Participant 2011
American College of Cardiology, Annual Scientific Sessions, Chicago, IL

Medical Device Safety – The MDEpiNet Program
Invited Speaker 2012
Brookings Institute, Washington D.C. (webinar)

Registries and Prospective Device Safety Surveillance
MDEpiNet Conference
Food and Drug Administration
Invited Speaker and Panel Participant 2012

Automated Safety Surveillance
Post-Market International Registries Workshop
Food and Drug Administration
Invited Speaker and Panel Participant 2012

Medical Device Post-Market Safety Surveillance in the United States
Medical Grand Rounds
Lahey Clinic Medical Center
Tufts University School of Medicine 2012

Active Surveillance – Update on the DELTA Project
Post Market Safety Monitoring Symposium
American College of Cardiology Annual Scientific Sessions
Invited Speaker 2013
San Francisco, CA

Medical Device Post-Market Safety Surveillance in the United States
Cardiovascular Grand Rounds
Tufts University School of Medicine 2013

Evolution of Continuous Safety Surveillance for Medical Devices
Medical Informatics Research Conference
Tennessee Valley VA Healthcare System and Vanderbilt University Medical Center
Nashville, TN   1/27/14
10 attendees

Active Surveillance of Medical Device Safety: The DELTA Project
Epidemiology Grand Rounds
Center for Devices and Radiologic Health
Food and Drug Administration
White Oak, MD   6/11/14
40 attendees: PhD, MD, FDA scientists.

Post-Market Safety Surveillance for Medical Devices: Current Status and Future Opportunities.
CUTEHeart Symposium – Sponsored by the Portuguese Ministry of Health
Faculty of Medicine, University of Lisbon
Lisbon, Portugal   12/11/14
60 attendees: MD, MPH, PhD international health services researchers

Active Surveillance and National Clinical Registries
S.M.A.R.T. Informatics Workshop
AHRQ Forum on Medical Device Safety
Rockville, MD 2/24/15
200 attendees; MD, PhD, MPH and Federal public health officials