Publications
Lab publications by year. For organization by project, see projects.
How to make this wordcloud from a bibtex library
2024
- Interpretable Machine Learning Predicts Postpartum Hemorrhage with Severe Maternal Morbidity in a Lower Risk Laboring Obstetric PopulationAmerican Journal of Obstetrics & Gynecology MFM, 2024
Background Early identification of patients at increased risk for postpartum hemorrhage (PPH) associated with severe maternal morbidity (SMM) is critical for preparation and preventative intervention. However, prediction is challenging in patients without obvious risk factors for postpartum hemorrhage with severe maternal morbidity. Current tools for hemorrhage risk assessment use lists of risk factors rather than predictive models. Objective To develop, validate (internally and externally), and compare a machine learning model for predicting PPH associated with SMM against a standard hemorrhage risk assessment tool in a lower-risk laboring obstetric population. Study Design This retrospective cross-sectional study included clinical data from singleton, term births (>=37 weeks’ gestation) at 19 US hospitals (2016-2021) using data from 44,509 births at 11 hospitals to train a generalized additive model (GAM) and 21,183 births at 8 held-out hospitals to externally validate the model. The outcome of interest was PPH with severe maternal morbidity (blood transfusion, hysterectomy, vascular embolization, intrauterine balloon tamponade, uterine artery ligation suture, uterine compression suture, or admission to intensive care). Cesarean birth without a trial of vaginal birth and patients with a history of cesarean were excluded. We compared the model performance to that of the California Maternal Quality Care Collaborative (CMQCC) Obstetric Hemorrhage Risk Factor Assessment Screen. Results The GAM predicted PPH with an area under the receiver-operating characteristic curve (AUROC) of 0.67 (95% CI 0.64-0.68) on external validation, significantly outperforming the CMQCC risk screen AUROC of 0.52 (95% CI 0.50-0.53). Additionally, the GAM had better sensitivity of 36.9% (95% CI 33.01, 41.02) than the CMQCC screen sensitivity of 20.30% (95% CI 17.40, 22.52) at the CMQCC screen positive rate of 16.8%. The GAM identified in-vitro fertilization as a risk factor (adjusted OR 1.5; 95% CI 1.2-1.8) and nulliparous births as the highest PPH risk factor (adjusted OR 1.5; 95% CI; 1.4-1.6). Conclusion Our model identified almost twice as many cases of PPH as the CMQCC rules-based approach for the same screen positive rate and identified in-vitro fertilization and first-time births as risk factors for PPH. Adopting predictive models over traditional screens can enhance PPH prediction.
@article{lengerich2024interpretable, title = {Interpretable Machine Learning Predicts Postpartum Hemorrhage with Severe Maternal Morbidity in a Lower Risk Laboring Obstetric Population}, author = {Lengerich, Benjamin J and Caruana, Rich and Painter, Ian and Weeks, William B and Sitcov, Kristin and Souter, Vivienne}, journal = {American Journal of Obstetrics \& Gynecology MFM}, pages = {101391}, year = {2024}, publisher = {Elsevier}, } - Contextualized: Heterogeneous Modeling ToolboxJournal of Open Source Software, 2024
Heterogeneous and context-dependent systems are common in real-world processes, such as those in biology, medicine, finance, and the social sciences. However, learning accurate and interpretable models of these heterogeneous systems remains an unsolved problem. Most statistical modeling approaches make strict assumptions about data homogeneity, leading to inaccurate models, while more flexible approaches are often too complex to interpret directly. Fundamentally, existing modeling tools force users to choose between accuracy and interpretability. Recent work on Contextualized Machine Learning (Lengerich et al., 2023) has introduced a new paradigm for modeling heterogeneous and context-dependent systems, which uses contextual metadata to generate sample-specific models, providing context-specific model-based insights and representing data heterogeneity with context-dependent model parameters. Here, we present Contextualized, a SKLearn-style Python package for estimating and analyzing personalized context-dependent models based on Contextualized Machine Learning. Contextualized implements two reusable and extensible concepts: a context encoder which translates sample context or metadata into model parameters, and sample-specific model which is defined by the context-specific parameters. With the flexibility of context-dependent parameters, each context-specific model can be a simple model class, such as a linear or Gaussian model, providing direct model-based interpretability without sacrificing overall accuracy.
@article{ellington2024contextualized, doi = {10.21105/joss.06469}, url = {https://doi.org/10.21105/joss.06469}, year = {2024}, publisher = {The Open Journal}, volume = {9}, number = {97}, pages = {6469}, author = {Ellington, Caleb N. and Lengerich, Benjamin J. and Lo, Wesley and Alvarez, Aaron and Rubbi, Andrea and Kellis, Manolis and Xing, Eric P.}, title = {Contextualized: Heterogeneous Modeling Toolbox}, journal = {Journal of Open Source Software}, } - Contextualized Policy Recovery: Modeling and Interpreting Medical Decisions with Adaptive Imitation LearningInternational Conference on Machine Learning (ICML), 2024
Interpretable policy learning seeks to estimate intelligible decision policies from observed actions; however, existing models fall short by forcing a tradeoff between accuracy and interpretability. This tradeoff limits data-driven interpretations of human decision-making process. e.g. to audit medical decisions for biases and suboptimal practices, we require models of decision processes which provide concise descriptions of complex behaviors. Fundamentally, existing approaches are burdened by this tradeoff because they represent the underlying decision process as a universal policy, when in fact human decisions are dynamic and can change drastically with contextual information. Thus, we propose Contextualized Policy Recovery (CPR), which re-frames the problem of modeling complex decision processes as a multi-task learning problem in which complex decision policies are comprised of context-specific policies. CPR models each context-specific policy as a linear observation-to-action mapping, and generates new decision models on-demand as contexts are updated with new observations. CPR is compatible with fully offline and partially observable decision environments, and can be tailored to incorporate any recurrent black-box model or interpretable decision model. We assess CPR through studies on simulated and real data, achieving state-of-the-art performance on the canonical tasks of predicting antibiotic prescription in intensive care units (+22% AUROC vs. previous SOTA) and predicting MRI prescription for Alzheimer’s patients (+7.7% AUROC vs. previous SOTA). With this improvement in predictive performance, CPR closes the accuracy gap between interpretable and black-box methods for policy learning, allowing high-resolution exploration and analysis of context-specific decision models.
@article{deuschel2024contextualized, title = {Contextualized Policy Recovery: Modeling and Interpreting Medical Decisions with Adaptive Imitation Learning}, author = {Deuschel, Jannik and Ellington, Caleb and Luo, Yingtao and Lengerich, Ben and Friederich, Pascal and Xing, Eric}, informal_venue = {ICML}, year = {2024}, journal = {International Conference on Machine Learning (ICML)} }
2023
- Contextualized Machine Learning2023
We examine Contextualized Machine Learning (ML), a paradigm for learning heterogeneous and context-dependent effects. Contextualized ML estimates heterogeneous functions by applying deep learning to the meta-relationship between contextual information and context-specific parametric models. This is a form of varying-coefficient modeling that unifies existing frameworks including cluster analysis and cohort modeling by introducing two reusable concepts: a context encoder which translates sample context into model parameters, and sample-specific model which operates on sample predictors. We review the process of developing contextualized models, nonparametric inference from contextualized models, and identifiability conditions of contextualized models. Finally, we present the open-source PyTorch package ContextualizedML.
@article{lengerich2023contextualized, title = {Contextualized Machine Learning}, author = {Lengerich, Ben and Ellington, Caleb N. and Rubbi, Andrea and Kellis, Manolis and Xing, Eric P.}, year = {2023}, archiveprefix = {arXiv}, } - Interpretable Predictive Models to Understand Risk Factors for Maternal and Fetal OutcomesTomas M. Bosschieter, Zifei Xu, Hui Lan, and 5 more authorsJournal of Healthcare Informatics Research, 2023
Although most pregnancies result in a good outcome, complications are not uncommon and can be associated with serious implications for mothers and babies. Predictive modeling has the potential to improve outcomes through a better understanding of risk factors, heightened surveillance for high-risk patients, and more timely and appropriate interventions, thereby helping obstetricians deliver better care. We identify and study the most important risk factors for four types of pregnancy complications: (i) severe maternal morbidity, (ii) shoulder dystocia, (iii) preterm preeclampsia, and (iv) antepartum stillbirth. We use an Explainable Boosting Machine (EBM), a high-accuracy glass-box learning method, for the prediction and identification of important risk factors. We undertake external validation and perform an extensive robustness analysis of the EBM models. EBMs match the accuracy of other black-box ML methods, such as deep neural networks and random forests, and outperform logistic regression, while being more interpretable. EBMs prove to be robust. The interpretability of the EBM models reveal surprising insights into the features contributing to risk (e.g., maternal height is the second most important feature for shoulder dystocia) and may have potential for clinical application in the prediction and prevention of serious complications in pregnancy.
@article{bosschieter2023interpretable, title = {Interpretable Predictive Models to Understand Risk Factors for Maternal and Fetal Outcomes}, author = {Bosschieter, Tomas M. and Xu, Zifei and Lan, Hui and Lengerich, Benjamin J. and Nori, Harsha and Painter, Ian and Souter, Vivienne and Caruana, Rich}, year = {2023}, journal = {Journal of Healthcare Informatics Research}, informal_venue = {JHIR}, keywords = {Healthcare, Pregnancy}, } - Data Science with LLMs and Interpretable ModelsAAAI Explainable AI for Science, 2023
Recent years have seen important advances in the building of interpretable models, machine learning models that are designed to be easily understood by humans. In this work, we show that large language models (LLMs) are remarkably good at working with interpretable models, too. In particular, we show that LLMs can describe, interpret, and debug Generalized Additive Models (GAMs). Combining the flexibility of LLMs with the breadth of statistical patterns accurately described by GAMs enables dataset summarization, question answering, and model critique. LLMs can also improve the interaction between domain experts and interpretable models, and generate hypotheses about the underlying phenomenon. We release TalkToEBM as an open-source LLM-GAM interface.
@article{bordt2024data, author = {Bordt, Sebastian and Lengerich, Ben and Nori, Harsha and Caruana, Rich}, title = {Data Science with LLMs and Interpretable Models}, journal = {AAAI Explainable AI for Science}, year = {2023}, informal_venue = {AAAI XAI4Sci}, keywords = {Interpretable, LLMs}, } - Integrating single-cell RNA-seq datasets with substantial batch effectsKarin Hrovatin, Amir Ali Moinfar, Alejandro Tejada Lapuerta, and 4 more authors2023
@article{hrovatin2023evaluation, author = {Hrovatin, Karin and Ali Moinfar, Amir and Lapuerta, Alejandro Tejada and Zappia, Luke and Lengerich, Ben and Kellis, Manolis and Theis, Fabian J.}, title = {Integrating single-cell RNA-seq datasets with substantial batch effects}, year = {2023}, } - LLMs Understand Glass-Box Models, Discover Surprises, and Suggest Repairs2023
We show that large language models (LLMs) are remarkably good at working with interpretable models that decompose complex outcomes into univariate graph-represented components. By adopting a hierarchical approach to reasoning, LLMs can provide comprehensive model-level summaries without ever requiring the entire model to fit in context. This approach enables LLMs to apply their extensive background knowledge to automate common tasks in data science such as detecting anomalies that contradict prior knowledge, describing potential reasons for the anomalies, and suggesting repairs that would remove the anomalies. We use multiple examples in healthcare to demonstrate the utility of these new capabilities of LLMs, with particular emphasis on Generalized Additive Models (GAMs). Finally, we present the package 𝚃𝚊𝚕𝚔𝚃𝚘𝙴𝙱𝙼 as an open-source LLM-GAM interface.
@article{lengerich2023llms, author = {Lengerich, Benjamin J. and Bordt, Sebastian and Nori, Harsha and Nunnally, Mark E. and Aphinyanaphongs, Yin and Kellis, Manolis and Caruana, Rich}, title = {LLMs Understand Glass-Box Models, Discover Surprises, and Suggest Repairs}, year = {2023}, }
2022
- Automated interpretable discovery of heterogeneous treatment effectiveness: A COVID-19 case studyBenjamin J Lengerich, Mark E Nunnally, Yin Aphinyanaphongs, and 2 more authorsJournal of biomedical informatics, 2022
Testing multiple treatments for heterogeneous (varying) effectiveness with respect to many underlying risk factors requires many pairwise tests; we would like to instead automatically discover and visualize patient archetypes and predictors of treatment effectiveness using multitask machine learning. In this paper, we present a method to estimate these heterogeneous treatment effects with an interpretable hierarchical framework that uses additive models to visualize expected treatment benefits as a function of patient factors (identifying personalized treatment benefits) and concurrent treatments (identifying combinatorial treatment benefits). This method achieves state-of-the-art predictive power for COVID-19 in-hospital mortality and interpretable identification of heterogeneous treatment benefits. We first validate this method on the large public MIMIC-IV dataset of ICU patients to test recovery of heterogeneous treatment effects. Next we apply this method to a proprietary dataset of over 3000 patients hospitalized for COVID-19, and find evidence of heterogeneous treatment effectiveness predicted largely by indicators of inflammation and thrombosis risk: patients with few indicators of thrombosis risk benefit most from treatments against inflammation, while patients with few indicators of inflammation risk benefit most from treatments against thrombosis. This approach provides an automated methodology to discover heterogeneous and individualized effectiveness of treatments.
- Dropout as a Regularizer of Interaction EffectsIn Proceedings of the Twenty Fifth International Conference on Artificial Intelligence and Statistics , 2022
We examine Dropout through the perspective of interactions: effects that require multiple variables. Given N variables, there are N \choose k possible sets of k variables (N univariate effects, \mathcalO(N^2) pairwise interactions, \mathcalO(N^3) 3-way interactions); we can thus imagine that models with large representational capacity could be dominated by high-order interactions. In this paper, we show that Dropout contributes a regularization effect which helps neural networks (NNs) explore functions of lower-order interactions before considering functions of higher-order interactions. Dropout imposes this regularization by reducing the effective learning rate of higher-order interactions. As a result, Dropout encourages models to learn lower-order functions of additive components. This understanding of Dropout has implications for choosing Dropout rates: higher Dropout rates should be used when we need stronger regularization against interactions. This perspective also issues caution against using Dropout to measure term salience because Dropout regularizes against high-order interactions. Finally, this view of Dropout as a regularizer of interactions provides insight into the varying effectiveness of Dropout across architectures and datasets. We also compare Dropout to weight decay and early stopping and find that it is difficult to obtain the same regularization with these alternatives.
@inproceedings{lengerich2022dropout, title = {Dropout as a Regularizer of Interaction Effects}, author = {Lengerich, Ben and Xing, Eric P. and Caruana, Rich}, journal = {{Proceedings of the Twenty Fifth International Conference on Artificial Intelligence and Statistics (AISTATS)}}, year = {2022}, informal_venue = {AISTATS}, booktitle = {Proceedings of the Twenty Fifth International Conference on Artificial Intelligence and Statistics}, keywords = {Deep Learning, Theory}, } - Ten quick tips for deep learning in biologyBenjamin D Lee, Anthony Gitter, Casey S Greene, and 17 more authorsPLoS computational biology, 2022
@article{lee2022ten, title = {Ten quick tips for deep learning in biology}, author = {Lee, Benjamin D and Gitter, Anthony and Greene, Casey S and Raschka, Sebastian and Maguire, Finlay and Titus, Alexander J and Kessler, Michael D and Lee, Alexandra J and Chevrette, Marc G and Stewart, Paul Allen and Britto-Borges, Thiago and Cofer, Evan M. Cofer and Yu, Kun-Hsing and Carmona, Juan Jose and Fertig, Elana J. and Kalinin, Alexandr A. and Signal, Brandon and Lengerich, Benjamin J. and Triche, Timothy J. Jr. and Boca, Simina M.}, journal = {PLoS computational biology}, informal_venue = {PLoS CompBio}, volume = {18}, number = {3}, pages = {e1009803}, year = {2022}, publisher = {Public Library of Science San Francisco, CA USA}, keywords = {Deep Learning, Biology, Computational Genomics}, } - Unique insights into risk factors for antepartum stillbirth using explainable AITomas Bosschieter, Zifei Xu, Hui Lan, and 6 more authorsAmerican Journal of Obstetrics & Gynecology, 2022
@article{bosschieter2022smfm2, title = {Unique insights into risk factors for antepartum stillbirth using explainable AI}, author = {Bosschieter, Tomas and Xu, Zifei and Lan, Hui and Lengerich, Benjamin and Nori, Harsha and Sitcov, Kristin and Painter, Ian and Souter, Vivienne and Caruana, Rich}, journal = {American Journal of Obstetrics \& Gynecology}, informal_venue = {SMFM}, volume = {}, number = {}, pages = {}, year = {2022}, publisher = {Elsevier}, keywords = {Healthcare, Pregnancy}, } - Understanding risk factors for shoulder dystocia using interpretable machine learningHui Lan, Zifei Xu, Tomas Bosschieter, and 6 more authorsAmerican Journal of Obstetrics & Gynecology, 2022
@article{lan2022smfm, title = {Understanding risk factors for shoulder dystocia using interpretable machine learning}, author = {Lan, Hui and Xu, Zifei and Bosschieter, Tomas and Lengerich, Benjamin and Nori, Harsha and Sitcov, Kristin and Painter, Ian and Souter, Vivienne and Caruana, Rich}, journal = {American Journal of Obstetrics \& Gynecology}, informal_venue = {SMFM}, volume = {}, number = {}, pages = {}, year = {2022}, publisher = {Elsevier}, keywords = {Healthcare, Pregnancy}, } - Preterm preeclampsia prediction using intelligible machine learningTomas Bosschieter, Zifei Xu, Hui Lan, and 6 more authorsAmerican Journal of Obstetrics & Gynecology, 2022
@article{bosschieter2022smfm, title = {Preterm preeclampsia prediction using intelligible machine learning}, author = {Bosschieter, Tomas and Xu, Zifei and Lan, Hui and Lengerich, Benjamin and Nori, Harsha and Sitcov, Kristin and Painter, Ian and Souter, Vivienne and Caruana, Rich}, journal = {American Journal of Obstetrics \& Gynecology}, informal_venue = {SMFM}, volume = {}, number = {}, pages = {}, year = {2022}, publisher = {Elsevier}, keywords = {Healthcare, Pregnancy}, } - Predicting severe maternal morbidity at admission for delivery using intelligible machine learningZifei Xu, Tomas Bosschieter, Hui Lan, and 6 more authorsAmerican Journal of Obstetrics & Gynecology, 2022
@article{xu2022smfm, title = {Predicting severe maternal morbidity at admission for delivery using intelligible machine learning}, author = {Xu, Zifei and Bosschieter, Tomas and Lan, Hui and Lengerich, Benjamin and Nori, Harsha and Sitcov, Kristin and Painter, Ian and Souter, Vivienne and Caruana, Rich}, journal = {American Journal of Obstetrics \& Gynecology}, informal_venue = {SMFM}, volume = {}, number = {}, pages = {}, year = {2022}, publisher = {Elsevier}, keywords = {Healthcare, Pregnancy}, }
2021
- Neural Additive Models: Interpretable Machine Learning with Neural NetsRishabh Agarwal, Levi Melnick, Nicholas Frosst, and 4 more authorsAdvances in Neural Information Processing Systems, 2021
Deep neural networks (DNNs) are powerful black-box predictors that have achieved impressive performance on a wide variety of tasks. However, their accuracy comes at the cost of intelligibility: it is usually unclear how they make their decisions. This hinders their applicability to high stakes decision-making domains such as healthcare. We propose Neural Additive Models (NAMs) which combine some of the expressivity of DNNs with the inherent intelligibility of generalized additive models. NAMs learn a linear combination of neural networks that each attend to a single input feature. These networks are trained jointly and can learn arbitrarily complex relationships between their input feature and the output. Our experiments on regression and classification datasets show that NAMs are more accurate than widely used intelligible models such as logistic regression and shallow decision trees. They perform similarly to existing state-of-the-art generalized additive models in accuracy, but are more flexible because they are based on neural nets instead of boosted trees. To demonstrate this, we show how NAMs can be used for multitask learning on synthetic data and on the COMPAS recidivism data due to their composability, and demonstrate that the differentiability of NAMs allows them to train more complex interpretable models for COVID-19.
@article{agarwal2022neural, title = {Neural Additive Models: Interpretable Machine Learning with Neural Nets}, author = {Agarwal, Rishabh and Melnick, Levi and Frosst, Nicholas and Zhang, Xuezhou and Lengerich, Ben and Caruana, Rich and Hinton, Geoffrey E}, journal = {Advances in Neural Information Processing Systems}, volume = {34}, pages = {4699--4711}, year = {2021}, informal_venue = {NeurIPS}, keywords = {Interpretable, Deep Learning, Generalized Additive Models}, } - How Interpretable and Trustworthy are GAMs?Chun-Hao Chang, Sarah Tan, Ben Lengerich, and 2 more authorsIn Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 2021
Generalized additive models (GAMs) have become a leading model class for interpretable machine learning. However, there are many algorithms for training GAMs, and these can learn different or even contradictory models, while being equally accurate. Which GAM should we trust? In this paper, we quantitatively and qualitatively investigate a variety of GAM algorithms on real and simulated datasets. We find that GAMs with high feature sparsity (only using a few variables to make predictions) can miss patterns in the data and be unfair to rare subpopulations. Our results suggest that inductive bias plays a crucial role in what interpretable models learn and that tree-based GAMs represent the best balance of sparsity, fidelity and accuracy and thus appear to be the most trustworthy GAM models.
@inproceedings{chang2021how, title = {How Interpretable and Trustworthy are GAMs?}, author = {Chang, Chun-Hao and Tan, Sarah and Lengerich, Ben and Goldenberg, Anna and Caruana, Rich}, journal = {Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discovery \& Data Mining}, booktitle = {Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discovery \& Data Mining}, year = {2021}, informal_venue = {KDD}, keywords = {Interpretable, Generalized Additive Models}, } - Length of labor and severe maternal morbidity in the NTSV populationAmerican Journal of Obstetrics & Gynecology, 2021
@article{lengerich2021length, title = {Length of labor and severe maternal morbidity in the NTSV population}, author = {Lengerich, Benjamin J. and Caruana, Rich and Weeks, William B and Painter, Ian and Spencer, Sydney and Sitcov, Kristin and Daly, Colleen and Souter, Vivienne}, journal = {American Journal of Obstetrics \& Gynecology}, volume = {224}, number = {2}, pages = {S33}, year = {2021}, publisher = {Elsevier}, keywords = {Healthcare, Pregnancy}, } - Insights into severe maternal morbidity in the NTSV populationAmerican Journal of Obstetrics & Gynecology, 2021
@article{lengerich2021insights, title = {Insights into severe maternal morbidity in the NTSV population}, author = {Lengerich, Benjamin J. and Caruana, Rich and Weeks, William B and Painter, Ian and Spencer, Sydney and Sitcov, Kristin and Daly, Colleen and Souter, Vivienne}, journal = {American Journal of Obstetrics \& Gynecology}, volume = {224}, number = {2}, pages = {S629--S630}, year = {2021}, publisher = {Elsevier}, keywords = {Healthcare, Pregnancy}, } - Data-Driven Patterns in Protective Effects of Ibuprofen and Ketorolac on Hospitalized Covid-19 PatientsRich Caruana, Benjamin Lengerich, and Yin AphinyanaphongsIn , 2021
The impact of nonsteroidal anti-inflammatory drugs (NSAIDs) on patients with Covid-19 has been unclear. A major reason for this uncertainty is the confounding between treatments, patient comorbidities, and illness severity. Here, we perform an observational analysis of over 3000 patients hospitalized for Covid-19 in a New York hospital system to identify the relationship between in-patient treatment with Ibuprofen or Ketorolac and mortality. Our analysis finds evidence consitent with a protective effect for Ibuprofen and Ketorolac, with evidence stronger for a protective effect of Ketorolac than for a protective effect of Ibuprofen.
@inproceedings{caruana2021data, title = {Data-Driven Patterns in Protective Effects of Ibuprofen and Ketorolac on Hospitalized Covid-19 Patients}, author = {Caruana, Rich and Lengerich, Benjamin and Aphinyanaphongs, Yin}, journal = {American Medical Informatics Association (AMIA) Annual Symposium}, year = {2021}, keywords = {Healthcare, Covid-19}, }
2020
- Purifying Interaction Effects with the Functional ANOVA: An Efficient Algorithm for Recovering Identifiable Additive ModelsBen Lengerich, Sarah Tan, Chun-Hao Chang, and 2 more authorsIn Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics (AISTATS) , 26–28 aug 2020
Models which estimate main effects of individual variables alongside interaction effects have an identifiability challenge: effects can be freely moved between main effects and interaction effects without changing the model prediction. This is a critical problem for interpretability because it permits “contradictory" models to represent the same function. To solve this problem, we propose pure interaction effects: variance in the outcome which cannot be represented by any subset of features. This definition has an equivalence with the Functional ANOVA decomposition. To compute this decomposition, we present a fast, exact algorithm that transforms any piecewise-constant function (such as a tree-based model) into a purified, canonical representation. We apply this algorithm to Generalized Additive Models with interactions trained on several datasets and show large disparity, including contradictions, between the apparent and the purified effects. These results underscore the need to specify data distributions and ensure identifiability before interpreting model parameters.
@inproceedings{lengerich2020purifying, title = {Purifying Interaction Effects with the Functional ANOVA: An Efficient Algorithm for Recovering Identifiable Additive Models}, author = {Lengerich, Ben and Tan, Sarah and Chang, Chun-Hao and Hooker, Giles and Caruana, Rich}, booktitle = {Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics (AISTATS)}, pages = {2402--2412}, year = {2020}, editor = {Chiappa, Silvia and Calandra, Roberto}, volume = {108}, series = {Proceedings of Machine Learning Research}, month = {26--28 Aug}, publisher = {PMLR}, informal_venue = {AISTATS}, keywords = {Interpretable, Generalized Additive Models, Theory}, }
2019
- Learning Sample-Specific Models with Low-Rank Personalized RegressionIn Advances in Neural Information Processing Systems (NeurIPS) , 26–28 aug 2019
Modern applications of machine learning (ML) deal with increasingly heterogeneous datasets comprised of data collected from overlapping latent subpopulations. As a result, traditional models trained over large datasets may fail to recognize highly predictive localized effects in favour of weakly predictive global patterns. This is a problem because localized effects are critical to developing individualized policies and treatment plans in applications ranging from precision medicine to advertising. To address this challenge, we propose to estimate sample-specific models that tailor inference and prediction at the individual level. In contrast to classical ML models that estimate a single, complex model (or only a few complex models), our approach produces a model personalized to each sample. These sample-specific models can be studied to understand subgroup dynamics that go beyond coarse-grained class labels. Crucially, our approach does not assume that relationships between samples (e.g. a similarity network) are known a priori. Instead, we use unmodeled covariates to learn a latent distance metric over the samples. We apply this approach to financial, biomedical, and electoral data as well as simulated data and show that sample-specific models provide fine-grained interpretations of complicated phenomena without sacrificing predictive accuracy compared to state-of-the-art models such as deep neural networks.
@inproceedings{lengerich2019learning, title = {Learning Sample-Specific Models with Low-Rank Personalized Regression}, author = {Lengerich, Benjamin J. and Aragam, Bryon and Xing, Eric P.}, journal = {Advances in Neural Information Processing Systems (NeurIPS)}, booktitle = {Advances in Neural Information Processing Systems (NeurIPS)}, year = {2019}, informal_venue = {NeurIPS}, keywords = {Interpretable, Contextualized}, }
2018
- Precision Lasso: Accounting for Correlations and Linear Dependencies in High-Dimensional Genomic DataBioinformatics, 26–28 aug 2018
Association studies to discover links between genetic markers and phenotypes are central to bioinformatics. Methods of regularized regression, such as variants of the Lasso, are popular for this task. Despite the good predictive performance of these methods in the average case, they suffer from unstable selections of correlated variables and inconsistent selections of linearly dependent variables. Unfortunately, as we demonstrate empirically, such problematic situations of correlated and linearly dependent variables often exist in genomic datasets and lead to under-performance of classical methods of variable selection. To address these challenges, we propose the Precision Lasso. Precision Lasso is a Lasso variant that promotes sparse variable selection by regularization governed by the covariance and inverse covariance matrices of explanatory variables. We illustrate its capacity for stable and consistent variable selection in simulated data with highly correlated and linearly dependent variables. We then demonstrate the effectiveness of the Precision Lasso to select meaningful variables from transcriptomic profiles of breast cancer patients. Our results indicate that in settings with correlated and linearly dependent variables, the Precision Lasso outperforms popular methods of variable selection such as the Lasso, the Elastic Net and Minimax Concave Penalty (MCP) regression.
@article{wang2018precision, title = {Precision Lasso: Accounting for Correlations and Linear Dependencies in High-Dimensional Genomic Data}, author = {Wang, Haohan and Lengerich, Benjamin J. and Aragam, Bryon and Xing, Eric P}, journal = {Bioinformatics}, volume = {35}, number = {7}, pages = {1181--1187}, year = {2018}, informal_venue = {Bioinformatics}, publisher = {Oxford University Press}, keywords = {Statistical Genetics, Genomics}, } - Retrofitting Distributional Embeddings to Knowledge Graphs with Functional RelationsBenjamin J. Lengerich, Andrew Maas, and Christopher PottsIn Proceedings of the 27th International Conference on Computational Linguistics (COLING) , 26–28 aug 2018
Knowledge graphs are a versatile framework to encode richly structured data relationships, but it can be challenging to combine these graphs with unstructured data. Methods for retrofitting pre-trained entity representations to the structure of a knowledge graph typically assume that entities are embedded in a connected space and that relations imply similarity. However, useful knowledge graphs often contain diverse entities and relations (with potentially disjoint underlying corpora) which do not accord with these assumptions. To overcome these limitations, we present Functional Retrofitting, a framework that generalizes current retrofitting methods by explicitly modeling pairwise relations. Our framework can directly incorporate a variety of pairwise penalty functions previously developed for knowledge graph completion. Further, it allows users to encode, learn, and extract information about relation semantics. We present both linear and neural instantiations of the framework. Functional Retrofitting significantly outperforms existing retrofitting methods on complex knowledge graphs and loses no accuracy on simpler graphs (in which relations do imply similarity). Finally, we demonstrate the utility of the framework by predicting new drug–disease treatment pairs in a large, complex health knowledge graph.
@inproceedings{lengerich2018retrofitting, title = {Retrofitting Distributional Embeddings to Knowledge Graphs with Functional Relations}, author = {Lengerich, Benjamin J. and Maas, Andrew and Potts, Christopher}, journal = {Proceedings of the 27th International Conference on Computational Linguistics (COLING)}, booktitle = {Proceedings of the 27th International Conference on Computational Linguistics (COLING)}, year = {2018}, informal_venue = {COLING}, keywords = {Natural Language Processing, Knowledge Graphs}, } - Personalized Regression Enables Sample-specific Pan-cancer AnalysisBioinformatics, 26–28 aug 2018
In many applications, inter-sample heterogeneity is crucial to understanding the complex biological processes under study. For example, in genomic analysis of cancers, each patient in a cohort may have a different driver mutation, making it difficult or impossible to identify causal mutations from an averaged view of the entire cohort. Unfortunately, many traditional methods for genomic analysis seek to estimate a single model which is shared by all samples in a population, ignoring this inter-sample heterogeneity entirely. In order to better understand patient heterogeneity, it is necessary to develop practical, personalized statistical models. To uncover this inter-sample heterogeneity, we propose a novel regularizer for achieving patient-specific personalized estimation. This regularizer operates by learning two latent distance metrics—one between personalized parameters and one between clinical covariates—and attempting to match the induced distances as closely as possible. Crucially, we do not assume these distance metrics are already known. Instead, we allow the data to dictate the structure of these latent distance metrics. Finally, we apply our method to learn patient-specific, interpretable models for a pan-cancer gene expression dataset containing samples from more than 30 distinct cancer types and find strong evidence of personalization effects between cancer types as well as between individuals. Our analysis uncovers sample-specific aberrations that are overlooked by population-level methods, suggesting a promising new path for precision analysis of complex diseases such as cancer.
@article{lengerich2018personalized, author = {Lengerich, Benjamin J. and Aragam, Bryon and Xing, Eric P}, title = {Personalized Regression Enables Sample-specific Pan-cancer Analysis}, journal = {Bioinformatics}, volume = {34}, number = {13}, pages = {i178-i186}, year = {2018}, informal_venue = {ISMB}, doi = {10.1093/bioinformatics/bty250}, url = {http://dx.doi.org/10.1093/bioinformatics/bty250}, eprint = {/oup/backfile/content_public/journal/bioinformatics/34/13/10.1093_bioinformatics_bty250/1/bty250.pdf}, keywords = {Interpretable, Contextualized, Statistical Genetics, Genomics, Cancer}, } - Opportunities and Obstacles for Deep Learning in Biology and MedicineTravers Ching, Daniel S. Himmelstein, Brett K. Beaulieu-Jones, and 33 more authorsJournal of The Royal Society Interface, 26–28 aug 2018
Deep learning describes a class of machine learning algorithms that are capable of combining raw inputs into layers of intermediate features. These algorithms have recently shown impressive results across a variety of domains. Biology and medicine are data-rich disciplines, but the data are complex and often ill-understood. Hence, deep learning techniques may be particularly well suited to solve problems of these fields. We examine applications of deep learning to a variety of biomedical problems—patient classification, fundamental biological processes and treatment of patients—and discuss whether deep learning will be able to transform these tasks or if the biomedical sphere poses unique challenges. Following from an extensive literature review, we find that deep learning has yet to revolutionize biomedicine or definitively resolve any of the most pressing challenges in the field, but promising advances have been made on the prior state of the art. Even though improvements over previous baselines have been modest in general, the recent progress indicates that deep learning methods will provide valuable means for speeding up or aiding human investigation. Though progress has been made linking a specific neural network’s prediction to input features, understanding how users should interpret these models to make testable hypotheses about the system under study remains an open challenge. Furthermore, the limited amount of labelled data for training presents problems in some domains, as do legal and privacy constraints on work with sensitive health records. Nonetheless, we foresee deep learning enabling changes at both bench and bedside with the potential to transform several areas of biology and medicine.
@article{ching2018opportunities, author = {Ching, Travers and Himmelstein, Daniel S. and Beaulieu-Jones, Brett K. and Kalinin, Alexandr A. and Do, Brian T. and Way, Gregory P. and Ferrero, Enrico and Agapow, Paul-Michael and Zietz, Michael and Hoffman, Michael M. and Xie, Wei and Rosen, Gail L. and Lengerich, Benjamin J. and Israeli, Johnny and Lanchantin, Jack and Woloszynek, Stephen and Carpenter, Anne E. and Shrikumar, Avanti and Xu, Jinbo and Cofer, Evan M. and Lavender, Christopher A. and Turaga, Srinivas C. and Alexandari, Amr M. and Lu, Zhiyong and Harris, David J. and DeCaprio, Dave and Qi, Yanjun and Kundaje, Anshul and Peng, Yifan and Wiley, Laura K. and Segler, Marwin H. S. and Boca, Simina M. and Swamidass, S. Joshua and Huang, Austin and Gitter, Anthony and Greene, Casey S.}, title = {Opportunities and Obstacles for Deep Learning in Biology and Medicine}, volume = {15}, number = {141}, year = {2018}, doi = {10.1098/rsif.2017.0387}, publisher = {The Royal Society}, informal_venue = {JRSI}, issn = {1742-5689}, url = {http://rsif.royalsocietypublishing.org/content/15/141/20170387}, eprint = {http://rsif.royalsocietypublishing.org/content/15/141/20170387.full.pdf}, journal = {Journal of The Royal Society Interface}, keywords = {Deep Learning, Biology, Computational Genomics}, }