- title: 'Preface' volume: 179 URL: https://proceedings.mlr.press/v179/johansson22a.html PDF: https://proceedings.mlr.press/v179/johansson22a/johansson22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-johansson22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 1-3 id: johansson22a issued: date-parts: - 2022 - 8 - 30 firstpage: 1 lastpage: 3 published: 2022-08-30 00:00:00 +0000 - title: 'Tutorial for using conformal prediction in KNIME' abstract: 'KNIME is an end-to-end software platform for data science with an open source analytics platform for creating solutions and a commercial server solution for productionization. Redfield have previously developed nodes for conformal classification in KNIME. We introduce an extended conformal prediction package with added support for conformal regression. The conformal prediction package include class-conditional conformal classification, conformal regression and normalized conformal regression. The updated package also includes several new and updated nodes that focus on ease-of-use. This paper provide an introduction to various use cases for both simplified and advanced use as well as experiments to prove validity and showcase functionality. All examples are publicly available and the package is available through KNIME’s official software channels. ' volume: 179 URL: https://proceedings.mlr.press/v179/lofstrom22a.html PDF: https://proceedings.mlr.press/v179/lofstrom22a/lofstrom22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-lofstrom22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Tuwe family: Löfström - given: Artem family: Ryasik - given: Ulf family: Johansson editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 4-23 id: lofstrom22a issued: date-parts: - 2022 - 8 - 30 firstpage: 4 lastpage: 23 published: 2022-08-30 00:00:00 +0000 - title: 'crepes: a Python Package for Generating Conformal Regressors and Predictive Systems' abstract: 'The recently released Python package crepes can be used to generate both conformal regressors, which transform point predictions into prediction intervals for specified levels of confidence, and conformal predictive systems, which transform the point predictions into cumulative distribution functions (conformal predictive distributions). The \texttt{crepes} package implements standard, normalized and Mondrian conformal regressors and predictive systems, and is completely model-agnostic, using only the residuals for the calibration instances, possibly together with difficulty estimates and Mondrian categories as input, when forming the conformal regressors and predictive systems. This allows the user to easily incorporate and evaluate novel difficulty estimates and ways of forming Mondrian categories, as well as combinations thereof. Examples from using the package are given, illustrating how to incorporate some standard options for difficulty estimation, forming Mondrian categories and the use of out-of-bag predictions for calibration, through helper functions defined in a separate module, called \texttt{crepes.fillings}. The relation to other software packages for conformal regression is also pointed out.' volume: 179 URL: https://proceedings.mlr.press/v179/bostrom22a.html PDF: https://proceedings.mlr.press/v179/bostrom22a/bostrom22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-bostrom22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Henrik family: Boström editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 24-41 id: bostrom22a issued: date-parts: - 2022 - 8 - 30 firstpage: 24 lastpage: 41 published: 2022-08-30 00:00:00 +0000 - title: 'Assessing Explanation Quality by Venn Prediction' abstract: ' Rules output by explainable machine learning techniques naturally come with a degree of uncertainty, as the complex functionality of the underlying black-box model often can be difficult to approximate by a single, interpretable rule. However, the uncertainty of these approximations is not properly quantified by current explanatory techniques. The use of Venn prediction is here proposed and investigated as a means to quantify the uncertainty of the explanations and thereby also allow for competing explanation techniques to be evaluated with respect to their relative uncertainty. A number of metrics of rule explanation quality based on uncertainty are proposed and discussed, including metrics that capture the tendency of the explanations to predict the correct outcome of a black-box model on new instances, how informative (tight) the produced intervals are, and how certain a rule is when predicting one class. An empirical investigation is presented, in which explanations produced by the state-of-the-art technique Anchors are compared to explanatory rules obtained from association rule mining. The results suggest that the association rule mining approach may provide explanations with less uncertainty towards the correct label, as predicted by the black-box model, compared to Anchors. The results also show that the explanatory rules obtained through association rule mining result in tighter intervals and are closer to either one or zero compared to Anchors, i.e., they are more certain towards a specific class label. ' volume: 179 URL: https://proceedings.mlr.press/v179/alkhatib22a.html PDF: https://proceedings.mlr.press/v179/alkhatib22a/alkhatib22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-alkhatib22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Amr family: Alkhatib - given: Henrik family: Boström - given: Ulf family: Johansson editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 42-54 id: alkhatib22a issued: date-parts: - 2022 - 8 - 30 firstpage: 42 lastpage: 54 published: 2022-08-30 00:00:00 +0000 - title: ' Calibration of Natural Language Understanding Models with Venn–ABERS Predictors' abstract: 'Transformers, currently the state-of-the-art in natural language understanding (NLU) tasks, are prone to generate uncalibrated predictions or extreme probabilities, making the process of taking different decisions based on their output relatively difficult. In this paper we propose to build several inductive Venn–ABERS predictors (IVAP), which are guaranteed to be well calibrated under minimal assumptions, based on a selection of pre-trained transformers. We test their performance over a set of diverse NLU tasks and show that they are capable of producing well-calibrated probabilistic predictions that are uniformly spread over the [0,1] interval – all while retaining the original model�s predictive accuracy. ' volume: 179 URL: https://proceedings.mlr.press/v179/giovannotti22a.html PDF: https://proceedings.mlr.press/v179/giovannotti22a/giovannotti22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-giovannotti22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Patrizio family: Giovannotti editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 55-71 id: giovannotti22a issued: date-parts: - 2022 - 8 - 30 firstpage: 55 lastpage: 71 published: 2022-08-30 00:00:00 +0000 - title: 'Well-Calibrated Rule Extractors' abstract: ' While explainability is widely considered necessary for trustworthy predictive models, most explanation modules give only a limited understanding of the reasoning behind the predictions. In pedagogical rule extraction, an opaque model is approximated with a transparent model induced using original training instances, but with the predictions from the opaque model as targets. The result is an interpretable model revealing the exact reasoning used for every possible prediction. The pedagogical approach can be applied to any opaque model and use any learning algorithm producing transparent models as the actual rule extractor. Unfortunately, even if the extracted model is induced to mimic the opaque, test set fidelity may still be poor, thus clearly limiting the value of using the extracted model for explanations and analyses. In this paper, it is suggested to alleviate this problem by extracting probabilistic predictors with well-calibrated fitness estimates. For the calibration, Venn-Abers with its unique validity guarantees, is employed. Using a setup where decision trees are extracted from MLP neural networks, the suggested approach is first demonstrated in detail on one real-world data set. After that, a large-scale empirical evaluation using 25 publicly available benchmark data sets is presented. The results show that the method indeed extracts interpretable models with well-calibrated fitness estimates, i.e., the extracted model can be used for explaining the opaque. Specifically, in the setup used, every leaf in a decision tree contains a label and a well-calibrated probability interval for the fidelity. Consequently, a user could, in addition to obtaining explanations of individual predictions, find the parts of feature space where the decision tree is a good approximation of the MLP and not. In fact, using the sizes of the probability intervals, the models also provide an indication of how certain individual fitness estimates are. ' volume: 179 URL: https://proceedings.mlr.press/v179/johansson22b.html PDF: https://proceedings.mlr.press/v179/johansson22b/johansson22b.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-johansson22b.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Ulf family: Johansson - given: Tuwe family: Löfström - given: Niclas family: Ståhl editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 72-91 id: johansson22b issued: date-parts: - 2022 - 8 - 30 firstpage: 72 lastpage: 91 published: 2022-08-30 00:00:00 +0000 - title: 'Conformal prediction of small-molecule drug resistance in cancer cell lines' abstract: 'Drug design is a critical step in the drug discovery process, where promising drug molecules are engineered to be later evaluated preclinically and perhaps clinically. Phenotypic drug design has again gained traction. Cancer cell lines, a frequently adopted {\it in vitro} model for phenotype drug design, can be used to evaluate the drug resistance level (lack of inhibitory activity, for example) of a large number of molecules, and discard those that are the least likely to become drug candidates. By reusing these datasets, supervised learning models have been built to predict drug resistance on cancer cell lines. Usually, these methods have assigned reliability to the whole model rather than reliability to individual predictions (molecules). In problems such as drug design, accurately achieving the latter would revolutionize decision making. Conformal prediction is a model-agnostic method to assign reliability to each model prediction. In this study, we investigated the impact of conformal prediction on the prediction of inhibitory activity of molecules on a given cancer cell line. This analysis was carried out in each of the 60 cell lines from the NCI-60 panel to understand the variability of the results across cancer types. We also discussed the implications of predicting the molecules considered most potent. In addition, we investigated how the further subdivision of the training set to build conformal prediction models may affect the results obtained. Overall, we observed that those molecules deemed most reliable by conformal prediction are substantially better predicted than those that are not. This suggest that such computational tools are promising to guide phenotypic drug design. ' volume: 179 URL: https://proceedings.mlr.press/v179/hernandez-hernandez22a.html PDF: https://proceedings.mlr.press/v179/hernandez-hernandez22a/hernandez-hernandez22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-hernandez-hernandez22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Saiveth family: Hernández-Hernández - given: Sachin family: Vishwakarma - given: Pedro family: Ballester editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 92-108 id: hernandez-hernandez22a issued: date-parts: - 2022 - 8 - 30 firstpage: 92 lastpage: 108 published: 2022-08-30 00:00:00 +0000 - title: 'Uncertainty Estimation for Single-cell Label Transfer' abstract: 'Single-cell gene expression matrices require a cell type label for each cell for downstream analysis. A cell type label refers to a heterogeneous group to which a cell belongs. Machine learning algorithms that aim to automate the assignment of cell type labels train on reference datasets for which cell type labels are already defined. However, these methods are prone to error due to possible preprocessing errors and the dynamic nature of cellular states. Therefore, it is essential to measure the uncertainty associated with classifications. Here, we hypothesize that conformal prediction may provide a principled approach for this. We examine inductive conformal classifiers (ICPs) on the task of single-cell label transfer. ICPs lead to well-calibrated models that quantify uncertainties well. Results are motivating, and the uncertainties are intuitive and easy to interpret. We also consider a confidence-credibility conformal predictions setup that accurately predicts single labels with the desired error level. Such a model can also reject the classification of cell types unobserved in the reference dataset. However, the presence of unknown cell types violates the underlying assumption of a conformal predictor and is highly dependent on the quality of batch correction. We envision more work in detecting unknown cell types and using conformal predictions to evaluate batch correction methods. ' volume: 179 URL: https://proceedings.mlr.press/v179/khatri22a.html PDF: https://proceedings.mlr.press/v179/khatri22a/khatri22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-khatri22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Robin family: Khatri - given: Stefan family: Bonn editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 109-128 id: khatri22a issued: date-parts: - 2022 - 8 - 30 firstpage: 109 lastpage: 128 published: 2022-08-30 00:00:00 +0000 - title: 'Conformal prediction for hypersonic flight vehicle classification' abstract: 'This paper introduces a probabilistic guaranteed prediction method for trajectory data of the hypersonic flight vehicle classification problem. This paper devoted two problems: (1) hypersonic flight vehicle trajectory classification algorithm using functional data analysis method, and (2) a distributions-free uncertainty quantity for the classification results applying conformal prediction methodology. Our approach provides explicit finite-sample guarantees for any data set by using functional data analysis methods, which map the original data into feature space. The distribution-free uncertainty quantity results for the label of new objects include two indications, such as confidence and credibility respectively. Lastly, the proposed method aims to communicate instance-wise uncertainty under the probabilistic guaranteed and generate a prediction set at a user-specified confidence level for the hypersonic flight vehicle classification problem. ' volume: 179 URL: https://proceedings.mlr.press/v179/xi22a.html PDF: https://proceedings.mlr.press/v179/xi22a/xi22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-xi22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Zepu family: Xi - given: Xuebin family: Zhuang - given: Hongbo family: Chen editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 118-206 id: xi22a issued: date-parts: - 2022 - 8 - 30 firstpage: 118 lastpage: 206 published: 2022-08-30 00:00:00 +0000 - title: 'Cough-based COVID-19 detection with audio quality clustering and confidence measure based learning' abstract: ' COVID-19 cough classification has rapidly become a promising research avenue as an accessible and low-cost screening alternative, needing only a smartphone to collect and process cough samples. However, audio processing of recordings collected in uncontrolled environments and prediction confidence are key challenges that need to be addressed before cough-screening could be widely accepted as a trusted testing method. Therefore, we propose a novel approach for cough event detection that identifies {\it cough clusters} instead of individual coughs, significantly reducing onset detection’s usual hypersensitivity to energy fluctuations between cough phases. By using this technique to improve training sample quality and quantity by +200%, we improve Machine Learning performance on the minority COVID-19 class by up to 20%, achieving up to +47% precision and +15% recall compared to the dataset baseline. We propose a novel, class-agnostic Conformal Prediction non-conformity measure which takes the cough sample quality into account to counteract the variance caused by limiting segmentation to just the training set. Our Conformal Prediction model introduces uncertainty quantification to COVID-19 cough classification and achieves an additional 34% improvement to precision and recall.' volume: 179 URL: https://proceedings.mlr.press/v179/ashby22a.html PDF: https://proceedings.mlr.press/v179/ashby22a/ashby22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-ashby22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Alice E. family: Ashby - given: Julia A. family: Meister - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Werner family: Gentzke editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 129-148 id: ashby22a issued: date-parts: - 2022 - 8 - 30 firstpage: 129 lastpage: 148 published: 2022-08-30 00:00:00 +0000 - title: 'Online Portfolio Hedging with the Weak Aggregating Algorithm' abstract: 'In this paper we apply the Weak Aggregating Algorithm to find optimal risk management strategies for financial Market Makers (MMs). Here risk is caused by the market exposure. It is effectively represented by the MM’s overall net {\it position}, which is the aggregation of all the {\it buy} and {\it sell} trades carried out by the MM�s clients at a given point in time. So-called {\it hedging} strategies are used by MMs to manage their risk and reduce market exposure. In essence, the MM actively places trades in order to reduce its overall net position, keeping it within some predefined bounds and as neutral (or flat) as possible. A flatter net position allows the MM to counter any unfavourable price movements which could otherwise incur a significant loss. We apply the Weak Aggregating Algorithm (WAA) to hedging strategies, which are treated as the experts. We combine their hedging decisions with the goal of reducing portfolio risk and maximising profitability, whilst also attempting to smooth out significant drawdowns. We develop a variation of the WAA using discounting and evaluate the WAA on a subset of real life client risk data in three commonly traded Foreign Exchange (FX) currency symbols: EUR/USD, EUR/GBP and GBP/USD. The results show how varying loss parameters and application of discount factors can enable the WAA to give combinations of hedging strategies that can significantly improve profitability and reduce drawdowns as compared to the benchmark of not hedging.' volume: 179 URL: https://proceedings.mlr.press/v179/al-baghdadi22a.html PDF: https://proceedings.mlr.press/v179/al-baghdadi22a/al-baghdadi22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-al-baghdadi22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Najim family: Al-Baghdadi - given: Yuri family: Kalnishkan - given: David family: Lindsay - given: Siân family: Lindsay editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 149-168 id: al-baghdadi22a issued: date-parts: - 2022 - 8 - 30 firstpage: 149 lastpage: 168 published: 2022-08-30 00:00:00 +0000 - title: 'Robust Gas Demand Forecasting With Conformal Prediction' abstract: ' Predicting the future trends of customer gas demand as precisely as possible is vital for securing the supply chain from production to distribution. The operations at Air Liquide require the predictions of a Machine Learning forecaster to be coupled with rigorous Uncertainty Quantification (UQ), building trustworthy and informative prediction intervals. To address these industrial needs, we propose to apply Conformal Prediction (CP), a framework that can provide probabilistic guarantees for any underlying predictive model. The problem is formulated as time series forecasting, which may counter the CP hypothesis of data exchangeability. Nevertheless, our experiments show that CP methods enhance the predictive coverage of the tested UQ approaches. We also test EnbPI, a conformal method designed specifically for time series, and propose a locally adaptive variant. To carry out our experiments with prediction intervals using multiple regression models, we introduce our new python library PUNCC and a novel dataset (around 10k observations) provided by Air Liquide which leverages over 7 years of data of weekly gas consumption.' volume: 179 URL: https://proceedings.mlr.press/v179/mendil22a.html PDF: https://proceedings.mlr.press/v179/mendil22a/mendil22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-mendil22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Mouhcine family: Mendil - given: Luca family: Mossina - given: Marc family: Nabhan - given: Kevin family: Pasini editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 169-187 id: mendil22a issued: date-parts: - 2022 - 8 - 30 firstpage: 169 lastpage: 187 published: 2022-08-30 00:00:00 +0000 - title: 'Conformal testing: binary case with Markov alternatives' abstract: 'We continue study of conformal testing in binary model situations. In this paper we consider Markov alternatives to the null hypothesis of exchangeability. We propose two new classes of conformal test martingales; one class is statistically efficient in our experiments, and the other class partially sacrifices statistical efficiency to gain simplicity and computational efficiency. ' volume: 179 URL: https://proceedings.mlr.press/v179/vovk22a.html PDF: https://proceedings.mlr.press/v179/vovk22a/vovk22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-vovk22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Vladimir family: Vovk - given: Ilia family: Nouretdinov - given: Alex family: Gammerman editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 207-218 id: vovk22a issued: date-parts: - 2022 - 8 - 30 firstpage: 207 lastpage: 218 published: 2022-08-30 00:00:00 +0000 - title: 'A Betting Function for addressing Concept Drift with Conformal Martingales' abstract: 'An important issue that appears when using Conformal Martingales (CM) for detecting Concept Drift (CD), is that martingale values get very close to zero when the data generating mechanism remains the same for a large number of instances. In such cases, the martingale takes a long time to recover, resulting in detection delays or even totally failing to detect the occurrence of a CD. To address this issue we propose a new betting function we call Cautious, that avoids betting when there is no evidence that any change is taking place, therefore preventing the continuous reduction of the martingale value. The proposed betting function can be built on top of any existing betting function to mitigate the aforementioned problem. In this work, we combine it with the kernel and histogram betting functions and compare its performance with that of the two original betting functions as well as that of existing methods for addressing CD on five datasets. ' volume: 179 URL: https://proceedings.mlr.press/v179/eliades22a.html PDF: https://proceedings.mlr.press/v179/eliades22a/eliades22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-eliades22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Charalambos family: Eliades - given: Harris family: Papadopoulos editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 219-238 id: eliades22a issued: date-parts: - 2022 - 8 - 30 firstpage: 219 lastpage: 238 published: 2022-08-30 00:00:00 +0000 - title: 'On efficiency of Learning Under Privileged Information' abstract: 'The paradigm of Learning Under Privileged Information (LUPI) was used in various practical applications, including its combination with Conformal Prediction (CP) framework. In this note, we discuss possible sources and limitations of its efficiency. We try to argue that accuracy improvement coming from using privileged information is not occasional. For this goal, we consider some minimalistic models of LUPI where the contribution of the privileged information appears in its noise-free essence. Then, we discuss connection of LUPI paradigm and CP framework in relation with the models.' volume: 179 URL: https://proceedings.mlr.press/v179/nouretdinov22a.html PDF: https://proceedings.mlr.press/v179/nouretdinov22a/nouretdinov22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-nouretdinov22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Ilia family: Nouretdinov editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 239-252 id: nouretdinov22a issued: date-parts: - 2022 - 8 - 30 firstpage: 239 lastpage: 252 published: 2022-08-30 00:00:00 +0000 - title: 'Tensor-Train Kernel Learning for Gaussian Processes' abstract: 'We propose a new kernel learning approach based on efficient low-rank tensor compression for Gaussian process (GP) regression. The central idea is to compose a low-rank function represented in a hierarchical tensor format with a GP covariance function. Compared to similar deep neural network architectures, this approach facilitates to learn significantly more expressive features at lower computational costs as illustrated in the examples. Additionally, over-fitting is avoided with this compositional model by taking advantage of its inherent regularisation properties. Estimates of the generalisation error are compared to five baseline models on three synthetic and six real-world data sets. The experimental results show that the incorporated tensor network enables a highly accurate GP regression with a comparatively low number of trainable parameters. The observed performance is clearly superior (usually by an order of magnitude in mean squared error) to all examined standard models, in particular to deep neural networks with more than 1000 times as many parameters. ' volume: 179 URL: https://proceedings.mlr.press/v179/kirstein22a.html PDF: https://proceedings.mlr.press/v179/kirstein22a/kirstein22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-kirstein22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Max family: Kirstein - given: David family: Sommer - given: Martin family: Eigel editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 253-272 id: kirstein22a issued: date-parts: - 2022 - 8 - 30 firstpage: 253 lastpage: 272 published: 2022-08-30 00:00:00 +0000 - title: 'Pruning neural networks for inductive conformal prediction' abstract: 'Neural network pruning techniques are used to prune redundant parameters in overparameterized neural networks in order to compress the model size and reduce computational cost. The goal is to prune a neural network in such a way that it has the same, or nearly the same, predictive performance as the original. In this paper we study neural network pruning in the context of conformal prediction. In order to explore whether the neural network can be pruned while maintaining the predictive efficiency of conformal predictors, our work measures and compares the efficiency of the prediction sets provided by the inductive conformal predictor built with an underlying pruned neural network. We implement several existing pruning methods and propose a new pruning method based specifically on the conformal prediction framework. By evaluating with various neural network architectures and across several data sets, we find that the pruned network can maintain, or indeed improve, the efficiency of the conformal predictors up to a particular pruning ratio and this pruning ratio varies with different architectures and data sets. These results are instructive for deploying pruned neural network in real-work applications within the context of conformal predictors, where reliable predictions and reduced computational cost are relevant, e.g. in healthcare or safety-critical applications. This work is also relevant for further work applying continual learning techniques in the context of conformal predictors. ' volume: 179 URL: https://proceedings.mlr.press/v179/zhao22a.html PDF: https://proceedings.mlr.press/v179/zhao22a/zhao22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-zhao22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Xindi family: Zhao - given: Anthony family: Bellotti editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 273-293 id: zhao22a issued: date-parts: - 2022 - 8 - 30 firstpage: 273 lastpage: 293 published: 2022-08-30 00:00:00 +0000 - title: 'Ellipsoidal conformal inference for Multi-Target Regression' abstract: 'Quantifying the uncertainty of a predictive model output is of essential importance in learning scenarios involving critical applications. As the learning task becomes more complex, so does uncertainty quantification. In this paper, we consider the task of multi-target regression and propose a method to output ellipsoidal confidence regions whose shapes are tailored to each instance to predict. We also guarantee that those confidence regions are well-calibrated, i.e., that they cover the ground truth with a specified probability. To achieve such a feat, we propose a conformal prediction method outputting ellipsoidal prediction regions. Experiments on both simulated and real-world data sets show that our methods outperform existing ones. ' volume: 179 URL: https://proceedings.mlr.press/v179/messoudi22a.html PDF: https://proceedings.mlr.press/v179/messoudi22a/messoudi22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-messoudi22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Soundouss family: Messoudi - given: Sébastien family: Destercke - given: Sylvain family: Rousseau editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 294-306 id: messoudi22a issued: date-parts: - 2022 - 8 - 30 firstpage: 294 lastpage: 306 published: 2022-08-30 00:00:00 +0000 - title: ' Prediction of Energy Consumption with Inductive Venn-Abers Predictive Distribution' volume: 179 URL: https://proceedings.mlr.press/v179/nouretdinov22b.html PDF: https://proceedings.mlr.press/v179/nouretdinov22b/nouretdinov22b.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-nouretdinov22b.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Ilia family: Nouretdinov - given: James family: Gammerman editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 307-309 id: nouretdinov22b issued: date-parts: - 2022 - 8 - 30 firstpage: 307 lastpage: 309 published: 2022-08-30 00:00:00 +0000 - title: 'Communication-efficient Conformal Prediction for Distributed Datasets' volume: 179 URL: https://proceedings.mlr.press/v179/riquelme-granada22a.html PDF: https://proceedings.mlr.press/v179/riquelme-granada22a/riquelme-granada22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-riquelme-granada22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Nery family: Riquelme-Granada - given: Zhiyuan family: Luo - given: Khuong family: An Nguyen editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 310-312 id: riquelme-granada22a issued: date-parts: - 2022 - 8 - 30 firstpage: 310 lastpage: 312 published: 2022-08-30 00:00:00 +0000 - title: 'House Price Prediction with Confidence: Empirical Results from the Norwegian Market' abstract: ' Automated Valuation Models are statistical models used by banks and other financial institutions to estimate the price of a dwelling, typically motivated by financial risk management purposes. The preferred choice of model for this task is often tree based machine learning models such as gradient boosted trees or random forest, where uncertainty quantification is a major challenge. In this empirical contribution, we compare split conformal inference, conformalized quantile regression and Mondrian conformalized quantile regression on data from the Norwegian housing market, and use random forest as a point prediction. The data consists of $N$ = 29 993 transactions from Oslo (Norway) from the time period 2018-2019. The results indicate that the methods using conformalized quantile regression create narrower confidence regions than split conformal inference.' volume: 179 URL: https://proceedings.mlr.press/v179/hjort22a.html PDF: https://proceedings.mlr.press/v179/hjort22a/hjort22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-hjort22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Anders family: Hjort editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 313-315 id: hjort22a issued: date-parts: - 2022 - 8 - 30 firstpage: 313 lastpage: 315 published: 2022-08-30 00:00:00 +0000 - title: 'Conformal Multistep-Ahead Multivariate Time-Series Forecasting' abstract: 'This paper proposes a method for conformal multistep-ahead multivariate time-series forecasting. The method minimizes the coverage loss when the data exchangeability assumption does not properly hold. This is done by weighting residual quantiles while computing prediction intervals. Preliminary experiments on real data demonstrate the method’s utility.' volume: 179 URL: https://proceedings.mlr.press/v179/schlembach22a.html PDF: https://proceedings.mlr.press/v179/schlembach22a/schlembach22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-schlembach22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Filip family: Schlembach - given: Evgueni family: Smirnov - given: Irena family: Koprinska editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 316-318 id: schlembach22a issued: date-parts: - 2022 - 8 - 30 firstpage: 316 lastpage: 318 published: 2022-08-30 00:00:00 +0000 - title: 'Conformal Decision Rules' abstract: 'This paper proposes conformal decision rules. They are defined as decision rules with their own conformal predictors. Given a test instance, conformal decision rules provide a point prediction, an explanation, a p-value for that prediction plus a prediction set.' volume: 179 URL: https://proceedings.mlr.press/v179/abdelqader22a.html PDF: https://proceedings.mlr.press/v179/abdelqader22a/abdelqader22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-abdelqader22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Husam family: Abdelqader - given: Evgueni family: Smirnov - given: Marc family: Pont - given: Marciano family: Geijselaers editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 319-321 id: abdelqader22a issued: date-parts: - 2022 - 8 - 30 firstpage: 319 lastpage: 321 published: 2022-08-30 00:00:00 +0000 - title: 'Uncertain data in learning: challenges and opportunities' abstract: 'Dealing with uncertain data in statistical estimation problems or in machine learning is not really a new issue. However, such uncertainty has so far mostly been modelled either as sets, being called for instance coarse data or partial labels, or as probability distributions over data values, being called for instance soft labels. Integrating this uncertainty in the learning process can be challenging, but also rewarding, as it can improve both the quality of the made predictions as well as our understanding of the obtained model. Within this setting, rich uncertainty models generalizing both probabilities and sets offer both new challenges and opportunities, and I will summarise some of them in this short note. ' volume: 179 URL: https://proceedings.mlr.press/v179/destercke22a.html PDF: https://proceedings.mlr.press/v179/destercke22a/destercke22a.pdf edit: https://github.com/mlresearch//v179/edit/gh-pages/_posts/2022-08-30-destercke22a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications' publisher: 'PMLR' author: - given: Sébastien family: Destercke editor: - given: Ulf family: Johansson - given: Henrik family: Boström - given: Khuong family: An Nguyen - given: Zhiyuan family: Luo - given: Lars family: Carlsson page: 322-332 id: destercke22a issued: date-parts: - 2022 - 8 - 30 firstpage: 322 lastpage: 332 published: 2022-08-30 00:00:00 +0000