- title: 'NeurIPS 2020 Competition and Demonstration Track: Revised selected papers'
abstract: 'This volume compiles a selection of papers associated with the fourth edition of the Demonstration and Competition Track at NeurIPS 2020. The track comprised 16 competitions and 20 demonstrations. Competition and demonstration proposals were subject to a strict reviewing process to ensure the quality of the accepted events. After a selective process, the accepted competitions and demonstrations were featured at the NeurIPS 2020 main conference. A wide diversity of machine learning topics were covered with competitions and demonstrations. The latter included innovative ways of interacting with participants due to the virtual format of NeurIPS2020. '
volume: 133
URL: https://proceedings.mlr.press/v133/escalante21a.html
PDF: http://proceedings.mlr.press/v133/escalante21a/escalante21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-escalante21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 1-2
id: escalante21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 1
lastpage: 2
published: 2021-08-07 00:00:00 +0000
- title: 'Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020'
abstract: 'This paper presents the results and insights from the black-box optimization (BBO) challenge at NeurIPS2020 which ran from July–October, 2020. The challenge emphasized the importance of evaluating derivative-free optimizers for tuning the hyperparameters of machine learning models. This was the first black-box optimization challenge with a machine learning emphasis. It was based on tuning (validation set) performance of standard machine learning models on real datasets. This competition has widespread impact as black-box optimization (e.g., Bayesian optimization) is relevant for hyperparameter tuning in almost every machine learning project as well as many applications outside of machine learning. The final leaderboard was determined using the optimization performance on held-out (hidden) objective functions, where the optimizers ran without human intervention. Baselines were set using the default settings of several open source black-box optimization packages as well as random search.'
volume: 133
URL: https://proceedings.mlr.press/v133/turner21a.html
PDF: http://proceedings.mlr.press/v133/turner21a/turner21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-turner21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Ryan
family: Turner
- given: David
family: Eriksson
- given: Michael
family: McCourt
- given: Juha
family: Kiili
- given: Eero
family: Laaksonen
- given: Zhen
family: Xu
- given: Isabelle
family: Guyon
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 3-26
id: turner21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 3
lastpage: 26
published: 2021-08-07 00:00:00 +0000
- title: 'tspDB: Time Series Predict DB'
abstract: 'A major bottleneck of the current Machine Learning (ML) workflow is the time consuming, error prone engineering required to get data from a datastore or a database (DB) to the point an ML algorithm can be applied to it. This is further exacerbated since ML algorithms are now trained on large volumes of data, yet we need predictions in real-time, especially in a variety of time-series applications such as finance and real-time control systems. Hence, we explore the feasibility of directly integrating prediction functionality on top of a data store or DB. Such a system ideally: (i) provides an intuitive prediction query interface which alleviates the unwieldy data engineering; (ii) provides state-of-the-art statistical accuracy while ensuring incremental model update, low model training time and low latency for making predictions. As the main contribution we explicitly instantiate a proof-of-concept, tspDB which directly integrates with PostgreSQL. We rigorously test tspDB’s statistical and computational performance against the state-of-the-art time series algorithms, including a Long-Short-Term-Memory (LSTM) neural network and DeepAR (industry standard deep learning library by Amazon). Statistically, on standard time series benchmarks, tspDB outperforms LSTM and DeepAR with 1.1-1.3x higher relative accuracy. Computationally, tspDB is 59-62x and 94-95x faster compared to LSTM and DeepAR in terms of median ML model training time and prediction query latency, respectively. Further, compared to PostgreSQL’s bulk insert time and its SELECT query latency, tspDB is slower only by 1.3x and 2.6x respectively. That is, tspDB is a real-time prediction system in that its model training / prediction query time is similar to just inserting, reading data from a DB. As an algorithmic contribution, we introduce an incremental multivariate matrix factorization based time series method, which tspDB is built off. We show this method also allows one to produce reliable prediction intervals by accurately estimating the time-varying variance of a time series, thereby addressing an important problem in time series analysis.'
volume: 133
URL: https://proceedings.mlr.press/v133/agarwal21a.html
PDF: http://proceedings.mlr.press/v133/agarwal21a/agarwal21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-agarwal21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Anish
family: Agarwal
- given: Abdullah
family: Alomar
- given: Devavrat
family: Shah
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 27-56
id: agarwal21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 27
lastpage: 56
published: 2021-08-07 00:00:00 +0000
- title: 'Learning Cloth Dynamics: 3D+Texture Garment Reconstruction Benchmark'
abstract: 'Human avatars are important targets in many computer applications. Accurately tracking, capturing, reconstructing and animating the human body, face and garments in 3D are critical for human-computer interaction, gaming, special effects and virtual reality. In the past, this has required extensive manual animation. Regardless of the advances in human body and face reconstruction, still modeling, learning and analyzing human dynamics need further attention. In this paper we plan to push the research in this direction, e.g. understanding human dynamics in 2D and 3D, with special attention to garments. We provide a large-scale dataset (more than 2M frames) of animated garments with variable topology and type, calledCLOTH3D++. The dataset contains RGBA video sequences paired with its corresponding 3D data. We pay special care to garment dynamics and realistic rendering of RGB data, including lighting, fabric type and texture. With this dataset, we hold a competition at NeurIPS2020. We design three tracks so participants can compete to develop the best method to perform 3D garment reconstruction in a sequence from (1) 3D-to-3D garments, (2) RGB-to-3D garments, and (3) RGB-to-3D garments plus texture. We also provide a baseline method, based on graph convolutional networks, for each track. Baseline results show that there is a lot of room for improvements. However, due to the challenging nature of the problem, no participant could outperform the baselines.'
volume: 133
URL: https://proceedings.mlr.press/v133/madadi21a.html
PDF: http://proceedings.mlr.press/v133/madadi21a/madadi21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-madadi21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Meysam
family: Madadi
- given: Hugo
family: Bertiche
- given: Wafa
family: Bouzouita
- given: Isabelle
family: Guyon
- given: Sergio
family: Escalera
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 57-76
id: madadi21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 57
lastpage: 76
published: 2021-08-07 00:00:00 +0000
- title: 'Solving Black-Box Optimization Challenge via Learning Search Space Partition for Local Bayesian Optimization'
abstract: 'Black-box optimization is one of the vital tasks in machine learning, since it approximates real-world conditions, in that we do not always know all the properties of a given system, up to knowing almost nothing but the results. This paper describes our approach to solving the black-box optimization challenge at NeurIPS 2020 through learning search space partition for local Bayesian optimization. We describe the task of the challenge as well as our algorithm for low budget optimization that we named SPBOpt. We optimize the hyper-parameters of our algorithm for the competition finals using multi-task Bayesian optimization on results from the first two evaluation settings. Our approach has ranked third in the competition finals.'
volume: 133
URL: https://proceedings.mlr.press/v133/sazanovich21a.html
PDF: http://proceedings.mlr.press/v133/sazanovich21a/sazanovich21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-sazanovich21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Mikita
family: Sazanovich
- given: Anastasiya
family: Nikolskaya
- given: Yury
family: Belousov
- given: Aleksei
family: Shpilman
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 77-85
id: sazanovich21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 77
lastpage: 85
published: 2021-08-07 00:00:00 +0000
- title: 'NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned'
abstract: 'We review the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers. The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets. These memory budgets were designed to encourage contestants to explore the trade-off between storing retrieval corpora or the parameters of learned models. In this report, we describe the motivation and organization of the competition, review the best submissions, and analyze system predictions to inform a discussion of evaluation for open-domain QA. '
volume: 133
URL: https://proceedings.mlr.press/v133/min21a.html
PDF: http://proceedings.mlr.press/v133/min21a/min21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-min21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Sewon
family: Min
- given: Jordan
family: Boyd-Graber
- given: Chris
family: Alberti
- given: Danqi
family: Chen
- given: Eunsol
family: Choi
- given: Michael
family: Collins
- given: Kelvin
family: Guu
- given: Hannaneh
family: Hajishirzi
- given: Kenton
family: Lee
- given: Jennimaria
family: Palomaki
- given: Colin
family: Raffel
- given: Adam
family: Roberts
- given: Tom
family: Kwiatkowski
- given: Patrick
family: Lewis
- given: Yuxiang
family: Wu
- given: Heinrich
family: Küttler
- given: Linqing
family: Liu
- given: Pasquale
family: Minervini
- given: Pontus
family: Stenetorp
- given: Sebastian
family: Riedel
- given: Sohee
family: Yang
- given: Minjoon
family: Seo
- given: Gautier
family: Izacard
- given: Fabio
family: Petroni
- given: Lucas
family: Hosseini
- given: Nicola De
family: Cao
- given: Edouard
family: Grave
- given: Ikuya
family: Yamada
- given: Sonse
family: Shimaoka
- given: Masatoshi
family: Suzuki
- given: Shumpei
family: Miyawaki
- given: Shun
family: Sato
- given: Ryo
family: Takahashi
- given: Jun
family: Suzuki
- given: Martin
family: Fajcik
- given: Martin
family: Docekal
- given: Karel
family: Ondrej
- given: Pavel
family: Smrz
- given: Hao
family: Cheng
- given: Yelong
family: Shen
- given: Xiaodong
family: Liu
- given: Pengcheng
family: He
- given: Weizhu
family: Chen
- given: Jianfeng
family: Gao
- given: Barlas
family: Oguz
- given: Xilun
family: Chen
- given: Vladimir
family: Karpukhin
- given: Stan
family: Peshterliev
- given: Dmytro
family: Okhonko
- given: Michael
family: Schlichtkrull
- given: Sonal
family: Gupta
- given: Yashar
family: Mehdad
- given: Wen-tau
family: Yih
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 86-111
id: min21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 86
lastpage: 111
published: 2021-08-07 00:00:00 +0000
- title: 'Learning to run a Power Network Challenge: a Retrospective Analysis'
abstract: 'Power networks, responsible for transporting electricity across large geographical regions, are complex infrastructures on which modern life critically depend. Variations in demand and production profiles, with increasing renewable energy integration, as well as the high voltage network technology, constitute a real challenge for human operators when optimizing electricity transportation while avoiding blackouts. Motivated to investigate the potential of Artificial Intelligence methods in enabling adaptability in power network operation, we have designed a L2RPN challenge to encourage the development of reinforcement learning solutions to key problems present in the next-generation power networks. The NeurIPS 2020 competition was well received by the international community attracting over 300 participants worldwide. The main contribution of this challenge is our proposed comprehensive ’Grid2Op’ framework, and associated benchmark, which plays realistic sequential network operations scenarios. The Grid2Op framework, which is open-source and easily re-usable, allows users to define new environments with its companion GridAlive ecosystem. Grid2Op relies on existing non-linear physical power network simulators and let users create a series of perturbations and challenges that are representative of two important problems: a) the uncertainty resulting from the increased use of unpredictable renewable energy sources, and b) the robustness required with contingent line disconnections. In this paper, we give the highlights of the NeurIPS 2020 competition. We present the benchmark suite and analyse the winning solutions, including one super-human performance demonstration. We propose our organizational insights for a successful competition and conclude on open research avenues. Given the challenge success, we expect our work will foster research to create more sustainable solutions for power network operations. '
volume: 133
URL: https://proceedings.mlr.press/v133/marot21a.html
PDF: http://proceedings.mlr.press/v133/marot21a/marot21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-marot21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Antoine
family: Marot
- given: Benjamin
family: Donnot
- given: Gabriel
family: Dulac-Arnold
- given: Adrian
family: Kelly
- given: Aidan
family: O’Sullivan
- given: Jan
family: Viebahn
- given: Mariette
family: Awad
- given: Isabelle
family: Guyon
- given: Patrick
family: Panciatici
- given: Camilo
family: Romero
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 112-132
id: marot21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 112
lastpage: 132
published: 2021-08-07 00:00:00 +0000
- title: 'MosAIc: Finding Artistic Connections across Culture with Conditional Image Retrieval'
abstract: 'We introduce MosAIc, an interactive web app that allows users to find pairs of semantically related artworks that span different cultures, media, and millennia. To create this application, we introduce Conditional Image Retrieval (CIR) which combines visual similarity search with user supplied filters or “conditions”. This technique allows one to find pairs of similar images that span distinct subsets of the image corpus. We provide a generic way to adapt existing image retrieval data-structures to this new domain and provide theoretical bounds on our approach’s efficiency. To quantify the performance of CIR systems, we introduce new datasets for evaluating CIR methods and show that CIR performs non-parametric style transfer. Finally, we demonstrate that our CIR data-structures can identify “blind spots” in Generative Adversarial Networks (GAN) where they fail to properly model the true data distribution.'
volume: 133
URL: https://proceedings.mlr.press/v133/hamilton21a.html
PDF: http://proceedings.mlr.press/v133/hamilton21a/hamilton21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-hamilton21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Mark
family: Hamilton
- given: Stephanie
family: Fu
- given: Mindren
family: Lu
- given: Johnny
family: Bui
- given: Darius
family: Bopp
- given: Zhenbang
family: Chen
- given: Felix
family: Tran
- given: Margaret
family: Wang
- given: Marina
family: Rogers
- given: Lei
family: Zhang
- given: Chris
family: Hoder
- given: William T.
family: Freeman
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 133-155
id: hamilton21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 133
lastpage: 155
published: 2021-08-07 00:00:00 +0000
- title: 'Semi-Automated Data Labeling'
abstract: 'Labeling data is often a tedious and error-prone activity. However, organizing the labeling experience as a human-machine collaboration has the potential to improve label quality and reduce human effort. In this paper we describe a semi-automated data labeling system which employs a predictive model to guide and assist the human labeler. The model learns by observing labeling decisions, and is used to recommend labels and automate basic functions in the labeling interface. Agreement between the labeler and the model is tracked and presented via a system of checkpoints. At each checkpoint the labeler has the opportunity to delegate the remainder of the labeling task to the model. '
volume: 133
URL: https://proceedings.mlr.press/v133/desmond21a.html
PDF: http://proceedings.mlr.press/v133/desmond21a/desmond21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-desmond21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Michael
family: Desmond
- given: Evelyn
family: Duesterwald
- given: Kristina
family: Brimijoin
- given: Michelle
family: Brachman
- given: Qian
family: Pan
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 156-169
id: desmond21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 156
lastpage: 169
published: 2021-08-07 00:00:00 +0000
- title: 'Methods and Analysis of The First Competition in Predicting Generalization of Deep Learning'
abstract: 'Deep learning has been recently successfully applied to an ever larger number of problems, ranging from pattern recognition to complex decision making. However, several concerns have been raised, including guarantees of good generalization, which is of foremost importance. Despite numerous attempts, conventional statistical learning approaches fall short of providing a satisfactory explanation on why deep learning works. In a competition hosted at the Thirty-Fourth Conference on Neural Information Processing Systems (NeurIPS 2020), we invited the community to design robust and general complexity measures that can accurately predict the generalization of models. In this paper, we describe the competition design, the protocols, and the solutions of the top-three teams at the competition in details. In addition, we discuss the outcomes, common failure modes, and potential future directions for the competition.'
volume: 133
URL: https://proceedings.mlr.press/v133/jiang21a.html
PDF: http://proceedings.mlr.press/v133/jiang21a/jiang21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-jiang21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Yiding
family: Jiang
- given: Parth
family: Natekar
- given: Manik
family: Sharma
- given: Sumukh K.
family: Aithal
- given: Dhruva
family: Kashyap
- given: Natarajan
family: Subramanyam
- given: Carlos
family: Lassance
- given: Daniel M.
family: Roy
- given: Gintare Karolina
family: Dziugaite
- given: Suriya
family: Gunasekar
- given: Isabelle
family: Guyon
- given: Pierre
family: Foret
- given: Scott
family: Yak
- given: Hossein
family: Mobahi
- given: Behnam
family: Neyshabur
- given: Samy
family: Bengio
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 170-190
id: jiang21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 170
lastpage: 190
published: 2021-08-07 00:00:00 +0000
- title: 'Results and Insights from Diagnostic Questions: The NeurIPS 2020 Education Challenge'
abstract: 'This competition concerns educational diagnostic questions, which are pedagogically effective, multiple-choice questions (MCQs) whose distractors embody misconceptions. With a large and ever-increasing number of such questions, it becomes overwhelming for teachers to know which questions are the best ones to use for their students. We thus seek to answer the following question: how can we use data on hundreds of millions of answers to MCQs to drive automatic personalized learning in large-scale learning scenarios where manual personalization is infeasible? Success in using MCQ data at scale helps build more intelligent, personalized learning platforms that ultimately improve the quality of education en masse. To this end, we introduce a new, large-scale, real-world dataset and formulate 4 data mining tasks on MCQs that mimic real learning scenarios and target various aspects of the above question in a competition setting at NeurIPS 2020. We report on our NeurIPS competition in which nearly 400 teams submitted approximately 4000 submissions, with encouragingly diverse and effective approaches to each of our tasks.'
volume: 133
URL: https://proceedings.mlr.press/v133/wang21a.html
PDF: http://proceedings.mlr.press/v133/wang21a/wang21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-wang21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Zichao
family: Wang
- given: Angus
family: Lamb
- given: Evgeny
family: Saveliev
- given: Pashmina
family: Cameron
- given: Jordan
family: Zaykov
- given: Jose Miguel
family: Hernandez-Lobato
- given: Richard E.
family: Turner
- given: Richard G.
family: Baraniuk
- given: Craig
family: Barton
- given: Simon
family: Peyton Jones
- given: Simon
family: Woodhead
- given: Cheng
prefix: Zhang
family:
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 191-205
id: wang21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 191
lastpage: 205
published: 2021-08-07 00:00:00 +0000
- title: 'Hide-and-Seek Privacy Challenge: Synthetic Data Generation vs. Patient Re-identification'
abstract: 'The clinical time-series setting poses a unique combination of challenges to data modelling and sharing. Due to the high dimensionality of clinical time series, adequate de-identification to preserve privacy while retaining data utility is difficult to achieve using common de-identification techniques. An innovative approach to this problem is synthetic data generation. From a technical perspective, a good generative model for time-series data should preserve temporal dynamics; new sequences should respect the original relationships between high-dimensional variables across time. From the privacy perspective, the model should prevent patient re-identification. The NeurIPS 2020 Hide-and-Seek Privacy Challenge was a novel two-tracked competition to simultaneously accelerate progress in tackling both problems. In our head-to-head format, participants in the generation track (?hiders?) and the patient re-identification track (?seekers?) were directly pitted against each other by way of a new, high-quality intensive care time-series dataset: the AmsterdamUMCdb dataset. In this paper we present an overview of the competition design, as well as highlighting areas we feel should be changed for future iterations of this competition.'
volume: 133
URL: https://proceedings.mlr.press/v133/jordon21a.html
PDF: http://proceedings.mlr.press/v133/jordon21a/jordon21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-jordon21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: James
family: Jordon
- given: Daniel
family: Jarrett
- given: Evgeny
family: Saveliev
- given: Jinsung
family: Yoon
- given: Paul
family: Elbers
- given: Patrick
family: Thoral
- given: Ari
family: Ercole
- given: Cheng
family: Zhang
- given: Danielle
family: Belgrave
- given: Mihaela
prefix: van der
family: Schaar
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 206-215
id: jordon21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 206
lastpage: 215
published: 2021-08-07 00:00:00 +0000
- title: 'The SpaceNet Multi-Temporal Urban Development Challenge'
abstract: 'Building footprints provide a useful proxy for a great many humanitarian applications. For example, building footprints are useful for high fidelity population estimates, and quantifying population statistics is fundamental to 1/4 of the United Nations Sustainable Development Goals Indicators. In this paper we (the SpaceNet Partners) discuss efforts to develop techniques for precise building footprint localization, tracking, and change detection via the SpaceNet Multi-Temporal Urban Development Challenge (also known as SpaceNet 7). In this NeurIPS 2020 competition, participants were asked identify and track buildings in satellite imagery time series collected over rapidly urbanizing areas. The competition centered around a brand new open source dataset of Planet Labs satellite imagery mosaics at 4m resolution, which includes 24 images (one per month) covering 100 unique geographies. Tracking individual buildings at this resolution is quite challenging, yet the winning participants demonstrated impressive performance with the newly developed SpaceNet Change and Object Tracking (SCOT) metric. This paper details the top-5 winning approaches, as well as analysis of results that yielded a handful of interesting anecdotes such as decreasing performance with latitude.'
volume: 133
URL: https://proceedings.mlr.press/v133/etten21a.html
PDF: http://proceedings.mlr.press/v133/etten21a/etten21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-etten21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Adam Van
family: Etten
- given: Daniel
family: Hogan
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 216-232
id: etten21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 216
lastpage: 232
published: 2021-08-07 00:00:00 +0000
- title: 'Towards robust and domain agnostic reinforcement learning competitions: MineRL 2020'
abstract: 'Reinforcement learning competitions have formed the basis for standard research benchmarks, galvanized advances in the state-of-the-art, and shaped the direction of the field. Despite this, a majority of challenges suffer from the same fundamental problems: participant solutions to the posed challenge are usually domain-specific, biased to maximally exploit compute resources, and not guaranteed to be reproducible. In this paper, we present a new framework of competition design that promotes the development of algorithms that overcome these barriers. We propose four central mechanisms for achieving this end: submission retraining, domain randomization, desemantization through domain obfuscation, and the limitation of competition compute and environment-sample budget. To demonstrate the efficacy of this design, we proposed, organized, and ran the MineRL 2020 Competition on Sample-Efficient Reinforcement Learning. In this work, we describe the organizational outcomes of the competition and show that the resulting participant submissions are reproducible, non-specific to the competition environment, and sample/resource efficient, despite the difficult competition task.'
volume: 133
URL: https://proceedings.mlr.press/v133/guss21a.html
PDF: http://proceedings.mlr.press/v133/guss21a/guss21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-guss21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: William Hebgen
family: Guss
- given: Stephanie
family: Milani
- given: Nicholay
family: Topin
- given: Brandon
family: Houghton
- given: Sharada
family: Mohanty
- given: Andrew
family: Melnik
- given: Augustin
family: Harter
- given: Benoit
family: Buschmaas
- given: Bjarne
family: Jaster
- given: Christoph
family: Berganski
- given: Dennis
family: Heitkamp
- given: Marko
family: Henning
- given: Helge
family: Ritter
- given: Chengjie
family: Wu
- given: Xiaotian
family: Hao
- given: Yiming
family: Lu
- given: Hangyu
family: Mao
- given: Yihuan
family: Mao
- given: Chao
family: Wang
- given: Michal
family: Opanowicz
- given: Anssi
family: Kanervisto
- given: Yanick
family: Schraner
- given: Christian
family: Scheller
- given: Xiren
family: Zhou
- given: Lu
family: Liu
- given: Daichi
family: Nishio
- given: Toi
family: Tsuneda
- given: Karolis
family: Ramanauskas
- given: Gabija
family: Juceviciute
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 233-252
id: guss21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 233
lastpage: 252
published: 2021-08-07 00:00:00 +0000
- title: 'Musical Speech: A Transformer-based Composition Tool'
abstract: 'In this paper, we propose a new compositional tool that will generate a musical outline of speech recorded/provided by the user for use as a musical building block in their compositions. The tool allows any user to use their own speech to generate musical material, while still being able to hear the direct connection between their recorded speech and the resulting music. The tool is built on our proposed pipeline. This pipeline begins with speech-based signal processing, after which some simple musical heuristics are applied, and finally these pre-processed signals are passed through Transformer models trained on new musical tasks. We illustrate the effectiveness of our pipeline – which does not require a paired dataset for training – through examples of music created by musicians making use of our tool.'
volume: 133
URL: https://proceedings.mlr.press/v133/d-eon21a.html
PDF: http://proceedings.mlr.press/v133/d-eon21a/d-eon21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-d-eon21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Jason
family: d’Eon
- given: Sri Harsha
family: Dumpla
- given: Chandramouli Shama
family: Sastry
- given: Daniel
family: Oore
- given: Sageev
family: Oore
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 253-274
id: d-eon21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 253
lastpage: 274
published: 2021-08-07 00:00:00 +0000
- title: 'Flatland Competition 2020: MAPF and MARL for Efficient Train Coordination on a Grid World'
abstract: 'The Flatland competition aimed at finding novel approaches to solve the vehicle re-scheduling problem (VRSP). The VRSP is concerned with scheduling trips in traffic networks and the re-scheduling of vehicles when disruptions occur, for example the breakdown of a vehicle. While solving the VRSP in various settings has been an active area in operations research (OR) for decades, the ever-growing complexity of modern railway networks makes dynamic real-time scheduling of traffic virtually impossible. Recently, multi-agent reinforcement learning (MARL) has successfully tackled challenging tasks where many agents need to be coordinated, such as multiplayer video games. However, the coordination of hundreds of agents in a real-life setting like a railway network remains challenging and the Flatland environment used for the competition models these real-world properties in a simplified manner. Submissions had to bring as many trains (agents) to their target stations in as little time as possible. While the best submissions were in the OR category, participants found many promising MARL approaches. Using both centralized and decentralized learning based approaches, top submissions used graph representations of the environment to construct tree-based observations. Further, different coordination mechanisms were implemented, such as communication and prioritization between agents. This paper presents the competition setup, four outstanding solutions to the competition, and a cross-comparison between them. '
volume: 133
URL: https://proceedings.mlr.press/v133/laurent21a.html
PDF: http://proceedings.mlr.press/v133/laurent21a/laurent21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-laurent21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Florian
family: Laurent
- given: Manuel
family: Schneider
- given: Christian
family: Scheller
- given: Jeremy
family: Watson
- given: Jiaoyang
family: Li
- given: Zhe
family: Chen
- given: Yi
family: Zheng
- given: Shao-Hung
family: Chan
- given: Konstantin
family: Makhnev
- given: Oleg
family: Svidchenko
- given: Vladimir
family: Egorov
- given: Dmitry
family: Ivanov
- given: Aleksei
family: Shpilman
- given: Evgenija
family: Spirovska
- given: Oliver
family: Tanevski
- given: Aleksandar
family: Nikov
- given: Ramon
family: Grunder
- given: David
family: Galevski
- given: Jakov
family: Mitrovski
- given: Guillaume
family: Sartoretti
- given: Zhiyao
family: Luo
- given: Mehul
family: Damani
- given: Nilabha
family: Bhattacharya
- given: Shivam
family: Agarwal
- given: Adrian
family: Egli
- given: Erik
family: Nygren
- given: Sharada
family: Mohanty
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 275-301
id: laurent21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 275
lastpage: 301
published: 2021-08-07 00:00:00 +0000
- title: 'NeurIPS 2020 NLC2CMD Competition: Translating Natural Language to Bash Commands'
abstract: 'The NLC2CMD Competition hosted at NeurIPS 2020 aimed to bring the power of natural language processing to the command line. Participants were tasked with building models that can transform descriptions of command line tasks in English to their Bash syntax. This is a report on the competition with details of the task, metrics, data, attempted solutions, and lessons learned.'
volume: 133
URL: https://proceedings.mlr.press/v133/agarwal21b.html
PDF: http://proceedings.mlr.press/v133/agarwal21b/agarwal21b.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-agarwal21b.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Mayank
family: Agarwal
- given: Tathagata
family: Chakraborti
- given: Quchen
family: Fu
- given: David
family: Gros
- given: Xi Victoria
family: Lin
- given: Jaron
family: Maene
- given: Kartik
family: Talamadupula
- given: Zhongwei
family: Teng
- given: Jules
family: White
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 302-324
id: agarwal21b
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 302
lastpage: 324
published: 2021-08-07 00:00:00 +0000
- title: 'Traffic4cast at NeurIPS 2020 - yet more on the unreasonable effectiveness of gridded geo-spatial processes'
abstract: 'The IARAI Traffic4cast competition at NeurIPS 2019 showed that neural networks can successfully predict future traffic conditions 15 minutes into the future on simply aggregated GPS probe data in time and space bins, thus interpreting the challenge of forecasting traffic conditions as a movie completion task. U-nets proved to be the winning architecture then, demonstrating an ability to extract relevant features in the complex, real-world, geo-spatial process that is traffic derived from a large data set. The IARAI Traffic4cast challenge at NeurIPS 2020 build on the insights of the previous year and sought to both challenge some assumptions inherent in our 2019 competition design and explore how far this neural network technique can be pushed. We found that the prediction horizon can be extended successfully to 60 minutes into the future, that there is further evidence that traffic depends more on recent dynamics than on the additional static or dynamic location specific data provided and that a reasonable starting point when exploring a general aggregated geo-spatial process in time and space is a U-net architecture.'
volume: 133
URL: https://proceedings.mlr.press/v133/kopp21a.html
PDF: http://proceedings.mlr.press/v133/kopp21a/kopp21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-kopp21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Michael
family: Kopp
- given: David
family: Kreil
- given: Moritz
family: Neun
- given: David
family: Jonietz
- given: Henry
family: Martin
- given: Pedro
family: Herruzo
- given: Aleksandra
family: Gruca
- given: Ali
family: Soleymani
- given: Fanyou
family: Wu
- given: Yang
family: Liu
- given: Jingwei
family: Xu
- given: Jianjin
family: Zhang
- given: Jay
family: Santokhi
- given: Alabi
family: Bojesomo
- given: Hasan Al
family: Marzouqi
- given: Panos
family: Liatsis
- given: Pak Hay
family: Kwok
- given: Qi
family: Qi
- given: Sepp
family: Hochreiter
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 325-343
id: kopp21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 325
lastpage: 343
published: 2021-08-07 00:00:00 +0000
- title: 'The Hateful Memes Challenge: Competition Report'
abstract: 'Machine learning and artificial intelligence play an ever more crucial role in mitigating important societal problems, such as the prevalence of hate speech. We describe the Hateful Memes Challenge competition, held at NeurIPS 2020, focusing on multimodal hate speech. The aim of the challenge is to facilitate further research into multimodal reasoning and understanding.'
volume: 133
URL: https://proceedings.mlr.press/v133/kiela21a.html
PDF: http://proceedings.mlr.press/v133/kiela21a/kiela21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-kiela21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Douwe
family: Kiela
- given: Hamed
family: Firooz
- given: Aravind
family: Mohan
- given: Vedanuj
family: Goswami
- given: Amanpreet
family: Singh
- given: Casey A.
family: Fitzpatrick
- given: Peter
family: Bull
- given: Greg
family: Lipstein
- given: Tony
family: Nelli
- given: Ron
family: Zhu
- given: Niklas
family: Muennighoff
- given: Riza
family: Velioglu
- given: Jewgeni
family: Rose
- given: Phillip
family: Lippe
- given: Nithin
family: Holla
- given: Shantanu
family: Chandra
- given: Santhosh
family: Rajamanickam
- given: Georgios
family: Antoniou
- given: Ekaterina
family: Shutova
- given: Helen
family: Yannakoudakis
- given: Vlad
family: Sandulescu
- given: Umut
family: Ozertem
- given: Patrick
family: Pantel
- given: Lucia
family: Specia
- given: Devi
family: Parikh
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 344-360
id: kiela21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 344
lastpage: 360
published: 2021-08-07 00:00:00 +0000
- title: 'Measuring Sample Efficiency and Generalization in Reinforcement Learning Benchmarks: NeurIPS 2020 Procgen Benchmark'
abstract: 'The NeurIPS 2020 Procgen Competition was designed as a centralized benchmark with clearly defined tasks for measuring Sample Efficiency and Generalization in Reinforcement Learning. Generalization remains one of the most fundamental challenges in deep reinforcement learning, and yet we do not have enough benchmarks to measure the progress of the community on Generalization in Reinforcement Learning. We present the design of a centralized benchmark for Reinforcement Learning which can help measure Sample Efficiency and Generalization in Reinforcement Learning by doing end to end evaluation of the training and rollout phases of thousands of user submitted code bases in a scalable way. We designed the benchmark on top of the already existing Procgen Benchmark by defining clear tasks and standardizing the end to end evaluation setups. The design aims to maximize the flexibility available for researchers who wish to design future iterations of such benchmarks, and yet imposes necessary practical constraints to allow for a system like this to scale. This paper presents the competition setup and the details and analysis of the top solutions identified through this setup in context of 2020 iteration of the competition at NeurIPS.'
volume: 133
URL: https://proceedings.mlr.press/v133/mohanty21a.html
PDF: http://proceedings.mlr.press/v133/mohanty21a/mohanty21a.pdf
edit: https://github.com/mlresearch//v133/edit/gh-pages/_posts/2021-08-07-mohanty21a.md
series: 'Proceedings of Machine Learning Research'
container-title: 'Proceedings of the NeurIPS 2020 Competition and Demonstration Track'
publisher: 'PMLR'
author:
- given: Sharada
family: Mohanty
- given: Jyotish
family: Poonganam
- given: Adrien
family: Gaidon
- given: Andrey
family: Kolobov
- given: Blake
family: Wulfe
- given: Dipam
family: Chakraborty
- given: Graz̆vydas
family: S̆emetulskis
- given: João
family: Schapke
- given: Jonas
family: Kubilius
- given: Jurgis
family: Paükonis
- given: Linas
family: Klimas
- given: Matthew
family: Hausknecht
- given: Patrick
family: MacAlpine
- given: Quang Nhat
family: Tran
- given: Thomas
family: Tumiel
- given: Xiaocheng
family: Tang
- given: Xinwei
family: Chen
- given: Christopher
family: Hesse
- given: Jacob
family: Hilton
- given: William Hebgen
family: Guss
- given: Sahika
family: Genc
- given: John
family: Schulman
- given: Karl
family: Cobbe
editor:
- given: Hugo Jair
family: Escalante
- given: Katja
family: Hofmann
page: 361-395
id: mohanty21a
issued:
date-parts:
- 2021
- 8
- 7
firstpage: 361
lastpage: 395
published: 2021-08-07 00:00:00 +0000