- title: 'NeurIPS 2019 Competition and Demonstration Track: Revised selected papers' abstract: 'Machine learning competitions have grown in popularity and impact over the last decade, emerging as an effective means to advance the state of the art by posing well-structured, relevant, and challenging problems to the community at large. Motivated by a reward or merely the satisfaction of seeing their machine learning algorithm reach the top of a leaderboard, practitioners innovate, improve, and tune their approach before evaluating on a held-out dataset or environment. The competition track of NeurIPS has matured in 2019, its third year, with a considerable increase in both the number of challenges and the diversity of domains and topics. Demonstrations offer a complementary dimension to the competitions, focusing on areas of machine learning which are either human interactive or demonstrable in some way, for instance robotics applications or generative models. This volume is a compilation of selected papers associated with the NeurIPS 2019 Demonstration and Competition Track. The scope of the volume includes the design of the competitions, analysis of the results, novel methodologies developed to respond to the competitions’ challenges, and the design and development of creative demonstrations. ' volume: 123 URL: https://proceedings.mlr.press/v123/escalante20a.html PDF: http://proceedings.mlr.press/v123/escalante20a/escalante20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-escalante20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 1-12 id: escalante20a issued: date-parts: - 2020 - 8 - 19 firstpage: 1 lastpage: 12 published: 2020-08-19 00:00:00 +0000 - title: 'Efficient Model for Image Classification With Regularization Tricks' abstract: 'In the MicroNet Challenge 2019, competitors attempted to design the neural network architecture with fewer resource budgets, e.g., the number of parameters and FLOPS. In this study, we describe the approaches of team KAIST, using which they won the second and third places, respectively, in the CIFAR-100 classification task in the contest. We solve the task into four steps. First, we design a novel baseline network appropriate for the CIFAR-100 dataset. Second, we train this network using our novel structural regularization methods, which penalize the orthogonality of weights and replace the ground-truth label of each data with a noise vector that has class-wise similarity information from the representative feature vectors of each class in the course of training. Third, we seek the most potent data-augmentation methods for significant improvements in accuracy. At last, we perform the sparse training via a pruning technique. Our final score is 0.0054, which represents 370x improvements over the baseline for the CIFAR-100 dataset. This is the only work that finished in the top 10 percent of both parameter storage and computation over the CIFAR-100 classification task. The source code is at {https://github.com/Kthyeon/}micronet_neurips_challenge.' volume: 123 URL: https://proceedings.mlr.press/v123/kim20a.html PDF: http://proceedings.mlr.press/v123/kim20a/kim20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-kim20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Taehyeon family: Kim - given: Jonghyup family: Kim - given: Seyoung family: Yun editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 13-26 id: kim20a issued: date-parts: - 2020 - 8 - 19 firstpage: 13 lastpage: 26 published: 2020-08-19 00:00:00 +0000 - title: 'Causal structure learning from time series: Large regression coefficients may predict causal links better in practice than small p-values' abstract: 'In this article, we describe the algorithms for causal structure learning from time series data that won the Causality 4 Climate competition at the Conference on Neural Information Processing Systems 2019 (NeurIPS). We examine how our combination of established ideas achieves competitive performance on semi-realistic and realistic time series data exhibiting common challenges in real-world Earth sciences data. In particular, we discuss a) a rationale for leveraging linear methods to identify causal links in non-linear systems, b) a simulation-backed explanation as to why large regression coefficients may predict causal links better in practice than small p-values and thus why normalising the data may sometimes hinder causal structure learning. For benchmark usage, we detail the algorithms here and provide implementations at {https://github.com/sweichwald/tidybench}. We propose the presented competition-proven methods for baseline benchmark comparisons to guide the development of novel algorithms for structure learning from time series.' volume: 123 URL: https://proceedings.mlr.press/v123/weichwald20a.html PDF: http://proceedings.mlr.press/v123/weichwald20a/weichwald20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-weichwald20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Sebastian family: Weichwald - given: Martin E. family: Jakobsen - given: Phillip B. family: Mogensen - given: Lasse family: Petersen - given: Nikolaj family: Thams - given: Gherardo family: Varando editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 27-36 id: weichwald20a issued: date-parts: - 2020 - 8 - 19 firstpage: 27 lastpage: 36 published: 2020-08-19 00:00:00 +0000 - title: 'A Deep-learning-aided Automatic Vision-based Control Approach for Autonomous Drone Racing in Game of Drones Competition' abstract: 'In Game of Drones - Competition at NeurIPS 2019, this autonomous drone racing requires the drone to maneuver through the series of the gates without crashing. To complete the track, the drone has to be able to perceive the gates in the challenging environment from the FPV image in real-time and adjust its attitude accordingly. By utilizing deep-learning-aided detection and vision-based control approach, Team USRG completed the tier 2 challenge track passing the whole 21 gates in 81.19 seconds, and complete the tier 3 challenge track passing the whole 22 gates in 110.73 seconds.' volume: 123 URL: https://proceedings.mlr.press/v123/kim20b.html PDF: http://proceedings.mlr.press/v123/kim20b/kim20b.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-kim20b.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Donghwi family: Kim - given: Hyunjee family: Ryu - given: Jedsadakorn family: Yonchorhor - given: David Hyunchul family: Shim editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 37-46 id: kim20b issued: date-parts: - 2020 - 8 - 19 firstpage: 37 lastpage: 46 published: 2020-08-19 00:00:00 +0000 - title: 'Recurrent Autoencoder with Skip Connections and Exogenous Variables for Traffic Forecasting' abstract: 'The increasing complexity of mobility plus the growing population in cities, together with the importance of privacy when sharing data from vehicles or any device, makes traffic forecasting that uses data from infrastructure and citizens an open and challenging task. In this paper, we introduce a novel approach to deal with predictions of volume, speed and main traffic direction, in a new aggregated way of traffic data presented as videos. Our approach leverages the continuity in a sequence of frames, learning to embed them into a low dimensional space with an encoder and making predictions there using recurrent layers, ensuring good performance through an embedded loss, and then, recovering back spatial dimensions with a decoder using a second loss at a pixel level. Exogenous variables like weather, time and calendar are also added in the model. Furthermore, we introduce a novel sampling approach for sequences that ensures diversity when creating batches, running in parallel to the optimization process.' volume: 123 URL: https://proceedings.mlr.press/v123/herruzo20a.html PDF: http://proceedings.mlr.press/v123/herruzo20a/herruzo20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-herruzo20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Pedro family: Herruzo - given: Josep L. family: Larriba-Pey editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 47-55 id: herruzo20a issued: date-parts: - 2020 - 8 - 19 firstpage: 47 lastpage: 55 published: 2020-08-19 00:00:00 +0000 - title: 'Playing Minecraft with Behavioural Cloning' abstract: 'MineRL 2019 competition challenged participants to train sample-efficient agents to play Minecraft, by using a dataset of human gameplay and a limit number of steps the environment. We approached this task with behavioural cloning by predicting what actions human players would take, and reached fifth place in the final ranking. Despite being a simple algorithm, we observed the performance of such an approach can vary significantly, based on when the training is stopped. In this paper, we detail our submission to the competition, run further experiments to study how performance varied over training and study how different engineering decisions affected these results.' volume: 123 URL: https://proceedings.mlr.press/v123/kanervisto20a.html PDF: http://proceedings.mlr.press/v123/kanervisto20a/kanervisto20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-kanervisto20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Anssi family: Kanervisto - given: Janne family: Karttunen - given: Ville family: Hautamäki editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 56-66 id: kanervisto20a issued: date-parts: - 2020 - 8 - 19 firstpage: 56 lastpage: 66 published: 2020-08-19 00:00:00 +0000 - title: 'Sample Efficient Reinforcement Learning through Learning from Demonstrations in Minecraft' abstract: ' Sample inefficiency of deep reinforcement learning methods is a major obstacle for their use in real-world applications. In this work, we show how human demonstrations can improve final performance of agents on the Minecraft minigame ObtainDiamond with only 8M frames of environment interaction. We propose a training procedure where policy networks are first trained on human data and later fine-tuned by reinforcement learning. Using a policy exploitation mechanism, experience replay and an additional loss against catastrophic forgetting, our best agent was able to achieve a mean score of 48. Our proposed solution placed 3rd in the NeurIPS MineRL Competition for Sample-Efficient Reinforcement Learning.' volume: 123 URL: https://proceedings.mlr.press/v123/scheller20a.html PDF: http://proceedings.mlr.press/v123/scheller20a/scheller20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-scheller20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Christian family: Scheller - given: Yanick family: Schraner - given: Manfred family: Vogel editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 67-76 id: scheller20a issued: date-parts: - 2020 - 8 - 19 firstpage: 67 lastpage: 76 published: 2020-08-19 00:00:00 +0000 - title: 'F1TENTH: An Open-source Evaluation Environment for Continuous Control and Reinforcement Learning' abstract: 'The deployment and evaluation of learning algorithms on autonomous vehicles (AV) is expensive, slow, and potentially unsafe. This paper details the F1TENTH autonomous racing platform, an open-source evaluation framework for training, testing, and evaluating autonomous systems. With 1/10th-scale low-cost hardware and multiple virtual environments, F1TENTH enables safe and rapid experimentation of AV algorithms even in laboratory research settings. We present three benchmark tasks and baselines in the setting of autonomous racing, demonstrating the flexibility and features of our evaluation environment.' volume: 123 URL: https://proceedings.mlr.press/v123/o-kelly20a.html PDF: http://proceedings.mlr.press/v123/o-kelly20a/o-kelly20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-o-kelly20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Matthew family: O’Kelly - given: Hongrui family: Zheng - given: Dhruv family: Karthik - given: Rahul family: Mangharam editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 77-89 id: o-kelly20a issued: date-parts: - 2020 - 8 - 19 firstpage: 77 lastpage: 89 published: 2020-08-19 00:00:00 +0000 - title: 'Catch Me, If You Can! A Mediated Perception Approach Towards Fully Autonomous Drone Racing' abstract: 'Automated flight, e.g. first person view drone racing is a challenging task involving many sub-problems like monocular object detection, 3D pose estimation, mapping, optimal path planning and collision avoidance. Treating this problem, we propose an intuitive solution for the NeurIPS (2019) Game of Drones competition, especially the perception focused tier. We formulate a modular system composed of three layers: machine learning based perception, mapping and planning. Fundamental is a robust gate detection for target guidance accompanied with a monocular depth estimation for collision avoidance. The estimated targets are used to create and update the 3D gate positions within a map. Rule based trajectory planning is finally used for optimal flying. Our approach runs in real-time on a state of the art GPU and is able to robustly navigate through different simulated race tracks under challenging conditions, e.g. high speeds, confusing gate positioning and irregular shapes.Our approach ranks on the 3rd place on the final leader board. In this paper we present our system design in detail and provide additional experimental results.' volume: 123 URL: https://proceedings.mlr.press/v123/olsner20a.html PDF: http://proceedings.mlr.press/v123/olsner20a/olsner20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-olsner20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Florian family: Ölsner - given: Stefan family: Milz editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 90-99 id: olsner20a issued: date-parts: - 2020 - 8 - 19 firstpage: 90 lastpage: 99 published: 2020-08-19 00:00:00 +0000 - title: 'Evolution Algorithm and Online Learning for Racing Drone' abstract: 'Drone racing has become one of the challenging topics in robotics and machine learning because such a drone requires to equip with high performing modules that carry out demanding tasks, such as obstacle avoidance, mapping, and planning. In addition, one of the most crucial aspects of the racing drone is its speed. However, this is the somewhat less studied area compared to conventional topics such as obstacle avoidance and path-finding, probably because designing a loss function for the speed optimization with the gradient-based method is difficult. In this paper, we propose an evolutionary scheme for optimizing the speed-related parameters for shortening the travel time rather than using the gradient-based loss for them. For the planning part, we use an online learning method with the racing parameter optimization. Therefore, our approach is to combine evolutionary algorithms for speed optimization and gradient-based online learning, achieving first place in Tier 2 and Tier 3 in Game of Drones competition at NeurIPS 2019. ' volume: 123 URL: https://proceedings.mlr.press/v123/shin20a.html PDF: http://proceedings.mlr.press/v123/shin20a/shin20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-shin20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Sangyun family: Shin - given: Yongwon family: Kang - given: Yong-Guk family: Kim editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 100-109 id: shin20a issued: date-parts: - 2020 - 8 - 19 firstpage: 100 lastpage: 109 published: 2020-08-19 00:00:00 +0000 - title: 'The Causality for Climate Competition' abstract: 'Understanding the complex interdependencies of processes in our climate system has become one of the most critical challenges for society with our main current tools being climate modeling and observational data analysis, in particular observational causal discovery. Causal discovery is still in its infancy in Earth sciences and a major issue is that current methods are not well adapted to climate data challenges. We here present an overview of a NeurIPS 2019 competition on causal discovery for climate time series. The Causality 4 Climate (C4C) competition was hosted on the benchmark platform {www.causeme.net}. C4C offers an extensive number of climate model-based time series datasets with known causal ground truth that incorporate the main challenges of causal discovery in climate research. We give an overview over the benchmark platform, the challenges modeled, how datasets were generated, and implementation details. The goal of C4C is to spur more focused methodological research on causal discovery for understanding our climate system.' volume: 123 URL: https://proceedings.mlr.press/v123/runge20a.html PDF: http://proceedings.mlr.press/v123/runge20a/runge20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-runge20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Jakob family: Runge - given: Xavier-Andoni family: Tibau - given: Matthias family: Bruhns - given: Jordi family: Muñoz-Marí - given: Gustau family: Camps-Valls editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 110-120 id: runge20a issued: date-parts: - 2020 - 8 - 19 firstpage: 110 lastpage: 120 published: 2020-08-19 00:00:00 +0000 - title: 'The First International Competition in Machine Reconnaissance Blind Chess' abstract: 'Reconnaissance blind chess (RBC) is a chess variant in which a player cannot see her opponent’s pieces but can learn about them through private, explicit sensing actions. The game presents numerous research challenges, and was the focus of a competition held in conjunction with of the 2019 Conference on Neural Information Processing Systems (NeurIPS). The 22 bots that played in the tournament leveraged a diverse set of algorithms, including variations of multi-state tracking, piece-wise probability estimation, Gibbs sampling, bandit algorithms, tree search, counterfactual regret minimization (CFR), deep learning, and others. None of the algorithms of which we are aware converges to an optimal strategy. Top algorithms generally incorporated sensing strategies that successfully minimized uncertainty (as measured in the number of possible opponent states). The top two approaches reduced this raw uncertainty metric less than some others. Successful strategies sometimes defied conventional wisdom in chess, as evidenced by deviations between win rate and aggregate move strength as assessed by the leading available chess engine.' volume: 123 URL: https://proceedings.mlr.press/v123/gardner20a.html PDF: http://proceedings.mlr.press/v123/gardner20a/gardner20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-gardner20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Ryan W. family: Gardner - given: Corey family: Lowman - given: Casey family: Richardson - given: Ashley J. family: Llorens - given: Jared family: Markowitz - given: Nathan family: Drenkow - given: Andrew family: Newman - given: Gregory family: Clark - given: Gino family: Perrotta - given: Robert family: Perrotta - given: Timothy family: Highley - given: Vlad family: Shcherbina - given: William family: Bernadoni - given: Mark family: Jordan - given: Asen family: Asenov editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 121-130 id: gardner20a issued: date-parts: - 2020 - 8 - 19 firstpage: 121 lastpage: 130 published: 2020-08-19 00:00:00 +0000 - title: 'Empathic AI Painter: A Computational Creativity System with Embodied Conversational Interaction' abstract: 'There is a growing recognition that artists use valuable ways to understand and work with cognitive and perceptual mechanisms to convey desired experiences and narrative in their created artworks. This paper documents our attempt to computationally model the creative process of a portrait painter, who relies on understanding human traits (i.e., personality and emotions) to inform their art. Our system includes an empathic conversational interaction component to capture the dominant personality category of the user and a generative AI Portraiture system that uses this categorization to create a personalized stylization of the user’s portrait. This paper includes the description of our systems and the real-time interaction results obtained during the demonstration session of the NeurIPS 2019 Conference.' volume: 123 URL: https://proceedings.mlr.press/v123/yalcin20a.html PDF: http://proceedings.mlr.press/v123/yalcin20a/yalcin20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-yalcin20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Özge Nilay family: Yalçın - given: Nouf family: Abukhodair - given: Steve family: DiPaola editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 131-141 id: yalcin20a issued: date-parts: - 2020 - 8 - 19 firstpage: 131 lastpage: 141 published: 2020-08-19 00:00:00 +0000 - title: 'REAL-2019: Robot open-Ended Autonomous Learning competition' abstract: 'Open-ended learning, also called life-long learning or autonomous curriculum learning, aims to program machines and robots that autonomously acquire knowledge and skills in a cumulative fashion. We illustrate the first edition of the REAL-2019 – Robot open-Ended Autonomous Learning competition, prompted by the EU project GOAL-Robots – Goal-based Open-ended Autonomous Learning Robots. The competition was based on a simulated robot that: (a) acquires sensorimotor competence to interact with objects on a table; (b) learns autonomously based on mechanisms such as curiosity, intrinsic motivations, and self-generated goals. The competition featured a first intrinsic phase, where the robots learned to interact with the objects in a fully autonomous way (no rewards, predefined tasks or human guidance), and a second extrinsic phase, where the acquired knowledge was evaluated with tasks unknown during the first phase. The competition ran online on AIcrowd for six months, involved 75 subscribers and 6 finalists, and was presented at NeurIPS-2019. The competition revealed very hard as it involved difficult machine learning challenges usually tackled in isolation, such as exploration, sparse rewards, object learning, generalisation, catastrophic interference, and autonomous skill learning. Following the participant’s positive feedback, the preparation of a second REAL-2020 competition is underway, improving on the formulation of a relevant benchmark for open-ended learning.' volume: 123 URL: https://proceedings.mlr.press/v123/cartoni20a.html PDF: http://proceedings.mlr.press/v123/cartoni20a/cartoni20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-cartoni20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Emilio family: Cartoni - given: Francesco family: Mannella - given: Vieri Giuliano family: Santucci - given: Jochen family: Triesch - given: Elmar family: Rueckert - given: Gianluca family: Baldassarre editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 142-152 id: cartoni20a issued: date-parts: - 2020 - 8 - 19 firstpage: 142 lastpage: 152 published: 2020-08-19 00:00:00 +0000 - title: 'Graph-ResNets for short-term traffic forecasts in almost unknown cities' abstract: 'The 2019 IARAI traffic4cast competition is a traffic forecasting problem based on traffic data from three cities that are encoded as images. We developed a ResNet-inspired graph convolutional neural network (GCN) approach that uses street network-based subgraphs of the image lattice graphs as a prior. We train the Graph-ResNet together with GCN and convolutional neural network (CNN) benchmark models on Moscow traffic data and use them to first predict the traffic in Moscow and then to predict the traffic in Berlin and Istanbul. The results suggest that the graph-based models have superior generalization properties than CNN-based models for this application. We argue that in contrast to purely image-based approaches, formulating the prediction problem on a graph allows the neural network to learn properties given by the underlying street network. This facilitates the transfer of a learned network to predict the traffic status at unknown locations.' volume: 123 URL: https://proceedings.mlr.press/v123/martin20a.html PDF: http://proceedings.mlr.press/v123/martin20a/martin20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-martin20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Henry family: Martin - given: Dominik family: Bucher - given: Ye family: Hong - given: René family: Buffat - given: Christian family: Rupprecht - given: Martin family: Raubal editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 153-163 id: martin20a issued: date-parts: - 2020 - 8 - 19 firstpage: 153 lastpage: 163 published: 2020-08-19 00:00:00 +0000 - title: 'The Animal-AI Testbed and Competition' abstract: 'Modern machine learning systems are still lacking in the kind of general intelligence and common sense reasoning found, not only in humans, but across the animal kingdom. Many animals are capable of solving seemingly simple tasks such as inferring object location through object persistence and spatial elimination, and navigating efficiently in out-of-distribution novel environments. Such tasks are difficult for AI, but provide a natural stepping stone towards the goal of more complex human-like general intelligence. The extensive literature on animal cognition provides methodology and experimental paradigms for testing such abilities but, so far, these experiments have not been translated en masse into an AI-friendly setting. We present a new testbed, Animal-AI, first released as part of the Animal-AI Olympics competition at NeurIPS 2019, which is a comprehensive environment and testing paradigm for tasks inspired by animal cognition. In this paper we outline the environment, the testbed, the results of the competition, and discuss the open challenges for building and testing artificial agents capable of the kind of nonverbal common sense reasoning found in many non-human animals.' volume: 123 URL: https://proceedings.mlr.press/v123/crosby20a.html PDF: http://proceedings.mlr.press/v123/crosby20a/crosby20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-crosby20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Matthew family: Crosby - given: Benjamin family: Beyret - given: Murray family: Shanahan - given: José family: Hernández-Orallo - given: Lucy family: Cheke - given: Marta family: Halina editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 164-176 id: crosby20a issued: date-parts: - 2020 - 8 - 19 firstpage: 164 lastpage: 176 published: 2020-08-19 00:00:00 +0000 - title: 'AirSim Drone Racing Lab' abstract: 'Autonomous drone racing is a challenging research problem at the intersection of computer vision, planning, state estimation, and control. We introduce AirSim Drone Racing Lab, a simulation framework for enabling fast prototyping of algorithms for autonomy and enabling machine learning research in this domain, with the goal of reducing the time, money, and risks associated with field robotics. Our framework enables generation of racing tracks in multiple photo-realistic environments, orchestration of drone races, comes with a suite of gate assets, allows for multiple sensor modalities (monocular, depth, neuromorphic events, optical flow), different camera models, and benchmarking of planning, control, computer vision, and learning-based algorithms. We used our framework to host a simulation based drone racing competition at NeurIPS 2019. The competition binaries are available at our github repository {https://github.com/microsoft/AirSim-NeurIPS2019-Drone-Racing}. ' volume: 123 URL: https://proceedings.mlr.press/v123/madaan20a.html PDF: http://proceedings.mlr.press/v123/madaan20a/madaan20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-madaan20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Ratnesh family: Madaan - given: Nicholas family: Gyde - given: Sai family: Vemprala - given: Matthew family: Brown - given: Keiko family: Nagami - given: Tim family: Taubner - given: Eric family: Cristofalo - given: Davide family: Scaramuzza - given: Mac family: Schwager - given: Ashish family: Kapoor editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 177-191 id: madaan20a issued: date-parts: - 2020 - 8 - 19 firstpage: 177 lastpage: 191 published: 2020-08-19 00:00:00 +0000 - title: 'Visualizing and sonifying how an artificial ear hears music' abstract: ' A system is presented that visualizes and sonifies the inner workings of a sound processing neural network in real-time. The models that are employed have been trained on music datasets in a self-supervised way using contrastive predictive coding. An optimization procedure generates sounds that activate certain regions in the network. That way it can be rendered audible how music sounds to this artificial ear. In addition, the activations of the neurons at each point in time are visualized. For this, a force graph layout technique is used to create a vivid and dynamic representation of the neural network in action.' volume: 123 URL: https://proceedings.mlr.press/v123/herrmann20a.html PDF: http://proceedings.mlr.press/v123/herrmann20a/herrmann20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-herrmann20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Vincent family: Herrmann editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 192-202 id: herrmann20a issued: date-parts: - 2020 - 8 - 19 firstpage: 192 lastpage: 202 published: 2020-08-19 00:00:00 +0000 - title: 'Retrospective Analysis of the 2019 MineRL Competition on Sample Efficient Reinforcement Learning' abstract: 'To facilitate research in the direction of sample efficient reinforcement learning, we held the MineRL Competition on Sample Efficient Reinforcement Learning Using Human Priors at the Thirty-third Conference on Neural Information Processing Systems (NeurIPS 2019). The primary goal of this competition was to promote the development of algorithms that use human demonstrations alongside reinforcement learning to reduce the number of samples needed to solve complex, hierarchical, and sparse environments. We describe the competition, outlining the primary challenge, the competition design, and the resources that we provided to the participants. We provide an overview of the top solutions, each of which use deep reinforcement learning and/or imitation learning. We also discuss the impact of our organizational decisions on the competition and future directions for improvement.' volume: 123 URL: https://proceedings.mlr.press/v123/milani20a.html PDF: http://proceedings.mlr.press/v123/milani20a/milani20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-milani20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Stephanie family: Milani - given: Nicholay family: Topin - given: Brandon family: Houghton - given: William H. family: Guss - given: Sharada P. family: Mohanty - given: Keisuke family: Nakata - given: Oriol family: Vinyals - given: Noboru Sean family: Kuno editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 203-214 id: milani20a issued: date-parts: - 2020 - 8 - 19 firstpage: 203 lastpage: 214 published: 2020-08-19 00:00:00 +0000 - title: 'MicroNet for Efficient Language Modeling' abstract: 'It is important to design compact language models for efficient deployment. We improve upon recent advances in both the language modeling domain and the model-compression domain to construct parameter and computation efficient language models. We use an efficient transformer-based architecture with adaptive embedding and softmax, differentiable non-parametric cache, Hebbian softmax, knowledge distillation, network pruning, and low-bit quantization. In this paper, we provide the winning solution to the NeurIPS 2019 MicroNet Challenge in the language modeling track. Compared to the baseline language model provided by the MicroNet Challenge, our model is 90 times more parameter-efficient and 36 times more computation-efficient while achieving the required test perplexity of 35 on the Wikitext-103 dataset. We hope that this work will aid future research into efficient language models, and we have released our full source code at {https://github.com/mit-han-lab/neurips-micronet}.' volume: 123 URL: https://proceedings.mlr.press/v123/yan20a.html PDF: http://proceedings.mlr.press/v123/yan20a/yan20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-yan20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Zhongxia family: Yan - given: Hanrui family: Wang - given: Demi family: Guo - given: Song family: Han editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 215-231 id: yan20a issued: date-parts: - 2020 - 8 - 19 firstpage: 215 lastpage: 231 published: 2020-08-19 00:00:00 +0000 - title: 'The surprising efficiency of framing geo-spatial time series forecasting as a video prediction task – Insights from the IARAI Traffic4cast Competition at NeurIPS 2019' abstract: 'Deep Neural Networks models are state-of-the-art solutions in accurately forecasting future video frames in a movie. A successful video prediction model needs to extract and encode semantic features that describe the complex spatio-temporal correlations within image sequences of the real world. The IARAI Traffic4cast Challenge of the NeurIPS Competition Track 2019 for the first time introduced the novel argument that this is also highly relevant for urban traffic. By framing traffic prediction as a movie completion task, the challenge requires models to take advantage of complex geo-spatial and temporal patterns of the underlying process. We here report on the success and insights obtained in a first Traffic Map Movie forecasting challenge. Although short-term traffic prediction is considered hard, this novel approach allowed several research groups to successfully predict future traffic states in a purely data-driven manner from pixel space. We here expand on the original rationale, summarize key findings, and discuss promising future directions of the t4c competition at NeurIPS.' volume: 123 URL: https://proceedings.mlr.press/v123/kreil20a.html PDF: http://proceedings.mlr.press/v123/kreil20a/kreil20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-kreil20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: David P family: Kreil - given: Michael K family: Kopp - given: David family: Jonietz - given: Moritz family: Neun - given: Aleksandra family: Gruca - given: Pedro family: Herruzo - given: Henry family: Martin - given: Ali family: Soleymani - given: Sepp family: Hochreiter editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 232-241 id: kreil20a issued: date-parts: - 2020 - 8 - 19 firstpage: 232 lastpage: 241 published: 2020-08-19 00:00:00 +0000 - title: 'Towards Automated Deep Learning: Analysis of the AutoDL challenge series 2019' abstract: 'We present the design and results of recent competitions in Automated Deep Learning (AutoDL). In the AutoDL challenge series 2019, we organized 5 machine learning challenges: AutoCV, AutoCV2, AutoNLP, AutoSpeech and AutoDL. The first 4 challenges concern each a specific application domain, such as computer vision, natural language processing and speech recognition. At the time of March 2020, the last challenge AutoDL is still on-going and we only present its design. Some highlights of this work include: (1) a benchmark suite of baseline AutoML solutions, with emphasis on domains for which Deep Learning methods have had prior success (image, video, text, speech, etc); (2) a novel any-time learning framework, which opens doors for further theoretical consideration; (3) a repository of around 100 datasets (from all above domains) over half of which are released as public datasets to enable research on meta-learning; (4) analyses revealing that winning solutions generalize to new unseen datasets, validating progress towards universal AutoML solution; (5) open-sourcing of the challenge platform, the starting kit, the dataset formatting toolkit, and all winning solutions (All information available at {autodl.chalearn.org}).' volume: 123 URL: https://proceedings.mlr.press/v123/liu20a.html PDF: http://proceedings.mlr.press/v123/liu20a/liu20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-liu20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Zhengying family: Liu - given: Zhen family: Xu - given: Shangeth family: Rajaa - given: Meysam family: Madadi - given: Julio C. S. Jacques family: Junior - given: Sergio family: Escalera - given: Adrien family: Pavao - given: Sebastien family: Treguer - given: Wei-Wei family: Tu - given: Isabelle family: Guyon editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 242-252 id: liu20a issued: date-parts: - 2020 - 8 - 19 firstpage: 242 lastpage: 252 published: 2020-08-19 00:00:00 +0000 - title: 'A Global Health Gym Environment for RL Applications' abstract: 'This paper presents a platform for engaging in global health challenges, captured in the form of an OpenAI Gym styled environment. This platform has been used in three competitive challenges in 2019, and exposes a novel application domain for RL practitioners, along with the potential for significant social and scientific impact. While the platform has been demonstrated with problem formulations in global health, it is principally designed to facilitate general learning from simulation in a more abstract manner.' volume: 123 URL: https://proceedings.mlr.press/v123/remy20a.html PDF: http://proceedings.mlr.press/v123/remy20a/remy20a.pdf edit: https://github.com/mlresearch//v123/edit/gh-pages/_posts/2020-08-19-remy20a.md series: 'Proceedings of Machine Learning Research' container-title: 'Proceedings of the NeurIPS 2019 Competition and Demonstration Track' publisher: 'PMLR' author: - given: Sekou L. family: Remy - given: Oliver family: Bent editor: - given: Hugo Jair family: Escalante - given: Raia family: Hadsell page: 253-261 id: remy20a issued: date-parts: - 2020 - 8 - 19 firstpage: 253 lastpage: 261 published: 2020-08-19 00:00:00 +0000