Curating the Future: A Scalable Recipe for Training Open-Ended Forecasters

Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, Jonas Geiping
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:12596-12638, 2026.

Abstract

High-stakes decision making involves reasoning under uncertainty about the future. In this work, we train language models to make predictions on open-ended forecasting questions. To scale up training data, we synthesize novel forecasting questions from global events reported in daily news. While directly training on this data leads to performance drops, carefully curating questions creates a valuable training resource. We use the resulting dataset, OpenForesight, to post-train Qwen3 thinking models. To prevent leakage of future information during training and evaluation, we use an offline news corpus, both for data generation and retrieval in our forecasting system. Guided by a small validation set, we show the benefits of retrieval, and an improved reward function for reinforcement learning (RL). Once we obtain our final forecasting system, we perform held-out testing between May to August 2025. Our specialized model, OpenForecaster-8B, matches much larger proprietary models, with our training improving the accuracy, calibration, and consistency of predictions. We find calibration improvements from forecasting training generalize across popular benchmarks. We will open-source our models, code, and data to make LLM based forecasting research broadly accessible.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chandak26a, title = {Curating the Future: A Scalable Recipe for Training Open-Ended Forecasters}, author = {Chandak, Nikhil and Goel, Shashwat and Prabhu, Ameya and Hardt, Moritz and Geiping, Jonas}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {12596--12638}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chandak26a/chandak26a.pdf}, url = {https://proceedings.mlr.press/v306/chandak26a.html}, abstract = {High-stakes decision making involves reasoning under uncertainty about the future. In this work, we train language models to make predictions on open-ended forecasting questions. To scale up training data, we synthesize novel forecasting questions from global events reported in daily news. While directly training on this data leads to performance drops, carefully curating questions creates a valuable training resource. We use the resulting dataset, OpenForesight, to post-train Qwen3 thinking models. To prevent leakage of future information during training and evaluation, we use an offline news corpus, both for data generation and retrieval in our forecasting system. Guided by a small validation set, we show the benefits of retrieval, and an improved reward function for reinforcement learning (RL). Once we obtain our final forecasting system, we perform held-out testing between May to August 2025. Our specialized model, OpenForecaster-8B, matches much larger proprietary models, with our training improving the accuracy, calibration, and consistency of predictions. We find calibration improvements from forecasting training generalize across popular benchmarks. We will open-source our models, code, and data to make LLM based forecasting research broadly accessible.} }
Endnote
%0 Conference Paper %T Curating the Future: A Scalable Recipe for Training Open-Ended Forecasters %A Nikhil Chandak %A Shashwat Goel %A Ameya Prabhu %A Moritz Hardt %A Jonas Geiping %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chandak26a %I PMLR %P 12596--12638 %U https://proceedings.mlr.press/v306/chandak26a.html %V 306 %X High-stakes decision making involves reasoning under uncertainty about the future. In this work, we train language models to make predictions on open-ended forecasting questions. To scale up training data, we synthesize novel forecasting questions from global events reported in daily news. While directly training on this data leads to performance drops, carefully curating questions creates a valuable training resource. We use the resulting dataset, OpenForesight, to post-train Qwen3 thinking models. To prevent leakage of future information during training and evaluation, we use an offline news corpus, both for data generation and retrieval in our forecasting system. Guided by a small validation set, we show the benefits of retrieval, and an improved reward function for reinforcement learning (RL). Once we obtain our final forecasting system, we perform held-out testing between May to August 2025. Our specialized model, OpenForecaster-8B, matches much larger proprietary models, with our training improving the accuracy, calibration, and consistency of predictions. We find calibration improvements from forecasting training generalize across popular benchmarks. We will open-source our models, code, and data to make LLM based forecasting research broadly accessible.
APA
Chandak, N., Goel, S., Prabhu, A., Hardt, M. & Geiping, J.. (2026). Curating the Future: A Scalable Recipe for Training Open-Ended Forecasters. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:12596-12638 Available from https://proceedings.mlr.press/v306/chandak26a.html.

Related Material