PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Control

Ruijie Zheng, Ching-An Cheng, Hal Daumé Iii, Furong Huang, Andrey Kolobov
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:61267-61286, 2024.

Abstract

Temporal action abstractions, along with belief state representations, are a powerful knowledge sharing mechanism for sequential decision making. In this work, we propose a novel view that treats inducing temporal action abstractions as a sequence compression problem. To do so, we bring a subtle but critical component of LLM training pipelines – input tokenization via byte pair encoding (BPE) – to bear on the seemingly distant task of learning skills of variable time span in continuous control domains. We introduce an approach called Primitive Sequence Encoding (PRISE) that combines continuous action quantization with BPE to learn powerful action abstractions. We empirically show that high-level skills discovered by PRISE from a multitask set of robotic manipulation demonstrations significantly boost the learning performance of behavior cloning on downstream tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-zheng24b, title = {{PRISE}: {LLM}-Style Sequence Compression for Learning Temporal Action Abstractions in Control}, author = {Zheng, Ruijie and Cheng, Ching-An and Daum\'{e} Iii, Hal and Huang, Furong and Kolobov, Andrey}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {61267--61286}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/zheng24b/zheng24b.pdf}, url = {https://proceedings.mlr.press/v235/zheng24b.html}, abstract = {Temporal action abstractions, along with belief state representations, are a powerful knowledge sharing mechanism for sequential decision making. In this work, we propose a novel view that treats inducing temporal action abstractions as a sequence compression problem. To do so, we bring a subtle but critical component of LLM training pipelines – input tokenization via byte pair encoding (BPE) – to bear on the seemingly distant task of learning skills of variable time span in continuous control domains. We introduce an approach called Primitive Sequence Encoding (PRISE) that combines continuous action quantization with BPE to learn powerful action abstractions. We empirically show that high-level skills discovered by PRISE from a multitask set of robotic manipulation demonstrations significantly boost the learning performance of behavior cloning on downstream tasks.} }
Endnote
%0 Conference Paper %T PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Control %A Ruijie Zheng %A Ching-An Cheng %A Hal Daumé Iii %A Furong Huang %A Andrey Kolobov %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-zheng24b %I PMLR %P 61267--61286 %U https://proceedings.mlr.press/v235/zheng24b.html %V 235 %X Temporal action abstractions, along with belief state representations, are a powerful knowledge sharing mechanism for sequential decision making. In this work, we propose a novel view that treats inducing temporal action abstractions as a sequence compression problem. To do so, we bring a subtle but critical component of LLM training pipelines – input tokenization via byte pair encoding (BPE) – to bear on the seemingly distant task of learning skills of variable time span in continuous control domains. We introduce an approach called Primitive Sequence Encoding (PRISE) that combines continuous action quantization with BPE to learn powerful action abstractions. We empirically show that high-level skills discovered by PRISE from a multitask set of robotic manipulation demonstrations significantly boost the learning performance of behavior cloning on downstream tasks.
APA
Zheng, R., Cheng, C., Daumé Iii, H., Huang, F. & Kolobov, A.. (2024). PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Control. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:61267-61286 Available from https://proceedings.mlr.press/v235/zheng24b.html.

Related Material