The Vera C. Rubin Observatory Data Butler and pipeline execution system

Tim Jenness; James F. Bosch; Andrei Salnikov; Nate B. Lust; Nathan M. Pease; Michelle Gower; Mikolaj Kowalik; Gregory P. Dubois-Felsmann; Fritz Mueller; Pim Schellart

doi:10.1117/12.2629569

29 August 2022 The Vera C. Rubin Observatory Data Butler and pipeline execution system

Tim Jenness, James F. Bosch, Andrei Salnikov, Nate B. Lust, Nathan M. Pease, Michelle Gower, Mikolaj Kowalik, Gregory P. Dubois-Felsmann, Fritz Mueller, Pim Schellart

Author Affiliations +

Proceedings Volume 12189, Software and Cyberinfrastructure for Astronomy VII; 1218911 (2022) https://doi.org/10.1117/12.2629569
Event: SPIE Astronomical Telescopes + Instrumentation, 2022, Montréal, Québec, Canada

Abstract

The Rubin Observatory’s Data Butler is designed to allow data file location and file formats to be abstracted away from the people writing the science pipeline algorithms. The Butler works in conjunction with the workflow graph builder to allow pipelines to be constructed from the algorithmic tasks. These pipelines can be executed at scale using object stores and multi-node clusters, or on a laptop using a local file system. The Butler and pipeline system are now in daily use during Rubin construction and early operations.

Conference Presentation

Citation Download Citation

Tim Jenness, James F. Bosch, Andrei Salnikov, Nate B. Lust, Nathan M. Pease, Michelle Gower, Mikolaj Kowalik, Gregory P. Dubois-Felsmann, Fritz Mueller, and Pim Schellart "The Vera C. Rubin Observatory Data Butler and pipeline execution system", Proc. SPIE 12189, Software and Cyberinfrastructure for Astronomy VII, 1218911 (29 August 2022); https://doi.org/10.1117/12.2629569

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

;

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE