With active time-domain surveys like the Zwicky Transient Facility, the anticipated Rubin Observatory’s Legacy Survey of Space and Time, and multi-messenger experiments such as LIGO/VIRGO/KANGRA for gravitational wave detection and IceCube for high-energy neutrino events, there is a new era in both time-domain and multi-messenger astronomy. The Astro2020 decadal survey highlights effectively responding to these astronomical alerts in a timely manner as a priority, and thus, there is an urgent need for the development of a seamless follow-up infrastructure at existing facilities that are capable of following up on detections at the surveys’ depths. At the W. M. Keck Observatory (WMKO), we are actively constructing critical infrastructure, aimed at facilitating the Target-of-Opportunity (ToO) trigger, optimizing observational planning, streamlining data acquisition, and enhancing data product accessibility. In this document, we provide an overview of these developing services and place them in context of existing observatory infrastructure like the Keck Observatory Archive (KOA) and Data Services Initiative (DSI).
KEYWORDS: Data archive systems, Keck Observatory, Human-machine interfaces, Data modeling, Astronomy, Equipment, Data storage, Data communications, Observatories, Image quality
The W. M. Keck Observatory Archive (KOA) has released the Observers’ Data Access Portal (ODAP), a web-application that delivers astronomical data from the W. M. Keck Observatory to the scheduled program’s principal investigator and their collaborators anywhere in the world in near real-time. Data files and their associated metadata are streamed to a user’s desktop machine moments after they are written to disk and archived in KOA. The ODAP User Interface is built in React and uses the WebSocket protocol to stream data between KOA and the user. This document describes the design of the tool, challenges encountered, shows how ODAP is integrated into the Keck observing model, and provides an analysis of usage metrics.
For 25 years, W. M. Keck Observatory has relied on observers to do their own planning for their observing nights. This would usually result in a starlist and a notion of what would be best to observe next based on the priority to the science they were conducting. Under the Data Services Initiative, this will become a required part of observing. The Database-Driven Observing Infrastructure aims to supplement the creation of science-ready data by carrying observation metadata throughout the observing process. The result is a file with all the data about the observation ready to be processed by the pipelines. In order to facilitate this, tools are being developed to help create better observing plans. One of the big complexities is that W. M. Keck Observatory currently supports ten active instruments with more on the horizon and no clear plan of retiring old instruments. With that in mind, the Database-Driven Observing Infrastructure system has been developed to be modular and instrument agnostic so that differences are abstracted from the system and handled only at the entrance and exit points of an observation. The benefit to this is that new instruments are easy to implement and old instruments are easy to update.
To maintain and expand its scientific productivity and impact, the W. M. Keck Observatory is undertaking a new strategic project to redefine how the Observatory approaches the creation of science products: the Data Services Initiative (DSI). The philosophy of DSI is grounded in the principle that the future of astronomy requires that data must be usable, useful, and quick. Reaching these data goals requires significant changes to key elements of the observing process: observation preparation, observation execution and calibration association, data reduction, and data archiving.
In this presentation, we will introduce DSI and its components, and describe the science gains that are enabled by it.
The W. M. Keck Observatory is welcoming a new era where data reduction and archiving are tightly integrated into our observing model, under the auspices of the Observatory’s Data Services Initiative (DSI) project. While previously the Keck Observatory Archive (KOA) archived minimally processed, raw science data the day after observing, Keck is transitioning to a model in which it archives both raw frames and reduced data in near real-time. These data will be made available to observers and collaborators immediately upon ingestion through a dedicated new interface that will support collaboration and sharing among teams, as well as stream data directly to personal computers without access to Keck’s internal networks. Both the raw and science-ready data products will be made publicly available upon the expiration of data protections. The Keck Cosmic Web Imager (KCWI) instrument is the first whose data are managed this way. It showcases how KOA integrates into an observing night, provides the data needed to make real-time adjustments to observing, and delivers products that allow for faster publication by both our observers and archival researchers. This effort has involved the delivery of new, compact, Python-based data preparation and ingestion software. We also discuss the new and updated Data Reduction Pipelines (DRPs) required to generate science-ready data, how their development and deployment enables the delivery of these products, and how Keck’s commitment to maintaining DRPs in-house will result in more robust datasets for all our observers and KOA users.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.