Despite Python being the preferred programming language of choice for most astronomers, building or extending data reduction pipelines in the language can be problematic. A common approach is to write Python functions or classes as wrappers, calling individual pipeline recipes underneath, but this does not scale well with increasing pipeline complexity. Data management is also fraught since housekeeping code must be written to carefully handle input and output products between recipes. We have addressed these issues by creating an extensible pipeline development framework that leverages the Python bindings for the ESO Common Pipeline Library (PyCPL) toolkit. Pipeline recipes can be defined in a regulated manner using existing ESO pipeline recipes or new Python recipes compliant with ESO standards. Users can easily build their own pipeline workflows for execution by the PyCPL companion package PyEsorex. The ability to define Python recipes offers a powerful means to extend existing ESO pipelines or develop entirely new pipelines. An overview of the framework is presented along with an illustrative MUSE pipeline workflow.
Astronomers routinely have to collate heterogeneous observational data for one or several targets from a variety of online resources. Traditionally this process of data aggregation can be time consuming and error prone, especially if multiple telescope archives or data centres are searched individually. To streamline this task we have developed Data Central’s Data Aggregation Service (DAS), an interactive web application that leverages Aladin Lite to display images and catalogues resulting from multiple online service queries of a given target. The modern asynchronous Python design allows these queries to be sent simultaneously and individual results are quickly displayed as soon as they are received. The DAS also hosts Pipeline As a Web Service (PAWS) data reduction workflows that may be triggered on demand. The DAS can effectively unlock science from unreduced data in telescope archives and may help manage the massive data volumes expected from next generation facilities.
The Two-degree Field (2dF) facility of the Anglo-Australian Telescope (AAT) continues to take regular observations with millions of spectra collected over its lifetime. While individual projects have used the 2dFdr data reduction package to reduce and publish their own spectra, the majority of 2dF spectra are relatively inaccessible inside raw files located in the AAT archive. Here we introduce our 2dFdr Pipeline As a Web Service (PAWS) system that allows users to reduce 2dF-AAOmega observations on demand from the upgraded AAT archive. Without downloading data or installing 2dFdr, users can select science observations and reduction parameters before jobs are submitted for reduction. The system uses docker-py and Celery to robustly execute the reduction workflows, while a custom job tracking system keeps users informed of job progress. Data products may be downloaded and individual spectra can be viewed interactively. We intend to support additional instruments in the future.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.