KEYWORDS: Data archive systems, Keck Observatory, Human-machine interfaces, Data modeling, Astronomy, Equipment, Data storage, Data communications, Observatories, Image quality
The W. M. Keck Observatory Archive (KOA) has released the Observers’ Data Access Portal (ODAP), a web-application that delivers astronomical data from the W. M. Keck Observatory to the scheduled program’s principal investigator and their collaborators anywhere in the world in near real-time. Data files and their associated metadata are streamed to a user’s desktop machine moments after they are written to disk and archived in KOA. The ODAP User Interface is built in React and uses the WebSocket protocol to stream data between KOA and the user. This document describes the design of the tool, challenges encountered, shows how ODAP is integrated into the Keck observing model, and provides an analysis of usage metrics.
The Keck Planet Finder (KPF) is a fiber-fed, high-resolution, echelle spectrometer that specializes in the discovery and characterization of exoplanets using Doppler spectroscopy. In designing KPF, the guiding principles were high throughput to promote survey speed and access to faint targets, and high stability to keep uncalibrated systematic Doppler measurement errors below 30 cm s−1. KPF achieves optical illumination stability with a tip-tilt injection system, octagonal cross-section optical fibers, a double scrambler, and active fiber agitation. The optical bench and optics with integral mounts are made of Zerodur to provide thermo-mechanical stability. The spectrometer includes a slicer to reformat the optical input, green and red channels (445–600 nm and 600–870 nm), and achieves a resolving power of ∼97,000. Additional subsystems include a separate, medium-resolution UV spectrometer (383–402 nm) to record the Ca II H & K lines, an exposure meter for real-time flux monitoring, a solar feed for sunlight injection, and a calibration system with a laser frequency comb and etalon for wavelength calibration. KPF was installed and commissioned at the W. M. Keck Observatory in late 2022 and early 2023 and is now in regular use for scientific observations. This paper presents an overview of the as-built KPF instrument and its subsystems, design considerations, and initial on-sky performance.
To maintain and expand its scientific productivity and impact, the W. M. Keck Observatory is undertaking a new strategic project to redefine how the Observatory approaches the creation of science products: the Data Services Initiative (DSI). The philosophy of DSI is grounded in the principle that the future of astronomy requires that data must be usable, useful, and quick. Reaching these data goals requires significant changes to key elements of the observing process: observation preparation, observation execution and calibration association, data reduction, and data archiving.
In this presentation, we will introduce DSI and its components, and describe the science gains that are enabled by it.
The W. M. Keck Observatory is welcoming a new era where data reduction and archiving are tightly integrated into our observing model, under the auspices of the Observatory’s Data Services Initiative (DSI) project. While previously the Keck Observatory Archive (KOA) archived minimally processed, raw science data the day after observing, Keck is transitioning to a model in which it archives both raw frames and reduced data in near real-time. These data will be made available to observers and collaborators immediately upon ingestion through a dedicated new interface that will support collaboration and sharing among teams, as well as stream data directly to personal computers without access to Keck’s internal networks. Both the raw and science-ready data products will be made publicly available upon the expiration of data protections. The Keck Cosmic Web Imager (KCWI) instrument is the first whose data are managed this way. It showcases how KOA integrates into an observing night, provides the data needed to make real-time adjustments to observing, and delivers products that allow for faster publication by both our observers and archival researchers. This effort has involved the delivery of new, compact, Python-based data preparation and ingestion software. We also discuss the new and updated Data Reduction Pipelines (DRPs) required to generate science-ready data, how their development and deployment enables the delivery of these products, and how Keck’s commitment to maintaining DRPs in-house will result in more robust datasets for all our observers and KOA users.
The Keck Planet Finder (KPF) is a fiber-fed, high-resolution, high-stability spectrometer in development at the UC Berkeley Space Sciences Laboratory for the W.M. Keck Observatory. KPF is designed to characterize exoplanets via Doppler spectroscopy with a goal of a single measurement precision of 0.3 m s-1 or better, however its resolution and stability will enable a wide variety of astrophysical pursuits. Here we provide post-preliminary design review design updates for several subsystems, including: the main spectrometer, the fabrication of the Zerodur optical bench; the data reduction pipeline; fiber agitator; fiber cable design; fiber scrambler; VPH testing results and the exposure meter.
The progress achieved in implementing Point Spread Function reconstruction (PSF-R) capability at W. M. Keck Observatory (WMKO) is discussed. Observations of low-mass binary systems have been used to evaluate the improvements in astrometry and photometry using reconstructed PSFs. The on-sky performance of PSF-R is discussed by comparing the binary-fitting analysis using the reconstructed PSFs with the standard methods. We show that the PSFR in the NGS provides comparable performance to having a close reference star in the imaging science instrument. The on-sky troubleshooting efforts and the recent PSF-R technical developments are also presented. We find that the PSF-R reconstruction is more of a systems science problem that a post-processing problem. We close by discussing the lessons learned in the context of existing and future extremely large telescopes.
This paper describes how we have sustained the Montage image mosaic engine (http://montage.ipac.caltech.edu) first released in 2002, to support the ever-growing scale and complexity of modern data sets. The key to its longevity has been its design as a toolkit written in ANSI-C, with each tool performing one distinct task, for easy integration into scripts, pipelines and workflows. The same code base now supports Windows, JavaScript and Python by taking advantage of recent advances in compilers. The design has led to applicability of Montage far beyond what was anticipated when Montage was first built, such as supporting observation planning for the JWST. Moreover, Montage is highly scalable and is in wide use within the IT community to develop advanced, fault-tolerant cyber-infrastructure, such as job schedulers for grids, workflow orchestration, and restructuring techniques for processing complex workflows and pipelines.
G. Bruce Berriman, Richard Cohen, Andrew Colson, Christopher Gelino, John Good, Mihseh Kong, Anastasia Laity, Jeffrey Mader, Melanie Swain, Hien Tran, Shin-Ywan Wang
The Keck Observatory Archive (KOA) (https://koa.ipac.caltech.edu) curates all observations acquired at the W. M. Keck
Observatory (WMKO) since it began operations in 1994, including data from eight active instruments and two
decommissioned instruments. The archive is a collaboration between WMKO and the NASA Exoplanet Science Institute
(NExScI). Since its inception in 2004, the science information system used at KOA has adopted an architectural
approach that emphasizes software re-use and adaptability. This paper describes how KOA is currently leveraging and
extending open source software components to develop new services and to support delivery of a complete set of
instrument metadata, which will enable more sophisticated and extensive queries than currently possible.
In August 2015, KOA deployed a program interface to discover public data from all instruments equipped with an
imaging mode. The interface complies with version 2 of the Simple Imaging Access Protocol (SIAP), under
development by the International Virtual Observatory Alliance (IVOA), which defines a standard mechanism for
discovering images through spatial queries. The heart of the KOA service is an R-tree-based, database-indexing
mechanism prototyped by the Virtual Astronomical Observatory (VAO) and further developed by the Montage Image
Mosaic project, designed to provide fast access to large imaging data sets as a first step in creating wide-area image
mosaics (such as mosaics of subsets of the 4.7 million images of the SDSS DR9 release). The KOA service uses the
results of the spatial R-tree search to create an SQLite data database for further relational filtering. The service uses a
JSON configuration file to describe the association between instrument parameters and the service query parameters, and to make it applicable beyond the Keck instruments.
The images generated at the Keck telescope usually do not encode the image footprints as WCS fields in the FITS file
headers. Because SIAP searches are spatial, much of the effort in developing the program interface involved processing the instrument and telescope parameters to understand how accurately we can derive the WCS information for each instrument. This knowledge is now being fed back into the KOA databases as part of a program to include complete metadata information for all imaging observations.
The R-tree program was itself extended to support temporal (in addition to spatial) indexing, in response to requests
from the planetary science community for a search engine to discover observations of Solar System objects. With this
3D-indexing scheme, the service performs very fast time and spatial matches between the target ephemerides, obtained from the JPL SPICE service. Our experiments indicate these matches can be more than 100 times faster than when separating temporal and spatial searches. Images of the tracks of the moving targets, overlaid with the image footprints, are computed with a new command-line visualization tool, mViewer, released with the Montage distribution. The service is currently in test and will be released in late summer 2016.
Thomas McGlynn, Giuseppina Fabbiano, Alberto Accomazzi, Alan Smale, Richard White, Thomas Donaldson, Alessandra Aloisi, Theresa Dower, Joseph Mazzerella, Rick Ebert, Olga Pevunova, David Imel, Graham Berriman, Harry Teplitz, Steve Groom, Vandana Desai, Walter Landry
KEYWORDS: Observatories, Astronomy, Data modeling, Data archive systems, Standards development, Data archive systems, Data modeling, X-rays, Infrared telescopes, Field emission displays, Gamma radiation, Ultraviolet radiation
Since the turn of the millennium a constant concern of astronomical archives have begun providing data to the public through standardized protocols unifying data from disparate physical sources and wavebands across the electromagnetic spectrum into an astronomical virtual observatory (VO). In October 2014, NASA began support for the NASA Astronomical Virtual Observatories (NAVO) program to coordinate the efforts of NASA astronomy archives in providing data to users through implementation of protocols agreed within the International Virtual Observatory Alliance (IVOA). A major goal of the NAVO collaboration has been to step back from a piecemeal implementation of IVOA standards and define what the appropriate presence for the US and NASA astronomy archives in the VO should be. This includes evaluating what optional capabilities in the standards need to be supported, the specific versions of standards that should be used, and returning feedback to the IVOA, to support modifications as needed.
We discuss a standard archive model developed by the NAVO for data archive presence in the virtual observatory built upon a consistent framework of standards defined by the IVOA. Our standard model provides for discovery of resources through the VO registries, access to observation and object data, downloads of image and spectral data and general access to archival datasets. It defines specific protocol versions, minimum capabilities, and all dependencies. The model will evolve as the capabilities of the virtual observatory and needs of the community change.
The Keck Observatory Archive (KOA) currently serves ~ 42 TB of data spanning over 20 years from all ten past and current facility instruments at Keck. Although most of the available data are in the raw form, for four instruments (HIRES, NIRC2, OSIRIS, LWS), quick-look, browse products generated by automated pipelines are also offered to facilitate assessment of the scientific content and quality of the data. KOA underwrote the update of the MAKEE package to support reduction of the CCD upgrade to HIRES, developed scripts for reduction of NIRC2 data and automated the existing OSIRIS and LWS data reduction packages. We describe in some detail the recently completed automated pipeline for NIRSPEC, which will be used to create browse products in KOA and made available for quicklook of the data by the observers at the telescope. We review the currently available data reduction tools for Keck data, and present our plans and anticipated priorities for the development of automated pipelines and release of reduced data products for the rest of the current and future instruments. We also anticipate that Keck's newest instrument, NIRES, which will be delivered with a fully automated pipeline, will be the first to have both raw and level-1 data ingested at commissioning.
A collaboration between the W. M. Keck Observatory (WMKO) in Hawaii and the NASA Exoplanet Science Institute (NExScI) in California, the Keck Observatory Archive (KOA) was commissioned in 2004 to archive observing data from WMKO, which operates two classically scheduled 10 m ground-based telescopes. The observing data from Keck is not suitable for direct ingestion into the archive since the metadata contained in the original FITS headers lack the information necessary for proper archiving. Coupled with different standards among instrument builders and the heterogeneous nature of the data inherent in classical observing, in which observers have complete control of the instruments and their observations, the data pose a number of technical challenges for KOA. For example, it is often difficult to determine if an observation is a science target, a sky frame, or a sky flat. It is also necessary to assign the data to the correct owners and observing programs, which can be a challenge for time-domain and target-of-opportunity observations, or on split nights, during which two or more principle investigators share a given night. In addition, having uniform and adequate calibrations are important for the proper reduction of data. Therefore, KOA needs to distinguish science files from calibration files, identify the type of calibrations available, and associate the appropriate calibration files with each science frame. We describe the methodologies and tools that we have developed to successfully address these difficulties, adding content to the FITS headers and "retrofitting" the metadata in order to support archiving Keck data, especially those obtained before the archive was designed. With the expertise gained from having successfully archived observations taken with all eight currently active instruments at WMKO, we have developed lessons learned from handling this complex array of heterogeneous metadata that help ensure a smooth ingestion of data not only for current but also future instruments, as well as a better experience for the archive user.
The Infrared Processing and Analysis Center (IPAC) and the W. M. Keck Observatory (WMKO) operate an archive for the Keck Observatory. At the end of 2013, KOA completed the ingestion of data from all eight active observatory instruments. KOA will continue to ingest all newly obtained observations, at an anticipated volume of 4 TB per year. The data are transmitted electronically from WMKO to IPAC for storage and curation. Access to data is governed by a data use policy, and approximately two-thirds of the data in the archive are public.
KEYWORDS: Data archive systems, Astronomy, Observatories, Data storage, Data modeling, Space telescopes, Databases, Standards development, Ecosystems, Data centers
The Virtual Observatory (VO) is realizing global electronic integration of astronomy data. One of the long-term goals of
the U.S. VO project, the Virtual Astronomical Observatory (VAO), is development of services and protocols that
respond to the growing size and complexity of astronomy data sets. This paper describes how VAO staff are active in
such development efforts, especially in innovative strategies and techniques that recognize the limited operating budgets
likely available to astronomers even as demand increases. The project has a program of professional outreach whereby
new services and protocols are evaluated.
The U.S. Virtual Astronomical Observatory (VAO http://www.us-vao.org/) has been in operation since May 2010. Its goal is to enable new science through efficient integration of distributed multi-wavelength data. This paper describes the management and organization of the VAO, and emphasizes the techniques used to ensure efficiency in a distributed organization. Management methods include using an annual program plan as the basis for establishing contracts with member organizations, regular communication, and monitoring of processes.
Operation of the US Virtual Astronomical Observatory shares some issues with modern physical observatories, e.g.,
intimidating data volumes and rapid technological change, and must also address unique concerns like the lack of direct
control of the underlying and scattered data resources, and the distributed nature of the observatory itself. In this paper
we discuss how the VAO has addressed these challenges to provide the astronomical community with a coherent set of
science-enabling tools and services. The distributed nature of our virtual observatory-with data and personnel
spanning geographic, institutional and regime boundaries-is simultaneously a major operational headache and the
primary science motivation for the VAO. Most astronomy today uses data from many resources. Facilitation of
matching heterogeneous datasets is a fundamental reason for the virtual observatory. Key aspects of our approach
include continuous monitoring and validation of VAO and VO services and the datasets provided by the community,
monitoring of user requests to optimize access, caching for large datasets, and providing distributed storage services that
allow user to collect results near large data repositories. Some elements are now fully implemented, while others are
planned for subsequent years. The distributed nature of the VAO requires careful attention to what can be a
straightforward operation at a conventional observatory, e.g., the organization of the web site or the collection and
combined analysis of logs. Many of these strategies use and extend protocols developed by the international virtual
observatory community. Our long-term challenge is working with the underlying data providers to ensure high quality
implementation of VO data access protocols (new and better 'telescopes'), assisting astronomical developers to build
robust integrating tools (new 'instruments'), and coordinating with the research community to maximize the science
enabled.
We have used the Montage image mosaic engine to investigate the cost and performance of processing images on the
Amazon EC2 cloud, and to inform the requirements that higher-level products impose on provenance management
technologies. We will present a detailed comparison of the performance of Montage on the cloud and on the Abe high
performance cluster at the National Center for Supercomputing Applications (NCSA). Because Montage generates many
intermediate products, we have used it to understand the science requirements that higher-level products impose on
provenance management technologies. We describe experiments with provenance management technologies such as the
"Provenance Aware Service Oriented Architecture" (PASOA).
IRSA's scaleable and extensible architecture is inherited by new missions and data providers, and thus offers substantial
cost savings to missions. It has built archives for the W.M. Keck Observatory & the Spitzer Space Telescope Legacy
teams, among others. It provided archiving and databases support for 2MASS, when active, and will provide
corresponding support for the forthcoming WISE mission. IRSA acts as a resource to projects and missions by advising
on product design and providing tools for validating data products.
This paper describes the design of a grid-enabled version of Montage, an astronomical image mosaic service, suitable for large scale processing of the sky. All the re-projection jobs can be added to a pool of tasks and performed by as many processors as are available, exploiting the parallelization inherent in the Montage architecture. We show how we can describe the Montage application in terms of an abstract workflow so that a planning tool such as Pegasus can derive an executable workflow that can be run in the Grid environment. The execution of the workflow is performed by the workflow manager DAGMan and the associated Condor-G. The grid processing will support tiling of images to a manageable size when the input images can no longer be held in memory. Montage will ultimately run operationally on the Teragrid. We describe science applications of Montage, including its application to science product generation by Spitzer Legacy Program teams and large-scale, all-sky image processing projects.
The National Virtual Observatory (NVO) will provide on-demand access to data collections, data fusion services and compute intensive applications. The paper describes the development of a framework that will support two key aspects of these objectives: a compute engine that will deliver custom image mosaics, and a "request management system," based on an e-business applications server, for job processing, including monitoring, failover and status reporting. We will develop this request management system to support a diverse range of astronomical requests, including services scaled to operate on the emerging computational grid infrastructure. Data requests will be made through existing portals to demonstrate the system: the NASA/IPAC Extragalactic Database (NED), the On-Line Archive Science Information Services (OASIS) at the NASA/IPAC Infrared Science Archive (IRSA); the Virtual Sky service at Caltech's Center for Advanced Computing Research (CACR), and the yourSky mosaic server at the Jet Propulsion Laboratory (JPL).
The Diffuse Infrared Background Experiment (DIRBE) on board NASA's Cosmic Background Explorer (COBE) satellite has surveyed the entire sky in 10 broad photometric bands covering the wavelength region from 1 to 240 micrometers , at an angular resolution of 0.7 degree(s) (Boggess et al. 1992). the extensive spectral coverage of the DIRBE observations offers an unprecedented opportunity to undertake comprehensive large-scale studies of the content, structure, and energetics of the stellar and interstellar components of the Galaxy. Understanding the Galactic emission is not only a task of scientific value in its own right, but also a necessary step in the accurate extraction of faint cosmological emission from the DIRBE data.
The Cosmic Background Explorer (COBE) satellite was launched on November 18, 1989 from Vandenberg Air Force base on a Delta rocket. It carried two superfluid liquid-helium-cooled (LHe) infrared (IR) instruments in a 600 liter dewar, and three microwave radiometers mounted on the outside of the dewar. One of the LHe-cooled instruments is a ten-band photometer covering the spectral range from 1.2 to 240 micrometers - the Diffuse Infrared Background Experiment (DIRBE). A goal of the DIRBE program is to obtain full-sky infrared observations that can be used to model accurately the IR contributions arising from the interplanetary dust (IPD) and the Galaxy. Using such models, the foreground can be removed to expose and underlying extragalactic IR component produced early in formation of the universe. The nature of the IPD IR foreground detected by the DIRBE is found to be quite complex, but amenable to modelling.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.