CURRENT DSDS OPERATIONS

DSDS Operators
Physical Storage
Applications
Overview of Current Functions
Backup and Recovery Procedures

DSDS Operators (1-1.25 FTE)

Sean McManus (Database Administrator)

  • Systems administration for DSDS hardware and Oracle database software.

  • Writes & updates software to facilitate the archive and distribution of GONG data, including various pipeline and instrument support functions.

  • Processes data requests. Pulls tapes for internal users.

  • Performs manual queries to the database to process "special" data requests.

  • Performs manual updates to the database when automated applications fail.

  • Assists PIPE operators with archiving GONG data.
  • Creates and updates HTTP, CGI, and Perl applications for the GONG web page.

  • Maintains tape vault. Copies archive tapes for offsite storage.


Greg Ladd (Backup DSDS Operator)

  • Processes data requests. Pulls tapes for internal users.

  • Assists PIPE operators with archiving GONG data.
  • Performs media read testing.

  • Copies archive tapes for offsite storage, maintains offsite storage vault.

Physical Storage

The current DSDS storage facility consists of three media layers, to support the archive, distribution and backup/recovery of GONG data products:

1) Online: 64 GB available for GONG+ data products. 32 GB reserved for the Instrument Header Database.
2) Offline: GONG Classic Exabyte tapes, GONG+ DLT tapes.
3) Off site storage (Kitt Peak vault): Contains duplicates of all archived tape media.

Applications

Most DSDS functions are handled by custom applications written in PL/SQL, a proprietary "C" language that communicates with the Oracle Database. Also, some plain "C" applications are used for library functions that do not need direct database connectivity. C-shell is used as the CGI interface between PL/SQL database applications and HTML web pages. All source code is backed up daily.

Overview of Current Functions

The procedures below represent the basic day-to-day functions of the DSDS, to support the data reduction pipeline, and the archive and distribution of GONG data. The purpose of this section is to familiarize the reader with the DSDS, and to establish a context for possible improvements to the DSDS.

Archiving instrument header data

1) Electronically transferred instrument parameters from each site are staged daily on Mojave.
2) DSDS downloads the files and inserts the data into the Instrument Header Database.
3) FTR provides the DSDS with daytime image headers.
4) The DSDS archives the daytime image headers, replacing electronically transferred daytime data, while retaining nighttime information.

Archiving GONG data to online storage

1) PIPE operator submits a transfer request to the DSDS, which points to a TOC (Table of Contents) file on the user's workstation.
2) DSDS operator is notified of the transfer request via email.
3) DSDS operator processes the request using a GUI interface that displays pending requests.
4) Oracle applications automatically update all of the tables necessary to reflect the presence of new products.

Archiving GONG data directly to tape

1) New, labeled tape (with VSN, but no description label) physically arrives in DSDS inbox.
2) DSDS operator checks the tape in.
3) DSDS detects new tape automatically. It copies the TOC and Label files from the PIPE operator's workstation, and proceeds to archive the data.
4) DSDS operator makes a copy of the tape, and updates the database to reflect the copy.
5) A descriptive label is printed and adhered to the tape for identification.

Transferring online archive to tape

1) Every 6 months, or when the disk is nearly full, a reminder is sent to a DSDS operator to write a set of files to permanent tape archive and purge them from the online disk
2) Scripts are automatically generated to write the tape and update the database with the new file locations.
3) DSDS operator runs the scripts, checks the tape and makes a backup tape.
4) A script is generated to delete the appropriate files from the server, and is executed by the DSDS operator.

Data product availability and query web page

1) When a new data product is archived, the DSDS updates a "bitmap" file, reflecting the presence of the new product. Bitmap files are indexes that store dates and times for each type of data product. The files are useful for facilitating quick access to the DSDS file catalogue without running a full table scan.
2) Updated bitmap files are copied to the web server (Argo).
3) A group of applications on the GONG web page read the bitmaps, present data product availability, and provide query functions for retrieval of data products according to user-specified parameters.

Data distributions

1) Internal or external user requests data from GONG web page.
2) DSDS operator is notified of the data request via email.
3) DSDS automatically checks if the data is online. If not, it tells the operator which tape(s) to mount.
4) Tapes are manually mounted. Operator executes script that completes the data request.

Off-site Copy and Storage Facilities (OCF/OSF)

The OCF produces duplicate copies of all archived data products and stores them in the OSF located in the basement of the Kitt Peak Vacuum Telescope building. Greg Ladd periodically adds new tapes to the OSF, and retrieves older tapes for media testing:

1) Through a GUI interface, DSDS operator enters parameters to search for tapes that require media testing.
2) DSDS returns a list of tapes to check-out sorted by the last quality check date.
3) DSDS operator performs media testing, checks the tapes in, and last quality check date is updated automatically by the database.

Per Greg Ladd:

Records show that for the mountain (OSF) tape collection, we have brought down and tested approximately 2160 tapes since the beginning of May 2001. Records for the tape testing of the "downtown" (DMAC) tapes have not been copiously kept during the last year, but it is likely that 1000 tapes have been tested during that time. We have had to replace (copy) 27 tapes during that time, so it would appear that we're experiencing a failure rate of about 1 in 100 tapes. All but four of the failing tapes were from the OSF, and of those, 20 of the 23 were field tape originals, the others being data product tape copies. Further, the failing tapes were TYPICALLY of a vintage of four years and older with only a couple tapes falling into the 36-48 month category.

The "stalest" (longest since the tape was originally written or last tested) tapes presently being tested are right at 48 months for the OSF tapes and 47 months for the DMAC tapes.

We have added to the OSF in the last year 233 Exabyte-format field and product tapes, and 308 DLT-format field and product tapes.

Other pipeline support

Tape inventory

DSDS operators are responsible for physically checking tapes in/out of the library for PIPE usage, providing labeled blank volumes for new data products, and printing descriptive labels of new data products.

Production of reports

Various reporting mechanisms are available to keep PIPE operators, scientists, and managers informed of the Pipeline status, archive and distribution statistics.

PIPE operators, scientists, or managers may request custom reports from the DSDS, in which case the DSDS operator will create the report with Structured Query Language (SQL).

If a report needs to be generated frequently, a new GUI interface is designed and integrated into the DSDS operators menu.