Search:

[Advanced Search]



Analyzing NICER Data on Sciserver

Overview

SciServer is a data analysis platform hosted at Johns Hopkins University. It consists of a data hosting server which includes the entire HEASARC archive, along with an analysis environment which includes NICERDAS, HEASoft and other analysis systems. SciServer allows users the ability to access and analyze any (or all) public NICER data without having to download data or install/maintain analysis software locally. This thread shows how to access SciServer.

Read this thread if you want to: Analyze NICER data using the SciServer science platform.

Last update: 2023-07-12

Introduction

It is often useful to access and analyze large amounts of NICER data. Large datasets include observations of a single bright source, or individual observations of multiple sources. It's also useful to be able to analyze NICER data without having to worry that your software environment has the latest software installed. It can be time consuming to download large amounts of data from the HEASARC archive to a local disk, and to install and maintain analysis software and calibrations.

The SciServer science platform provides a user with direct, on-line access to all public NICER data along with pre-built analysis software including HEASoft and NICERDAS. SciServer enables analysis of large amounts of NICER data (along with comparison with data from other missions archived at the HEASARC) in a simple way.

This science thread describes how to create a SciServer account and how to access the HEASARC SciServer environment so users can access and analyze NICER observations.

Accessing the HEASARC environment on Sciserver

Creating a SciServer Account

Users should first create their SciServer account. Once the account is created the user can log in at the Sciserver login page.

Creating a Virtual Container

Once the user has logged in, on the SciServer Dashboard page, the user should create a virtual container which includes the HEASARC environment (data archive and software). To create the virtual container:
  1. Once you log in, click on the "Compute" button Under "SciServer Apps" on the SciServer Dashboard page
  2. On the "Compute" page, click on the green "Create Container" button.
  3. In the form that appears after clicking the "Create Container" button, enter:
    • In the "Container Name" field, enter a memorable name for your container ("NICER Analysis", for example)
    • In the "Compute Image" menu, select "HEASARCv6.31.1-X" (note: select the most recent version of "HEASARC" listed; v6.31.1x indicates the 6.31.1 version of HEASoft is installed in this HEASARC environment; the "-X" indicates that the container allows Xwindows displays)
    • Under "User Volumes", click on the "All" checkbox
    • Under "Data volumes", click on the "HEASARC data" checkbox (and check any other data that might be helpful to you)
    • Click the green "Create" button at the bottom of the form to create your HEASARC environment on SciServer
SciServer will then display the container you just created in the "Containers" list on the "Compute" page.

Using the Virtual Container

To start the container, which will start the virtual environment with access to the HEASARC data archive and the HEASoft analysis software (including NICERDAS), click the name of the container you created in the container list on the "Compute" page. The container will start a jupyterlab server, and open a jupyterlab notebook.

You can analyze data entirely in a Jupyterlab notebook, or on the terminal command line (or some combination of notebook and command line):

  • perform your data access and analysis in the jupyterlab notebook (in python, for example). Users can use heasoftpy, a python interface to HEASoft and NICERDAS, to analyze data within a python workflow; and/or

  • To start a terminal session, select "New Launcher" in the "File" menu at the top of the jupyterlab notebook window, then click on "Terminal" in the launcher window that appears, and analyze your data on the terminal command line.

NICER data On SciServer

The HEASARC data archive is mounted on SciServer as /home/idies/workspace/headata/FTP NICER data is stored in subdirectories under /FTP/nicer/data/obs where the subdirectories are organized by date of observation. NICER data is stored by NICER OBSID under these directories, so (for example) the data directory for NICER OBSID 3050450119 is /FTP/nicer/data/obs/2020_07/3050450119 on SciServer.

User Directories

SciServer users with a username = <username> have access to the following user directories: /home/idies/workspace/Storage/<username>/persistent. This volume is for long-term storage but has a limited storage capacity. It is best used to store jupyterlab notebooks, other analysis scripts or other relatively small files which are meant to be available "permanently"; and /home/idies/workspace/Temporary/<username> for temporary storage of large amounts of data

It is recommended that users analysis scripts (and anything else they want to persist in newly-created containers) in the "persistent" directory.

Large amounts of data to be processed (for example, large numbers of NICER datasets to reprocess with nicerl2) can either be copied from the appropriate subdirectories of /home/idies/workspace/headata/FTP/nicer/data/obs/ to /home/idies/workspace/Temporary/<username>, or processed in place with the output directed to /home/idies/workspace/Temporary/<username>

Cookbooks & a NICER Tutorial Notebook

There are a number of jupyterlab notebooks available in subdirectories of /home/idies/workspace/headata/software/cookbooks. These notebooks provide walk throughs in python of useful analysis tasks, such as analyzing data with heasoftpy, accessing HEASARC data using VO protocols, and other useful tasks.

We suggest copying the cookbooks to your persistent area so you can modify them if desired and save the modifications. To do this, in a terminal window % cp -r /home/idies/workspace/headata/software/cookbooks /home/idies/workspace/Storage/<username>/persistent/ where, as before <username> is your SciServer username. To load a notebook, click on the folder icon in the leftmost menu of the SciServer jupyterlab window, then navigate to the appropriate subfolder of /home/idies/workspace/Storage/<username>/persistent/cookbooks and load the desired notebook.

The NICER tutorial is heasoftpy_notebooks/nicer-example.ipynb of the cookbooks directory. This notebook shows you how to

  • copy data of a particular OBSID from the NICER archive,
  • re-process it with nicerl2 using heasoftpy,
  • extract a spectrum and lightcurve,
  • fit and display the spectrum using pyxspec, the python version of XSPEC,
  • and how to display the lightcurve of the source.

Related Topics

More information about the HEASARC environment on SciServer is available at https://heasarc.gsfc.nasa.gov/docs/sciserver/.

Information on using heasoftpy is available at https://heasarc.gsfc.nasa.gov/lheasoft/heasoftpy.

Modifications

  • 2023-07-12 - MFC: initial draft