Update information on data organisation and download of data

Jürgen Knödlseder
1 parent eab4eddc
Showing 11 changed files with 313 additions and 126 deletions
doc/source/users/glossary.rst
doc/source/users/tutorials/1dc/data_organisation.rst
doc/source/users/tutorials/1dc/event_file.png
doc/source/users/tutorials/1dc/event_header.png
doc/source/users/tutorials/1dc/event_list.png
doc/source/users/tutorials/1dc/first.rst
doc/source/users/tutorials/1dc/first_data.rst
doc/source/users/tutorials/1dc/getting.rst
doc/source/users/tutorials/1dc/getting_data.rst
doc/source/users/tutorials/1dc/getting_tools.rst
doc/source/users/tutorials/1dc/index.rst
@@ -13,6 +13,12 @@ Glossary
  
    Bla bla ...
  
+.. _glossary_gti:
+
+.. topic:: Good Time Interval
+
+   Bla bla ...
+
 .. _glossary_irf:
  
 .. topic:: Instrument Response Functions
+.. _1dc_data_organisation:
+
+Data organisation
+-----------------
+
+The content of the ``1dc`` folder should be as follows:
+
+.. code-block:: bash
+
+   caldb/
+   caldb/data
+   caldb/data/cta
+   caldb/data/cta/prod3b
+   caldb/data/cta/prod3b/caldb.indx
+   caldb/data/cta/prod3b/bcf
+   ...
+   data/
+   data/gc_baseline_000001.fits
+   data/gc_baseline_000002.fits
+   ...
+   data/gc_threshold_000001.fits
+   data/gc_threshold_000002.fits
+   ...
+   data/
+   data/gps_baseline_000001.fits
+   data/gps_baseline_000002.fits
+   ...
+   data/gps_threshold_000001.fits
+   data/gps_threshold_000002.fits
+   ...
+   data/egal_baseline_000001.fits
+   data/egal_baseline_000002.fits
+   ...
+   data/egal_threshold_000001.fits
+   data/egal_threshold_000002.fits
+   ...
+   data/obs_gc_baseline.xml
+   data/obs_gc_threshold.xml
+   data/obs_gps_baseline.xml
+   data/obs_gps_threshold.xml
+   data/obs_egal_baseline.xml
+   data/obs_egal_threshold.xml
+   data/obs_all_baseline.xml
+   data/obs_all_threshold.xml
+   models/
+   models/models_gc.xml
+   models/models_gps.xml
+   models/models_egal.xml
+   models/map_RXJ1713.fits.gz
+   models/map_VelaJunior.fits.gz
+   models/map_ics.fits.gz
+   models/map_pi0.fits.gz
+
+The ``caldb`` folder contains the
+:ref:`Instrument Response Functions <glossary_irf>`
+that are necessary for the analysis of the simulated CTA data.
+The folder contains the ``prod3b`` response that should be used for the
+:ref:`First CTA Data Challenge <glossary_1dc>`.
+The following response functions are available:
+
+ +-----------------------+-------+---------------+--------+----------+
+ | Response name         | Site  | Configuration | Zenith | Duration |
+ +=======================+=======+===============+========+==========+
+ | ``South_z20_50h``     | South | Baseline      | 20 deg | 50 hours |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``South_z40_50h``     | South | Baseline      | 40 deg | 50 hours |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``South_z20_5h``      | South | Baseline      | 20 deg | 5 hours  |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``South_z40_5h``      | South | Baseline      | 40 deg | 5 hours  |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``South_z20_0.5h``    | South | Baseline      | 20 deg | 30 min   |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``South_z40_0.5h``    | South | Baseline      | 40 deg | 30 min   |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``North_z20_50h``     | North | Baseline      | 20 deg | 50 hours |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``North_z40_50h``     | North | Baseline      | 40 deg | 50 hours |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``North_z20_5h``      | North | Baseline      | 20 deg | 5 hours  |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``North_z40_5h``      | North | Baseline      | 40 deg | 5 hours  |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``North_z20_0.5h``    | North | Baseline      | 20 deg | 30 min   |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``North_z40_0.5h``    | North | Baseline      | 40 deg | 30 min   |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``South_TS_z20_50h``  | South | Threshold     | 20 deg | 50 hours |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``South_TS_z40_50h``  | South | Threshold     | 40 deg | 50 hours |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``South_TS_z20_5h``   | South | Threshold     | 20 deg | 5 hours  |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``South_TS_z40_5h``   | South | Threshold     | 40 deg | 5 hours  |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``South_TS_z20_0.5h`` | South | Threshold     | 20 deg | 30 min   |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``South_TS_z40_0.5h`` | South | Threshold     | 40 deg | 30 min   |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``North_TS_z20_50h``  | North | Threshold     | 20 deg | 50 hours |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``North_TS_z40_50h``  | North | Threshold     | 40 deg | 50 hours |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``North_TS_z20_5h``   | North | Threshold     | 20 deg | 5 hours  |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``North_TS_z40_5h``   | North | Threshold     | 40 deg | 5 hours  |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``North_TS_z20_0.5h`` | North | Threshold     | 20 deg | 30 min   |
+ +-----------------------+-------+---------------+--------+----------+
+ | ``North_TS_z40_0.5h`` | North | Threshold     | 40 deg | 30 min   |
+ +-----------------------+-------+---------------+--------+----------+
+
+The ``data`` folder contains the calibrated, reconstructed and background
+reduced event data that were procuded for the
+:ref:`First CTA Data Challenge <glossary_1dc>`
+and that were stored into FITS files.
+Each event file contains the events for an
+:ref:`observation <glossary_obs>`
+(or run) of 30 minutes duration and comprises an
+:ref:`event lists <glossary_eventlist>`
+and a
+:ref:`Good Time Interval <glossary_gti>`
+binary table extension (see figure below).
+
+.. figure:: event_file.png
+   :width: 600px
+   :align: center
+
+   *Structure of an event file*
+
+The header of the ``EVENTS`` table contains information about the
+:ref:`observation <glossary_obs>`
+such as
+the start and stop date and time,
+the duration and livetime of the observation, and
+the pointing direction in Right Ascension and Declination (see figure below).
+
+.. figure:: event_header.png
+   :width: 500px
+   :align: center
+
+   *Header of an event list*
+
+.. note::
+   The pointing direction during an observation is fixed.
+
+.. warning::
+   Only the following header keywords have meaningful values:
+
+   * ``DATE_OBS`` - start date of observation (UTC)
+   * ``TIME_OBS`` - start time of observation (UTC)
+   * ``DATE_END`` - end date of observation (UTC)
+   * ``TIME_END`` - end time of observation (UTC)
+   * ``TSTART`` - start time of observation, counted from time reference (s)
+   * ``TSTOP`` - stop time of observation, counted from time reference (s)
+   * ``MJDREFI`` - integer part of time reference MJD (days)
+   * ``MJDREFF`` - fractional part of time reference MJD (days)
+   * ``TIMEUNIT`` - time unit
+   * ``TIMESYS`` - time system
+   * ``TIMEREF`` - time reference
+   * ``TELAPSE`` - elapsed time (s)
+   * ``ONTIME`` - exposure time (s)
+   * ``LIVETIME`` - livetime (s)
+   * ``DEADC`` - deadtime correction factor, livetime / exposure time
+   * ``TIMEDEL`` - time resolution
+   * ``RA_PNT`` - Right Ascension of pointing direction (deg)
+   * ``DEC_PNT`` - Declination of pointing direction (deg)
+   * ``RADECSYS`` - Coordinate system
+   * ``EQUINOX`` - Coordinate epoch
+
+   All remaining header keywords have arbitrary values and should not be
+   used for the analysis.
+
+Each row of the ``EVENTS`` table corresponds to a single event.
+Each event is characterised by
+
+ +--------------+-------------------------------------------+----------+
+ | Column       | Meaning                                   | Unit     |
+ +==============+===========================================+==========+
+ | ``EVENT_ID`` | Event number in file                      | unitless |
+ +--------------+-------------------------------------------+----------+
+ | ``TIME``     | Time stamp, countered from time reference | s        |
+ +--------------+-------------------------------------------+----------+
+ | ``RA``       | Reconstructed Right Ascension             | deg      |
+ +--------------+-------------------------------------------+----------+
+ | ``DEC``      | Reconstructed Declination                 | deg      |
+ +--------------+-------------------------------------------+----------+
+ | ``ENERGY``   | Reconstructed energy                      | TeV      |
+ +--------------+-------------------------------------------+----------+
+ | ``DETX``     | Reconstructed camera X coordinate         | deg      |
+ +--------------+-------------------------------------------+----------+
+ | ``DETY``     | Reconstructed camera Y coordinate         | deg      |
+ +--------------+-------------------------------------------+----------+
+
+An example of an ``EVENTS`` table is shown below.
+
+.. figure:: event_list.png
+   :width: 600px
+   :align: center
+
+   *Content of an event list*
+
+Event files exist for
+
+ +-----------------+---------------+--------------------+
+ | Survey          | Configuration | Prefix             |
+ +=================+===============+====================+
+ | Galactic Centre | Baseline      | ``gc_baseline``    |
+ +-----------------+---------------+--------------------+
+ | Galactic Plane  | Baseline      | ``gps_baseline``   |
+ +-----------------+---------------+--------------------+
+ | Extragalactic   | Baseline      | ``egal_baseline``  |
+ +-----------------+---------------+--------------------+
+ | Galactic Centre | Threshold     | ``gc_threshold``   |
+ +-----------------+---------------+--------------------+
+ | Galactic Plane  | Threshold     | ``gps_threshold``  |
+ +-----------------+---------------+--------------------+
+ | Extragalactic   | Threshold     | ``egal_threshold`` |
+ +-----------------+---------------+--------------------+
+
+The files ``obs_XXX_YYYYY.xml`` are so called
+:ref:`Observation Definition Files <glossary_obsdef>`
+that contain the information (or metadata) of a list of observations.
+The files are plain ASCII files in XML format that be inspected and
+manipulated by any text editor.
+The following files are included in the data distribution:
+
+ +----------------------------+--------------------------------------------------------+
+ | File                       | Purpose                                                |
+ +============================+========================================================+
+ | ``obs_gc_baseline.xml``    | Galactic Centre Survey performed with baseline arrays  |
+ +----------------------------+--------------------------------------------------------+
+ | ``obs_gc_threshold.xml``   | Galactic Centre Survey performed with threshold arrays |
+ +----------------------------+--------------------------------------------------------+
+ | ``obs_gps_baseline.xml``   | Galactic Plane Survey performed with baseline arrays   |
+ +----------------------------+--------------------------------------------------------+
+ | ``obs_gps_threshold.xml``  | Galactic Plane Survey performed with threshold arrays  |
+ +----------------------------+--------------------------------------------------------+
+ | ``obs_egal_baseline.xml``  | Extragalactic Survey performed with baseline arrays    |
+ +----------------------------+--------------------------------------------------------+
+ | ``obs_egal_threshold.xml`` | Extragalactic Survey performed with threshold arrays   |
+ +----------------------------+--------------------------------------------------------+
+ | ``obs_all_baseline.xml``   | All surveys combined performed with baseline arrays    |
+ +----------------------------+--------------------------------------------------------+
+ | ``obs_all_threshold.xml``  | All surveys combined performed with threshold arrays   |
+ +----------------------------+--------------------------------------------------------+
+
+The ``models`` folder contains the definitions of all source and background
+models that have been used for simulating the data.
+The file ``models_gc.xml`` is a so called
+:ref:`Model Definition File <glossary_moddef>`
+that collects the definition of all model components used for the Galactic
+Centre Survey simulation.
+The files ``models_gps.xml`` and ``models_egal.xml`` are the equivalent files
+for the Galactic Plane Survey and the Extragalactic Survey.
+The other files in the folder are FITS files containing sky maps that were
+used as spatial templates for modelling extended or diffuse emission components.
+There files are reference in the
+:ref:`Model Definition File <glossary_moddef>`
+files.
+
+
+
@@ -14,11 +14,12 @@ We assume that you have downloaded the data of the
 :ref:`First CTA Data Challenge <glossary_1dc>`
 and that you have successfully installed the ctools package.
 If this is not the case, please read :ref:`1dc_getting`.
+We also assume that you have read the section about
+:ref:`1dc_data_organisation`.
  
 .. toctree::
    :maxdepth: 2
  
-   first_data
    first_select_obs
    first_skymap
    first_detect
-.. _1dc_first_data:
-
-Data organisation
------------------
-
-After downloading the
-:ref:`First CTA Data Challenge <glossary_1dc>`
-data from **TBD** and uncompressing the tarball using
-
-.. code-block:: bash
-
-   $ tar xfvz 1dc.tar.gz
-
-you will have a directory ``1dc`` on your disk with the following content:
-
-.. code-block:: bash
-
-   caldb/
-   caldb/data
-   caldb/data/cta
-   caldb/data/cta/prod2
-   ...
-   caldb/data/cta/prod3
-   caldb/data/cta/prod3/caldb.indx
-   caldb/data/cta/prod3/bcf
-   ...
-   data/
-   data/gps_baseline_000001.fits
-   data/gps_baseline_000002.fits
-   data/gps_baseline_000003.fits
-   ...
-   data/gps_threshold_000001.fits
-   data/gps_threshold_000002.fits
-   data/gps_threshold_000003.fits
-   ...
-   data/egal_baseline_000001.fits
-   data/egal_baseline_000002.fits
-   data/egal_baseline_000003.fits
-   ...
-   data/egal_threshold_000001.fits
-   data/egal_threshold_000002.fits
-   data/egal_threshold_000003.fits
-   ...
-   data/obs_gps_baseline.xml
-   data/obs_gps_threshold.xml
-   data/obs_egal_baseline.xml
-   data/obs_egal_threshold.xml
-   data/obs_all_baseline.xml
-   data/obs_all_threshold.xml
-   models/
-   models/models_gps.xml
-   models/models_egal.xml
-   models/map_RXJ1713.fits.gz
-   models/map_VelaJunior.fits.gz
-   models/map_ics.fits.gz
-   models/map_pi0.fits.gz
-
-The ``caldb`` folder contains the
-:ref:`Instrument Response Functions <glossary_irf>`
-that are necessary for the analysis of the simulated CTA data.
-The folder contains the ``prod3`` response that should be used for the
-:ref:`First CTA Data Challenge <glossary_1dc>`.
-For legacy reasons, also the ``prod2`` response functions are shipped with
-the tarball.
-
-The ``data`` folder contains the calibrated, reconstructed and background
-reduced event data that were procuded for the
-:ref:`First CTA Data Challenge <glossary_1dc>`.
-Each event file corresponds to the
-:ref:`event lists <glossary_eventlist>`
-for an
-:ref:`observation (or run) <glossary_obs>`, and contains for each event a
-time stamp, the reconstructued photon arrival direction, and the reconstructed
-photon energy.
-The pointing direction during an observation is fixed.
-Event files exist for the Galactic Plane Survey (starting with ``gps_``) and
-for the extragalactic survey (starting with ``egal_``) for both
-the ``baseline`` configuration and the ``threshold`` implementation.
-
-The files ``obs_XXX_YYYYY.xml`` are so called
-:ref:`Observation Definition Files <glossary_obsdef>`
-that contain the information (or metadata) of a list of observations.
-The files are plain ASCII files in XML format that be inspected and
-manipulated by any text editor.
-The following files are included in the data distribution:
-
-* ``obs_gps_baseline.xml``: Galactic Plane Survey performed with baseline arrays
-* ``obs_gps_threshold.xml``: Galactic Plane Survey performed with threshold implementation
-* ``obs_egal_baseline.xml``: Extragalactic Survey performed with baseline arrays
-* ``obs_egal_threshold.xml``: Extragalactic Survey performed with threshold implementation
-* ``obs_all_baseline.xml``: Both surveys performed with baseline arrays
-* ``obs_all_threshold.xml``: Both surveys performed with threshold implementation
-
-The ``models`` folder contains the definitions of all source and background
-models that have been used for simulating the data.
-The file ``models_gps.xml`` is a so called
-:ref:`Model Definition File <glossary_moddef>`
-that collects the definition of all model components used for the Galactic
-Plane Survey simulation.
-The file ``models_egal.xml`` is the equivalent file for the Extragalactic
-Survey.
-The other files in the folder are FITS files containing sky maps that were
-used as spatial templates for modelling extended or diffuse emission components.
-There files are reference in the
-:ref:`Model Definition File <glossary_moddef>`
-files.
-
-
-
-.. _1dc_getting:
-
-Getting the data and analysis tools
-===================================
-
-Getting the data
-----------------
-
-To be written ...
-
-
-Getting the analysis tools
---------------------------
-
-To be written ...
+.. _1dc_getting_data:
+
+Getting the data
+================
+
+The data for the
+:ref:`First CTA Data Challenge <glossary_1dc>`
+are bundled with the
+:ref:`Instrument Response Functions <glossary_irf>`
+into a single tarball.
+Click on
+`1dc.tar.gz <https://www.dropbox.com/s/2aru9gak10h9su6/1dc.tar.gz?dl=0>`_
+to download the tarball (file size: 485 MB).
+You can then uncompress and deploy the data at any place by typing
+
+.. code-block:: bash
+
+   $ tar xfvz 1dc.tar.gz
+
+.. warning::
+   Don't type the ``$`` symbol. It indicates in the following that a command
+   should be typed on the command line of a console.
+
+After deploying you should have the folder ``1dc`` in your current working
+directory. Within that directory, please set the following environment variables:
+
+.. code-block:: bash
+
+   $ export CALDB=$PWD/1dc/caldb
+   $ export CTADATA=$PWD/1dc/data
+.. _1dc_getting_tools:
+
+Getting the analysis tools
+==========================
+
+We recommend that you use ctools to analyse the data from the
+:ref:`First CTA Data Challenge <glossary_1dc>`.
+Please read :ref:`getting the ctools <sec_getting>` to learn how to
+install and configure the ctools package.
@@ -7,6 +7,8 @@ First CTA Data Challenge
    :maxdepth: 2
  
    scope
-   getting
+   getting_tools
+   getting_data
+   data_organisation
    first