From c8af36c98c71b3c4dcc5ec50c3276a17d6e3c77a Mon Sep 17 00:00:00 2001 From: Jürgen Knödlseder Date: Thu, 2 Feb 2017 14:24:07 +0100 Subject: [PATCH] Update information on data organisation and download of data --- doc/source/users/glossary.rst | 6 ++++++ doc/source/users/tutorials/1dc/data_organisation.rst | 263 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ doc/source/users/tutorials/1dc/event_file.png | Bin 0 -> 19318 bytes doc/source/users/tutorials/1dc/event_header.png | Bin 0 -> 64707 bytes doc/source/users/tutorials/1dc/event_list.png | Bin 0 -> 88142 bytes doc/source/users/tutorials/1dc/first.rst | 3 ++- doc/source/users/tutorials/1dc/first_data.rst | 109 ------------------------------------------------------------------------------------------------------------- doc/source/users/tutorials/1dc/getting.rst | 15 --------------- doc/source/users/tutorials/1dc/getting_data.rst | 30 ++++++++++++++++++++++++++++++ doc/source/users/tutorials/1dc/getting_tools.rst | 9 +++++++++ doc/source/users/tutorials/1dc/index.rst | 4 +++- 11 files changed, 313 insertions(+), 126 deletions(-) create mode 100644 doc/source/users/tutorials/1dc/data_organisation.rst create mode 100644 doc/source/users/tutorials/1dc/event_file.png create mode 100644 doc/source/users/tutorials/1dc/event_header.png create mode 100644 doc/source/users/tutorials/1dc/event_list.png delete mode 100644 doc/source/users/tutorials/1dc/first_data.rst delete mode 100644 doc/source/users/tutorials/1dc/getting.rst create mode 100644 doc/source/users/tutorials/1dc/getting_data.rst create mode 100644 doc/source/users/tutorials/1dc/getting_tools.rst diff --git a/doc/source/users/glossary.rst b/doc/source/users/glossary.rst index c31e721..87ba50c 100644 --- a/doc/source/users/glossary.rst +++ b/doc/source/users/glossary.rst @@ -13,6 +13,12 @@ Glossary Bla bla ... +.. _glossary_gti: + +.. topic:: Good Time Interval + + Bla bla ... + .. _glossary_irf: .. topic:: Instrument Response Functions diff --git a/doc/source/users/tutorials/1dc/data_organisation.rst b/doc/source/users/tutorials/1dc/data_organisation.rst new file mode 100644 index 0000000..dc3e661 --- /dev/null +++ b/doc/source/users/tutorials/1dc/data_organisation.rst @@ -0,0 +1,263 @@ +.. _1dc_data_organisation: + +Data organisation +----------------- + +The content of the ``1dc`` folder should be as follows: + +.. code-block:: bash + + caldb/ + caldb/data + caldb/data/cta + caldb/data/cta/prod3b + caldb/data/cta/prod3b/caldb.indx + caldb/data/cta/prod3b/bcf + ... + data/ + data/gc_baseline_000001.fits + data/gc_baseline_000002.fits + ... + data/gc_threshold_000001.fits + data/gc_threshold_000002.fits + ... + data/ + data/gps_baseline_000001.fits + data/gps_baseline_000002.fits + ... + data/gps_threshold_000001.fits + data/gps_threshold_000002.fits + ... + data/egal_baseline_000001.fits + data/egal_baseline_000002.fits + ... + data/egal_threshold_000001.fits + data/egal_threshold_000002.fits + ... + data/obs_gc_baseline.xml + data/obs_gc_threshold.xml + data/obs_gps_baseline.xml + data/obs_gps_threshold.xml + data/obs_egal_baseline.xml + data/obs_egal_threshold.xml + data/obs_all_baseline.xml + data/obs_all_threshold.xml + models/ + models/models_gc.xml + models/models_gps.xml + models/models_egal.xml + models/map_RXJ1713.fits.gz + models/map_VelaJunior.fits.gz + models/map_ics.fits.gz + models/map_pi0.fits.gz + +The ``caldb`` folder contains the +:ref:`Instrument Response Functions ` +that are necessary for the analysis of the simulated CTA data. +The folder contains the ``prod3b`` response that should be used for the +:ref:`First CTA Data Challenge `. +The following response functions are available: + + +-----------------------+-------+---------------+--------+----------+ + | Response name | Site | Configuration | Zenith | Duration | + +=======================+=======+===============+========+==========+ + | ``South_z20_50h`` | South | Baseline | 20 deg | 50 hours | + +-----------------------+-------+---------------+--------+----------+ + | ``South_z40_50h`` | South | Baseline | 40 deg | 50 hours | + +-----------------------+-------+---------------+--------+----------+ + | ``South_z20_5h`` | South | Baseline | 20 deg | 5 hours | + +-----------------------+-------+---------------+--------+----------+ + | ``South_z40_5h`` | South | Baseline | 40 deg | 5 hours | + +-----------------------+-------+---------------+--------+----------+ + | ``South_z20_0.5h`` | South | Baseline | 20 deg | 30 min | + +-----------------------+-------+---------------+--------+----------+ + | ``South_z40_0.5h`` | South | Baseline | 40 deg | 30 min | + +-----------------------+-------+---------------+--------+----------+ + | ``North_z20_50h`` | North | Baseline | 20 deg | 50 hours | + +-----------------------+-------+---------------+--------+----------+ + | ``North_z40_50h`` | North | Baseline | 40 deg | 50 hours | + +-----------------------+-------+---------------+--------+----------+ + | ``North_z20_5h`` | North | Baseline | 20 deg | 5 hours | + +-----------------------+-------+---------------+--------+----------+ + | ``North_z40_5h`` | North | Baseline | 40 deg | 5 hours | + +-----------------------+-------+---------------+--------+----------+ + | ``North_z20_0.5h`` | North | Baseline | 20 deg | 30 min | + +-----------------------+-------+---------------+--------+----------+ + | ``North_z40_0.5h`` | North | Baseline | 40 deg | 30 min | + +-----------------------+-------+---------------+--------+----------+ + | ``South_TS_z20_50h`` | South | Threshold | 20 deg | 50 hours | + +-----------------------+-------+---------------+--------+----------+ + | ``South_TS_z40_50h`` | South | Threshold | 40 deg | 50 hours | + +-----------------------+-------+---------------+--------+----------+ + | ``South_TS_z20_5h`` | South | Threshold | 20 deg | 5 hours | + +-----------------------+-------+---------------+--------+----------+ + | ``South_TS_z40_5h`` | South | Threshold | 40 deg | 5 hours | + +-----------------------+-------+---------------+--------+----------+ + | ``South_TS_z20_0.5h`` | South | Threshold | 20 deg | 30 min | + +-----------------------+-------+---------------+--------+----------+ + | ``South_TS_z40_0.5h`` | South | Threshold | 40 deg | 30 min | + +-----------------------+-------+---------------+--------+----------+ + | ``North_TS_z20_50h`` | North | Threshold | 20 deg | 50 hours | + +-----------------------+-------+---------------+--------+----------+ + | ``North_TS_z40_50h`` | North | Threshold | 40 deg | 50 hours | + +-----------------------+-------+---------------+--------+----------+ + | ``North_TS_z20_5h`` | North | Threshold | 20 deg | 5 hours | + +-----------------------+-------+---------------+--------+----------+ + | ``North_TS_z40_5h`` | North | Threshold | 40 deg | 5 hours | + +-----------------------+-------+---------------+--------+----------+ + | ``North_TS_z20_0.5h`` | North | Threshold | 20 deg | 30 min | + +-----------------------+-------+---------------+--------+----------+ + | ``North_TS_z40_0.5h`` | North | Threshold | 40 deg | 30 min | + +-----------------------+-------+---------------+--------+----------+ + +The ``data`` folder contains the calibrated, reconstructed and background +reduced event data that were procuded for the +:ref:`First CTA Data Challenge ` +and that were stored into FITS files. +Each event file contains the events for an +:ref:`observation ` +(or run) of 30 minutes duration and comprises an +:ref:`event lists ` +and a +:ref:`Good Time Interval ` +binary table extension (see figure below). + +.. figure:: event_file.png + :width: 600px + :align: center + + *Structure of an event file* + +The header of the ``EVENTS`` table contains information about the +:ref:`observation ` +such as +the start and stop date and time, +the duration and livetime of the observation, and +the pointing direction in Right Ascension and Declination (see figure below). + +.. figure:: event_header.png + :width: 500px + :align: center + + *Header of an event list* + +.. note:: + The pointing direction during an observation is fixed. + +.. warning:: + Only the following header keywords have meaningful values: + + * ``DATE_OBS`` - start date of observation (UTC) + * ``TIME_OBS`` - start time of observation (UTC) + * ``DATE_END`` - end date of observation (UTC) + * ``TIME_END`` - end time of observation (UTC) + * ``TSTART`` - start time of observation, counted from time reference (s) + * ``TSTOP`` - stop time of observation, counted from time reference (s) + * ``MJDREFI`` - integer part of time reference MJD (days) + * ``MJDREFF`` - fractional part of time reference MJD (days) + * ``TIMEUNIT`` - time unit + * ``TIMESYS`` - time system + * ``TIMEREF`` - time reference + * ``TELAPSE`` - elapsed time (s) + * ``ONTIME`` - exposure time (s) + * ``LIVETIME`` - livetime (s) + * ``DEADC`` - deadtime correction factor, livetime / exposure time + * ``TIMEDEL`` - time resolution + * ``RA_PNT`` - Right Ascension of pointing direction (deg) + * ``DEC_PNT`` - Declination of pointing direction (deg) + * ``RADECSYS`` - Coordinate system + * ``EQUINOX`` - Coordinate epoch + + All remaining header keywords have arbitrary values and should not be + used for the analysis. + +Each row of the ``EVENTS`` table corresponds to a single event. +Each event is characterised by + + +--------------+-------------------------------------------+----------+ + | Column | Meaning | Unit | + +==============+===========================================+==========+ + | ``EVENT_ID`` | Event number in file | unitless | + +--------------+-------------------------------------------+----------+ + | ``TIME`` | Time stamp, countered from time reference | s | + +--------------+-------------------------------------------+----------+ + | ``RA`` | Reconstructed Right Ascension | deg | + +--------------+-------------------------------------------+----------+ + | ``DEC`` | Reconstructed Declination | deg | + +--------------+-------------------------------------------+----------+ + | ``ENERGY`` | Reconstructed energy | TeV | + +--------------+-------------------------------------------+----------+ + | ``DETX`` | Reconstructed camera X coordinate | deg | + +--------------+-------------------------------------------+----------+ + | ``DETY`` | Reconstructed camera Y coordinate | deg | + +--------------+-------------------------------------------+----------+ + +An example of an ``EVENTS`` table is shown below. + +.. figure:: event_list.png + :width: 600px + :align: center + + *Content of an event list* + +Event files exist for + + +-----------------+---------------+--------------------+ + | Survey | Configuration | Prefix | + +=================+===============+====================+ + | Galactic Centre | Baseline | ``gc_baseline`` | + +-----------------+---------------+--------------------+ + | Galactic Plane | Baseline | ``gps_baseline`` | + +-----------------+---------------+--------------------+ + | Extragalactic | Baseline | ``egal_baseline`` | + +-----------------+---------------+--------------------+ + | Galactic Centre | Threshold | ``gc_threshold`` | + +-----------------+---------------+--------------------+ + | Galactic Plane | Threshold | ``gps_threshold`` | + +-----------------+---------------+--------------------+ + | Extragalactic | Threshold | ``egal_threshold`` | + +-----------------+---------------+--------------------+ + +The files ``obs_XXX_YYYYY.xml`` are so called +:ref:`Observation Definition Files ` +that contain the information (or metadata) of a list of observations. +The files are plain ASCII files in XML format that be inspected and +manipulated by any text editor. +The following files are included in the data distribution: + + +----------------------------+--------------------------------------------------------+ + | File | Purpose | + +============================+========================================================+ + | ``obs_gc_baseline.xml`` | Galactic Centre Survey performed with baseline arrays | + +----------------------------+--------------------------------------------------------+ + | ``obs_gc_threshold.xml`` | Galactic Centre Survey performed with threshold arrays | + +----------------------------+--------------------------------------------------------+ + | ``obs_gps_baseline.xml`` | Galactic Plane Survey performed with baseline arrays | + +----------------------------+--------------------------------------------------------+ + | ``obs_gps_threshold.xml`` | Galactic Plane Survey performed with threshold arrays | + +----------------------------+--------------------------------------------------------+ + | ``obs_egal_baseline.xml`` | Extragalactic Survey performed with baseline arrays | + +----------------------------+--------------------------------------------------------+ + | ``obs_egal_threshold.xml`` | Extragalactic Survey performed with threshold arrays | + +----------------------------+--------------------------------------------------------+ + | ``obs_all_baseline.xml`` | All surveys combined performed with baseline arrays | + +----------------------------+--------------------------------------------------------+ + | ``obs_all_threshold.xml`` | All surveys combined performed with threshold arrays | + +----------------------------+--------------------------------------------------------+ + +The ``models`` folder contains the definitions of all source and background +models that have been used for simulating the data. +The file ``models_gc.xml`` is a so called +:ref:`Model Definition File ` +that collects the definition of all model components used for the Galactic +Centre Survey simulation. +The files ``models_gps.xml`` and ``models_egal.xml`` are the equivalent files +for the Galactic Plane Survey and the Extragalactic Survey. +The other files in the folder are FITS files containing sky maps that were +used as spatial templates for modelling extended or diffuse emission components. +There files are reference in the +:ref:`Model Definition File ` +files. + + + diff --git a/doc/source/users/tutorials/1dc/event_file.png b/doc/source/users/tutorials/1dc/event_file.png new file mode 100644 index 0000000..4c9f1e6 Binary files /dev/null and b/doc/source/users/tutorials/1dc/event_file.png differ diff --git a/doc/source/users/tutorials/1dc/event_header.png b/doc/source/users/tutorials/1dc/event_header.png new file mode 100644 index 0000000..de5d93f Binary files /dev/null and b/doc/source/users/tutorials/1dc/event_header.png differ diff --git a/doc/source/users/tutorials/1dc/event_list.png b/doc/source/users/tutorials/1dc/event_list.png new file mode 100644 index 0000000..d7014c7 Binary files /dev/null and b/doc/source/users/tutorials/1dc/event_list.png differ diff --git a/doc/source/users/tutorials/1dc/first.rst b/doc/source/users/tutorials/1dc/first.rst index d4f2d2d..51d3884 100644 --- a/doc/source/users/tutorials/1dc/first.rst +++ b/doc/source/users/tutorials/1dc/first.rst @@ -14,11 +14,12 @@ We assume that you have downloaded the data of the :ref:`First CTA Data Challenge ` and that you have successfully installed the ctools package. If this is not the case, please read :ref:`1dc_getting`. +We also assume that you have read the section about +:ref:`1dc_data_organisation`. .. toctree:: :maxdepth: 2 - first_data first_select_obs first_skymap first_detect diff --git a/doc/source/users/tutorials/1dc/first_data.rst b/doc/source/users/tutorials/1dc/first_data.rst deleted file mode 100644 index 17363fa..0000000 --- a/doc/source/users/tutorials/1dc/first_data.rst +++ /dev/null @@ -1,109 +0,0 @@ -.. _1dc_first_data: - -Data organisation ------------------ - -After downloading the -:ref:`First CTA Data Challenge ` -data from **TBD** and uncompressing the tarball using - -.. code-block:: bash - - $ tar xfvz 1dc.tar.gz - -you will have a directory ``1dc`` on your disk with the following content: - -.. code-block:: bash - - caldb/ - caldb/data - caldb/data/cta - caldb/data/cta/prod2 - ... - caldb/data/cta/prod3 - caldb/data/cta/prod3/caldb.indx - caldb/data/cta/prod3/bcf - ... - data/ - data/gps_baseline_000001.fits - data/gps_baseline_000002.fits - data/gps_baseline_000003.fits - ... - data/gps_threshold_000001.fits - data/gps_threshold_000002.fits - data/gps_threshold_000003.fits - ... - data/egal_baseline_000001.fits - data/egal_baseline_000002.fits - data/egal_baseline_000003.fits - ... - data/egal_threshold_000001.fits - data/egal_threshold_000002.fits - data/egal_threshold_000003.fits - ... - data/obs_gps_baseline.xml - data/obs_gps_threshold.xml - data/obs_egal_baseline.xml - data/obs_egal_threshold.xml - data/obs_all_baseline.xml - data/obs_all_threshold.xml - models/ - models/models_gps.xml - models/models_egal.xml - models/map_RXJ1713.fits.gz - models/map_VelaJunior.fits.gz - models/map_ics.fits.gz - models/map_pi0.fits.gz - -The ``caldb`` folder contains the -:ref:`Instrument Response Functions ` -that are necessary for the analysis of the simulated CTA data. -The folder contains the ``prod3`` response that should be used for the -:ref:`First CTA Data Challenge `. -For legacy reasons, also the ``prod2`` response functions are shipped with -the tarball. - -The ``data`` folder contains the calibrated, reconstructed and background -reduced event data that were procuded for the -:ref:`First CTA Data Challenge `. -Each event file corresponds to the -:ref:`event lists ` -for an -:ref:`observation (or run) `, and contains for each event a -time stamp, the reconstructued photon arrival direction, and the reconstructed -photon energy. -The pointing direction during an observation is fixed. -Event files exist for the Galactic Plane Survey (starting with ``gps_``) and -for the extragalactic survey (starting with ``egal_``) for both -the ``baseline`` configuration and the ``threshold`` implementation. - -The files ``obs_XXX_YYYYY.xml`` are so called -:ref:`Observation Definition Files ` -that contain the information (or metadata) of a list of observations. -The files are plain ASCII files in XML format that be inspected and -manipulated by any text editor. -The following files are included in the data distribution: - -* ``obs_gps_baseline.xml``: Galactic Plane Survey performed with baseline arrays -* ``obs_gps_threshold.xml``: Galactic Plane Survey performed with threshold implementation -* ``obs_egal_baseline.xml``: Extragalactic Survey performed with baseline arrays -* ``obs_egal_threshold.xml``: Extragalactic Survey performed with threshold implementation -* ``obs_all_baseline.xml``: Both surveys performed with baseline arrays -* ``obs_all_threshold.xml``: Both surveys performed with threshold implementation - -The ``models`` folder contains the definitions of all source and background -models that have been used for simulating the data. -The file ``models_gps.xml`` is a so called -:ref:`Model Definition File ` -that collects the definition of all model components used for the Galactic -Plane Survey simulation. -The file ``models_egal.xml`` is the equivalent file for the Extragalactic -Survey. -The other files in the folder are FITS files containing sky maps that were -used as spatial templates for modelling extended or diffuse emission components. -There files are reference in the -:ref:`Model Definition File ` -files. - - - diff --git a/doc/source/users/tutorials/1dc/getting.rst b/doc/source/users/tutorials/1dc/getting.rst deleted file mode 100644 index 18163a9..0000000 --- a/doc/source/users/tutorials/1dc/getting.rst +++ /dev/null @@ -1,15 +0,0 @@ -.. _1dc_getting: - -Getting the data and analysis tools -=================================== - -Getting the data ----------------- - -To be written ... - - -Getting the analysis tools --------------------------- - -To be written ... diff --git a/doc/source/users/tutorials/1dc/getting_data.rst b/doc/source/users/tutorials/1dc/getting_data.rst new file mode 100644 index 0000000..dc3f594 --- /dev/null +++ b/doc/source/users/tutorials/1dc/getting_data.rst @@ -0,0 +1,30 @@ +.. _1dc_getting_data: + +Getting the data +================ + +The data for the +:ref:`First CTA Data Challenge ` +are bundled with the +:ref:`Instrument Response Functions ` +into a single tarball. +Click on +`1dc.tar.gz `_ +to download the tarball (file size: 485 MB). +You can then uncompress and deploy the data at any place by typing + +.. code-block:: bash + + $ tar xfvz 1dc.tar.gz + +.. warning:: + Don't type the ``$`` symbol. It indicates in the following that a command + should be typed on the command line of a console. + +After deploying you should have the folder ``1dc`` in your current working +directory. Within that directory, please set the following environment variables: + +.. code-block:: bash + + $ export CALDB=$PWD/1dc/caldb + $ export CTADATA=$PWD/1dc/data diff --git a/doc/source/users/tutorials/1dc/getting_tools.rst b/doc/source/users/tutorials/1dc/getting_tools.rst new file mode 100644 index 0000000..acb34ae --- /dev/null +++ b/doc/source/users/tutorials/1dc/getting_tools.rst @@ -0,0 +1,9 @@ +.. _1dc_getting_tools: + +Getting the analysis tools +========================== + +We recommend that you use ctools to analyse the data from the +:ref:`First CTA Data Challenge `. +Please read :ref:`getting the ctools ` to learn how to +install and configure the ctools package. diff --git a/doc/source/users/tutorials/1dc/index.rst b/doc/source/users/tutorials/1dc/index.rst index 425dede..a0cea82 100644 --- a/doc/source/users/tutorials/1dc/index.rst +++ b/doc/source/users/tutorials/1dc/index.rst @@ -7,6 +7,8 @@ First CTA Data Challenge :maxdepth: 2 scope - getting + getting_tools + getting_data + data_organisation first -- libgit2 0.21.4