Add tutorial for first CTA Data Challenge.

The tutorial contains so far an incomplete version of the text, but the basic structure is outlined.

Add tutorial for first CTA Data Challenge.
The tutorial contains so far an incomplete version of the text, but the basic structure is outlined.
Jürgen Knödlseder
1 parent 30425a82
Showing 17 changed files with 417 additions and 4 deletions
cscripts/Makefile.am
doc/source/_templates/index.html
doc/source/users/glossary.rst
doc/source/users/index.rst
doc/source/users/tutorials/1dc/first.rst
doc/source/users/tutorials/1dc/first_data.rst
doc/source/users/tutorials/1dc/first_fitting.rst
doc/source/users/tutorials/1dc/first_residuals.rst
doc/source/users/tutorials/1dc/first_select_models.rst
doc/source/users/tutorials/1dc/first_select_obs.rst
doc/source/users/tutorials/1dc/first_skymap.png
doc/source/users/tutorials/1dc/first_skymap.rst
doc/source/users/tutorials/1dc/first_stacked.rst
doc/source/users/tutorials/1dc/getting.rst
doc/source/users/tutorials/1dc/index.rst
doc/source/users/tutorials/1dc/scope.rst
doc/source/users/tutorials/index.rst
@@ -112,6 +112,7 @@ testscripts = $(srcdir)/tests/__init__.py \
               $(top_srcdir)/test/test_csobs2caldb.py \
               $(top_srcdir)/test/test_csobsdef.py \
               $(top_srcdir)/test/test_csobsinfo.py \
+              $(top_srcdir)/test/test_csobsselect.py \
               $(top_srcdir)/test/test_cslightcrv.py \
               $(top_srcdir)/test/test_cspull.py \
               $(top_srcdir)/test/test_csresmap.py \
@@ -48,10 +48,12 @@ and next steps, and advance with the coding of the software. {%endtrans%}&lt;/p&gt;
 <h2 style="margin-bottom: 0">{%trans%}Documentation{%endtrans%}</h2>
  
 <ul>
+<li><a class="smalllink" href="{{ pathto("users/tutorials/index") }}">{%trans%}Tutorials{%endtrans%}</a></li>
 <li><a class="smalllink" href="{{ pathto("users/user_manual/index") }}">{%trans%}User Manual{%endtrans%}</a></li>
 <li><a class="smalllink" href="{{ pathto("users/reference_manual/index") }}">{%trans%}Reference Manual{%endtrans%}</a></li>
 <li><a class="smalllink" href="{{ pathto("users/faq") }}">{%trans%}Frequently asked questions{%endtrans%}</a></li>
 <li><a class="smalllink" href="{{ pathto("users/issues") }}">{%trans%}Known issues{%endtrans%}</a></li>
+<li><a class="smalllink" href="{{ pathto("users/glossary") }}">{%trans%}Glossary{%endtrans%}</a></li>
 <li><a class="smalllink" href="{{ pathto("users/release_history/index") }}">{%trans%}Release history{%endtrans%}</a></li>
 <li><a class="smalllink" href="doxygen/index.html">{%trans%}Source code documentation (Doxygen){%endtrans%}</a></li>
 </ul>
@@ -59,8 +61,8 @@ and next steps, and advance with the coding of the software. {%endtrans%}&lt;/p&gt;
 <h2 style="margin-bottom: 0">{%trans%}Links{%endtrans%}</h2>
  
 <ul>
-	<li><a href="https://cta-gitlab.irap.omp.eu/ctools/ctools">GitLab</a></li>
-	<li><a href="https://github.com/ctools/ctools">GitHub</a></li>
-	<li><a href="https://cta-redmine.irap.omp.eu/projects/ctools">Redmine</a></li>
-	<li><a href="https://www.openhub.net/p/ctools_">Open HUB</a>{% endblock %}</li>
+<li><a href="https://cta-gitlab.irap.omp.eu/ctools/ctools">GitLab</a></li>
+<li><a href="https://github.com/ctools/ctools">GitHub</a></li>
+<li><a href="https://cta-redmine.irap.omp.eu/projects/ctools">Redmine</a></li>
+<li><a href="https://www.openhub.net/p/ctools_">Open HUB</a>{% endblock %}</li>
 </ul>
+Glossary
+========
+
+.. _glossary_eventlist:
+
+.. topic:: Event list
+
+   Bla bla ...
+
+.. _glossary_1dc:
+
+.. topic:: First CTA Data Challenge
+
+   Bla bla ...
+
+.. _glossary_irf:
+
+.. topic:: Instrument Response Functions
+
+   Bla bla ...
+
+.. _glossary_moddef:
+
+.. topic:: Model definition file
+
+   Bla bla ...
+
+.. _glossary_obs:
+
+.. topic:: Observation
+
+   Bla bla ...
+
+.. _glossary_obsdef:
+
+.. topic:: Observation definition file
+
+   Bla bla ...
@@ -6,9 +6,11 @@ User Documentation
 .. toctree::
    :maxdepth: 2
  
+   tutorials/index
    user_manual/index
    reference_manual/index
    faq
    issues
+   glossary
    release_history/index
    help
+.. _1dc_first:
+
+First analysis steps
+====================
+
+The following sections describe how to analyse data from the
+:ref:`First CTA Data Challenge <glossary_1dc>`
+using ctools.
+If you have never used ctools before or if you have never analysed data from the
+:ref:`First CTA Data Challenge <glossary_1dc>`,
+this is the place where you should start.
+
+We assume that you have downloaded the data of the
+:ref:`First CTA Data Challenge <glossary_1dc>`
+and that you have successfully installed the ctools package.
+If this is not the case, please read :ref:`1dc_getting`.
+
+.. toctree::
+   :maxdepth: 2
+
+   first_data
+   first_select_obs
+   first_skymap
+   first_select_models
+   first_fitting
+   first_residuals
+   first_stacked
+.. _1dc_first_data:
+
+Data organisation
+-----------------
+
+After downloading the
+:ref:`First CTA Data Challenge <glossary_1dc>`
+data from **TBD** and uncompressing the tarball using
+
+.. code-block:: bash
+
+   $ tar xfvz 1dc.tar.gz
+
+you will have a directory ``1dc`` on your disk with the following content:
+
+.. code-block:: bash
+
+   caldb/
+   caldb/data
+   caldb/data/cta
+   caldb/data/cta/prod2
+   ...
+   caldb/data/cta/prod3
+   caldb/data/cta/prod3/caldb.indx
+   caldb/data/cta/prod3/bcf
+   ...
+   data/
+   data/gps_baseline_000001.fits
+   data/gps_baseline_000002.fits
+   data/gps_baseline_000003.fits
+   ...
+   data/gps_threshold_000001.fits
+   data/gps_threshold_000002.fits
+   data/gps_threshold_000003.fits
+   ...
+   data/egal_baseline_000001.fits
+   data/egal_baseline_000002.fits
+   data/egal_baseline_000003.fits
+   ...
+   data/egal_threshold_000001.fits
+   data/egal_threshold_000002.fits
+   data/egal_threshold_000003.fits
+   ...
+   data/obs_gps_baseline.xml
+   data/obs_gps_threshold.xml
+   data/obs_egal_baseline.xml
+   data/obs_egal_threshold.xml
+   data/obs_all_baseline.xml
+   data/obs_all_threshold.xml
+   models/
+   models/models_gps.xml
+   models/models_egal.xml
+   models/map_RXJ1713.fits.gz
+   models/map_VelaJunior.fits.gz
+   models/map_ics.fits.gz
+   models/map_pi0.fits.gz
+
+The ``caldb`` folder contains the
+:ref:`Instrument Response Functions <glossary_irf>`
+that are necessary for the analysis of the simulated CTA data.
+The folder contains the ``prod3`` response that should be used for the
+:ref:`First CTA Data Challenge <glossary_1dc>`.
+For legacy reasons, also the ``prod2`` response functions are shipped with
+the tarball.
+
+The ``data`` folder contains the calibrated, reconstructed and background
+reduced event data that were procuded for the
+:ref:`First CTA Data Challenge <glossary_1dc>`.
+Each event file corresponds to the
+:ref:`event lists <glossary_eventlist>`
+for an
+:ref:`observation (or run) <glossary_obs>`, and contains for each event a
+time stamp, the reconstructued photon arrival direction, and the reconstructed
+photon energy.
+The pointing direction during an observation is fixed.
+Event files exist for the Galactic Plane Survey (starting with ``gps_``) and
+for the extragalactic survey (starting with ``egal_``) for both
+the ``baseline`` configuration and the ``threshold`` implementation.
+
+The files ``obs_XXX_YYYYY.xml`` are so called
+:ref:`Observation Definition Files <glossary_obsdef>`
+that contain the information (or metadata) of a list of observations.
+The files are plain ASCII files in XML format that be inspected and
+manipulated by any text editor.
+The following files are included in the data distribution:
+
+* ``obs_gps_baseline.xml``: Galactic Plane Survey performed with baseline arrays
+* ``obs_gps_threshold.xml``: Galactic Plane Survey performed with threshold implementation
+* ``obs_egal_baseline.xml``: Extragalactic Survey performed with baseline arrays
+* ``obs_egal_threshold.xml``: Extragalactic Survey performed with threshold implementation
+* ``obs_all_baseline.xml``: Both surveys performed with baseline arrays
+* ``obs_all_threshold.xml``: Both surveys performed with threshold implementation
+
+The ``models`` folder contains the definitions of all source and background
+models that have been used for simulating the data.
+The file ``models_gps.xml`` is a so called
+:ref:`Model Definition File <glossary_moddef>`
+that collects the definition of all model components used for the Galactic
+Plane Survey simulation.
+The file ``models_egal.xml`` is the equivalent file for the Extragalactic
+Survey.
+The other files in the folder are FITS files containing sky maps that were
+used as spatial templates for modelling extended or diffuse emission components.
+There files are reference in the
+:ref:`Model Definition File <glossary_moddef>`
+files.
+
+
+
+.. _1dc_first_fitting:
+
+Fitting the model components to the data
+----------------------------------------
+
+.. code-block:: bash
+
+   $ ctlike
+   Input event list, counts cube or observation definition XML file [events.fits] obs.xml
+   Input model XML file [$CTOOLS/share/models/crab.xml] models.xml
+   Output model XML file [crab_results.xml] results.xml
+.. _1dc_first_residuals:
+
+Inspecting the residuals
+------------------------
+
+.. code-block:: bash
+
+   $ csresmap
+   Input event list, counts cube, or observation definition XML file [obs.xml]
+   Input model XML file [results.xml]
+   First coordinate of image center in degrees (RA or galactic l) (0-360) [0.0]
+   Second coordinate of image center in degrees (DEC or galactic b) (-90-90) [0.0]
+   Lower energy limit (TeV) [0.1]
+   Upper energy limit (TeV) [100.0]
+   Coordinate System (CEL|GAL) [GAL]
+   Projection method (AIT|AZP|CAR|MER|MOL|STG|TAN) [CAR]
+   Size of the X axis in pixels [200]
+   Size of the Y axis in pixels [200] 100
+   Pixel size (deg/pixel) [0.1] 0.05
+   Residual map computation algorithm (SUB|SUBDIV|SUBDIVSQRT) [SUB]
+   Output residual map file [resmap.fits]
+.. _1dc_select_models:
+
+Selecting the relevant model components
+---------------------------------------
+
+.. code-block:: bash
+
+   $ csmodelselect fluxlimit=1.0e-10
+   Input event list or observation definition XML file [obs.xml]
+   Input model definition XML file [inmodel.xml] ../models/models_gps.xml
+   Output model definition XML file [outmodel.xml] models.xml
+.. _1dc_first_select_obs:
+
+Selecting the relevant observations
+-----------------------------------
+
+Let us in the following assume that you want to analyse the Galactic Plane
+Survey data obtained with baseline arrays around the Galactic Centre.
+For the purpose of this first analysis, you should create a subfolder under
+the ``1dc`` folder (or any other convenient place) that will contain the
+results for your first analysis.
+You do this by typing:
+
+.. code-block:: bash
+
+   $ cd 1dc
+   $ mkdir my_first_analysis
+   $ cd my_first_analysis
+
+The first step of the analysis consists in selecting the observations from the
+:ref:`Observation Definition File <glossary_obsdef>`
+``obs_gps_baseline.xml`` that have pointing directions close to the Galactic
+Centre.
+You do this with the :ref:`csobsselect` script by typing:
+
+.. code-block:: bash
+
+   $ csobsselect
+   Input event list or observation definition XML file [obs.xml] ../data/obs_gps_baseline.xml
+   Pointing selection region shape (CIRCLE|BOX) [CIRCLE]
+   Coordinate system (CEL - celestial, GAL - galactic) (CEL|GAL) [CEL] GAL
+   Galactic longitude of selection centre (deg) (0-360) [184.56] 0.0
+   Galactic latitude of selection centre (deg) (-90-90) [-5.79] 0.0
+   Radius of selection circle (deg) (0-180) [5.0]
+   Output observation definition XML file [outobs.xml] obs.xml
+
+You selected ``CIRCLE`` as the shape of the pointing selection region,
+specified the Galactic Centre as the centre of the selection region, and
+selected all observations with pointing directions within 5 degrees from
+the Galactic Centre from the ``obs_gps_baseline.xml`` file.
+The selected observations are written into the file ``obs.xml`` that will be
+created in the ``my_first_analysis`` folder.
+The content of the file will look as follows:
+
+.. code-block:: xml
+
+   <?xml version="1.0" encoding="UTF-8" standalone="no"?>
+   <observation_list title="observation list">
+     <observation name="Galactic Plane Survey KSP" id="000020" instrument="CTA">
+       <parameter name="EventList" file="../data/gps_baseline_000020.fits" />
+       <parameter name="Calibration" database="prod2" response="South_50h" />
+     </observation>
+     <observation name="Galactic Plane Survey KSP" id="000021" instrument="CTA">
+       <parameter name="EventList" file="../data/gps_baseline_000021.fits" />
+       <parameter name="Calibration" database="prod2" response="South_50h" />
+     </observation>
+     <observation name="Galactic Plane Survey KSP" id="000022" instrument="CTA">
+       <parameter name="EventList" file="../data/gps_baseline_000022.fits" />
+       <parameter name="Calibration" database="prod2" response="South_50h" />
+     </observation>
+     ...
+   </observation_list>
+
+Each ``<observation>`` element corresponds to one observation that is identified
+by a name attribute and a unique identifier attribute.
+An ``<observation>`` element contains two parameters:
+the ``EventList`` parameter that specifies the name of the corresponding event
+file relative to the current working directory, and
+the ``Calibration`` parameter that specifies the
+:ref:`Instrument Response Function <glossary_irf>` that applies to the
+event file.
+
+.. _1dc_sky_map:
+
+Generating a sky map from the events
+------------------------------------
+
+Once the observations are selected, the first thing that you want to do is to
+visualize the content of the observations by generating a sky map from the
+events.
+You do this with the :ref:`ctskymap` tool by typing:
+
+.. code-block:: bash
+
+   $ ctskymap
+   Input event list or observation definition XML file [events.fits] obs.xml
+   First coordinate of image center in degrees (RA or galactic l) (0-360) [83.63] 0.0
+   Second coordinate of image center in degrees (DEC or galactic b) (-90-90) [22.01] 0.0
+   Projection method (AIT|AZP|CAR|MER|MOL|STG|TAN) [CAR]
+   Coordinate system (CEL - celestial, GAL - galactic) (CEL|GAL) [CEL] GAL
+   Image scale (in degrees/pixel) [0.02]
+   Size of the X axis in pixels [200] 500
+   Size of the Y axis in pixels [200] 250
+   Lower energy limit (TeV) [0.1]
+   Upper energy limit (TeV) [100.0]
+   Output skymap file [skymap.fits]
+
+We generate here a sky map centred on the Galactic Centre in Galactic
+coordinates using a cartesian projection.
+The sky map is 10° wide and 5° high, with an image scale of 0.02° per pixel.
+All events between 0.1 and 100 TeV have been collected in the skymap.
+The :ref:`ctskymap` tool writes the result into the FITS file ``skymap.fits``
+that will be created in the ``my_first_analysis`` folder.
+The FITS file can be displayed using for example ds9 (see below):
+
+.. figure:: first_skymap.png
+   :width: 600px
+   :align: center
+
+   *Sky map of the events recorded during the Galactic Plane Survey around the Galactic Centre*
+
+Bla
+.. _1dc_first_stacked:
+
+Performing a stacked analysis
+-----------------------------
+
+To be written ...
+.. _1dc_getting:
+
+Getting the data and analysis tools
+===================================
+
+Getting the data
+----------------
+
+To be written ...
+
+
+Getting the analysis tools
+--------------------------
+
+To be written ...
+.. _1dc:
+
+First CTA Data Challenge
+========================
+
+.. toctree::
+   :maxdepth: 2
+
+   scope
+   getting
+   first
+
+.. _1dc_scope:
+
+Scope of the First CTA Data Challenge
+=====================================
+
+The primary goal of the
+:ref:`First CTA Data Challenge <glossary_1dc>`
+is to make the members of the CTA Consortium Science Working Groups familiar
+with the CTA data format and tools and to stimulate the simulation and
+benchmarking activities in the groups.
+A secondary goal is to test CTA Science Analysis tools and to provide User
+feedback to the code developers.
+
+The data distributed in the
+:ref:`First CTA Data Challenge <glossary_1dc>`
+are high-level data in form of
+:ref:`event lists <glossary_eventlist>`
+and
+:ref:`Instrument Response Functions <glossary_irf>`
+(data level 3).
+The
+:ref:`event lists <glossary_eventlist>`
+have been simulated from high-level
+:ref:`Instrument Response Functions <glossary_irf>`
+that correspond to an ideal CTA with good and stable atmosphere and instrument
+conditions; only zenith angles of 20° and 40° **(TBC)** have been simulated.
+
+Data are provided for the Baseline arrays and the Threshold Implementation.
+
+Two Key Science Projects (KSPs) have been simulated using realistic pointing
+patterns:
+
+* the Galactic Plane Scan (GPS)
+* the extragalactic survey
+
+The source model used for the simulations is distributed together with the
+data to the CTA Consortium Science Working Groups, so that analysis results
+can be compared to the Monte Carlo truth.
+.. _tutorials:
+
+Tutorials
+=========
+
+.. toctree::
+   :maxdepth: 2
+
+   1dc/index