Commit c5d689f979adfc01e7d57165a01a82cfa0f40b8f

Authored by Jürgen Knödlseder
1 parent 9159e38d

Update IACT How To's

cscripts/csiactobs.par
... ... @@ -22,7 +22,7 @@ infile, f, a, runlist.lis,,, "Input runlist file"
22 22 inmodel, f, h, NONE,,, "Input model definition XML file (optional)"
23 23 master_indx, s, h, master.json,,, "Input master index file"
24 24 outobs, f, a, obs.xml,,, "Output observation definition XML file"
25   -outmodel, f, a, bgmodels.xml,,, "Output model definition XML file"
  25 +outmodel, f, a, bkgmodels.xml,,, "Output model definition XML file"
26 26  
27 27 #
28 28 # Script parameters
... ... @@ -31,7 +31,7 @@ bkgpars, i, a, 1,,, "Number of free parameters per background model"
31 31 bkg_scale, b, h, yes,,, "Apply background scaling factor from observation index file if available?"
32 32 ev_hiera, s, h, events,,, "Hierarchy of event formats"
33 33 aeff_hiera, s, h, aeff_2d,,, "Hierarchy of effective area formats"
34   -psf_hiera, s, h, psf_king|psf_3gauss,,, "Hierarchy of psf formats"
  34 +psf_hiera, s, h, psf_king|psf_3gauss|psf_table,,, "Hierarchy of psf formats"
35 35 edisp_hiera, s, h, edisp_2d,,, "Hierarchy of energy dispersion formats"
36 36 bkg_hiera, s, h, bkg_3d,,, "Hierarchy of background formats"
37 37 bkg_mod_hiera, s, h, irf|aeff|gauss,,, "Hierarchy of background models"
... ...
doc/source/users/tutorials/hess_dr1/index.rst
1   -.. _1dc:
  1 +.. _hess_dr1:
2 2  
3 3 Analysing public H.E.S.S. data
4 4 ==============================
... ...
doc/source/users/tutorials/hess_dr1/jointfermi.rst
1   -.. _hess_dr1_data:
  1 +.. _hess_dr1_fermi:
2 2  
3 3 Joint analysis of H.E.S.S. and Fermi data
4 4 =========================================
... ...
doc/source/users/tutorials/howto/iact/analysis.rst
1 1 .. _sec_iact_analysis:
2 2  
3   -Analysing IACT data
4   -===================
5   -
6   -In order to analyse data from current IACTs, you first have to retrieve the
7   -data. Information on how to do that can be found in the section :ref:`sec_iact_copy`.
8   -
9   -Check available FITS production
10   --------------------------------
11   -Before you start an analysis, it is important that you know what kind of data
12   -need to be analysed. In the current scheme of
13   -`storing IACT data <http://gamma-astro-data-formats.readthedocs.org/en/latest/data_storage/index.html>`_,
14   -every FITS data set is represented by a unique name, e.g.
15   -``"hess-fits-pa-release-1.0-Prod26-MppStd"``.
16   -You have to pass this string to some set of scripts to find and use the data
17   -for analysis.
18   -
19   -To check the available names of FITS data sets, you should run the
20   -:ref:`csiactdata` script:
21   -
22   -.. code-block:: bash
23   -
24   - $ csiactdata
25   - Path were data is located [] /path/to/fits/data/
26   -
27   -This script will list the names of available FITS data sets into the console.
28   -In the following it is assumed that you have selected a data set named
29   -``fits-data-name``.
30   -You can omit the query for the path where the data is located by setting the
31   -environment variable ``VHEFITS`` to this location. Also other scripts will
32   -benefit from having this environment variable set, and this avoids that you
33   -have to type in the path name every time you use one of these scripts.
34   -
35   -
36   -Find observations
37   ------------------
38   -
39   -Now that you know which data are available you have to assemble a list of
40   -observations (a.k.a. runs) that you want to use in an analysis. You will
41   -use the :ref:`csfindobs` for that task, which allows you to select the
42   -observations according to a number of user criteria.
43   -
44   -
45   -Select observations according to pointing direction
46   -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
47   -
48   -The most common search for observations is by pointing direction on the sky,
49   -and you do this by simply invoking the :ref:`csfindobs` script:
50   -
51   -.. code-block:: bash
52   -
53   - $ csfindobs
54   - Name of FITS production (Run csiactdata to view your options) [fits-data-name]
55   - Right ascension of selection region centre (deg) [83.63]
56   - Declination of selection region centre (deg) [22.01]
57   - Search radius of selection region (deg) [2.5]
58   - Output runlist file [runlist.lis]
59   -
60   -The script will create an ASCII file named ``runlist.lis`` that contains the
61   -identifiers for all observations with pointing directions located within 2.5°
62   -around the position of the Crab nebula.
63   -
64   -
65   -Select observations using additional criteria
66   -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
67   -
68   -You may add further selection criteria using the hidden ``expression`` parameter.
69   -You may use any FITS selection expression that is
70   -`supported by the cfitsio row filtering specification <https://heasarc.gsfc.nasa.gov/docs/software/fitsio/c/c_user/node97.html>`_.
71   -Column names that can be used in the expression can be found
72   -`here <http://gamma-astro-data-formats.readthedocs.org/en/latest/data_storage/obs_index/index.html>`_
73   -or by simply browsing the observation index file with a FITS viewer. For example
74   -
75   -.. code-block:: bash
76   -
77   - $ csfindobs expression="ZEN_PNT<30&&LIVETIME>1800"
78   -
79   -will select all observations that have a zenith angle less than 30° and a
80   -livetime larger than 1800 seconds.
81   -
82   -
83   -Select observations independent of pointing direction
84   -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
85   -
86   -You can also omit the pointing selection, for example in case that you do not
87   -know the sky coordinates of your object of interest. In that case, you may
88   -directly use the object name in the expression and pass ``NONE`` as coordinate
89   -to :ref:`csfindobs`:
90   -
91   -.. code-block:: bash
92   -
93   - $ csfindobs expression="OBJECT=='Crab Nebula'"
94   - Name of FITS production (Run csiactdata to view your options) [fits-data-name]
95   - Right ascension of selection region centre [NONE]
96   - Output runlist file [runlist.lis]
  3 +Performing a maximum likelihood fit
  4 +===================================
  5 +
  6 + .. admonition:: What you will learn
  7 +
  8 + You will learn how to perform a maximum likelihood fit of IACT data.
  9 +
  10 +Following a :ref:`csiactobs` run with the ``inmodel=$CTOOLS/share/models/crab.xml``
  11 +you have a model that is ready for maximum likelihood fitting.
  12 +You do such a fit using
  13 +
  14 +.. code-block:: bash
  15 +
  16 + $ ctlike
  17 + Input event list, counts cube or observation definition XML file [obs_selected.xml]
  18 + Input model definition XML file [models.xml]
  19 + Output model definition XML file [crab_results.xml]
  20 +
  21 +which will produce the following output in the log file ``ctlike.log``:
  22 +
  23 +.. code-block:: none
  24 +
  25 + 2019-04-05T14:18:17: +=================================+
  26 + 2019-04-05T14:18:17: | Maximum likelihood optimisation |
  27 + 2019-04-05T14:18:17: +=================================+
  28 + 2019-04-05T14:18:17: Parameter "Prefactor" has zero curvature. Fix parameter.
  29 + 2019-04-05T14:18:17: Parameter "Index" has zero curvature. Fix parameter.
  30 + 2019-04-05T14:18:17: >Iteration 0: -logL=203790.772, Lambda=1.0e-03
  31 + 2019-04-05T14:18:17: >Iteration 1: -logL=192855.522, Lambda=1.0e-03, delta=10935.250, step=1.0e+00, max(|grad|)=-2040.622705 [Prefactor:0]
  32 + 2019-04-05T14:18:17: >Iteration 2: -logL=182999.110, Lambda=1.0e-04, delta=9856.412, step=1.0e+00, max(|grad|)=-981.683624 [Prefactor:0]
  33 + 2019-04-05T14:18:17: >Iteration 3: -logL=174691.964, Lambda=1.0e-05, delta=8307.146, step=1.0e+00, max(|grad|)=620.662474 [Index:9]
  34 + 2019-04-05T14:18:17: >Iteration 4: -logL=168496.493, Lambda=1.0e-06, delta=6195.471, step=1.0e+00, max(|grad|)=587.098865 [Index:9]
  35 + 2019-04-05T14:18:17: >Iteration 5: -logL=164898.487, Lambda=1.0e-07, delta=3598.006, step=1.0e+00, max(|grad|)=423.800306 [Index:5]
  36 + 2019-04-05T14:18:18: >Iteration 6: -logL=163621.925, Lambda=1.0e-08, delta=1276.562, step=1.0e+00, max(|grad|)=237.546032 [Index:1]
  37 + 2019-04-05T14:18:18: >Iteration 7: -logL=163423.407, Lambda=1.0e-09, delta=198.518, step=1.0e+00, max(|grad|)=102.646722 [Index:1]
  38 + 2019-04-05T14:18:18: >Iteration 8: -logL=163409.717, Lambda=1.0e-10, delta=13.690, step=1.0e+00, max(|grad|)=35.371246 [Index:1]
  39 + 2019-04-05T14:18:18: >Iteration 9: -logL=163408.542, Lambda=1.0e-11, delta=1.175, step=1.0e+00, max(|grad|)=11.708402 [Index:1]
  40 + 2019-04-05T14:18:18: >Iteration 10: -logL=163408.416, Lambda=1.0e-12, delta=0.126, step=1.0e+00, max(|grad|)=3.907383 [Index:1]
  41 + 2019-04-05T14:18:18: >Iteration 11: -logL=163408.402, Lambda=1.0e-13, delta=0.014, step=1.0e+00, max(|grad|)=1.309476 [Index:1]
  42 + 2019-04-05T14:18:18: >Iteration 12: -logL=163408.400, Lambda=1.0e-14, delta=0.002, step=1.0e+00, max(|grad|)=0.439462 [Index:1]
  43 + 2019-04-05T14:18:18: Free parameter "Prefactor" after convergence was reached with frozen parameter.
  44 + 2019-04-05T14:18:18: Free parameter "Index" after convergence was reached with frozen parameter.
  45 + 2019-04-05T14:18:18:
  46 + 2019-04-05T14:18:18: +=========================================+
  47 + 2019-04-05T14:18:18: | Maximum likelihood optimisation results |
  48 + 2019-04-05T14:18:18: +=========================================+
  49 + 2019-04-05T14:18:18: === GOptimizerLM ===
  50 + 2019-04-05T14:18:18: Optimized function value ..: 163408.400
  51 + 2019-04-05T14:18:18: Absolute precision ........: 0.005
  52 + 2019-04-05T14:18:18: Acceptable value decrease .: 2
  53 + 2019-04-05T14:18:18: Optimization status .......: converged
  54 + 2019-04-05T14:18:18: Number of parameters ......: 26
  55 + 2019-04-05T14:18:18: Number of free parameters .: 12
  56 + 2019-04-05T14:18:18: Number of iterations ......: 12
  57 + 2019-04-05T14:18:18: Lambda ....................: 1e-15
  58 + 2019-04-05T14:18:18: Maximum log likelihood ....: -163408.400
  59 + 2019-04-05T14:18:18: Observed events (Nobs) ...: 18641.000
  60 + 2019-04-05T14:18:18: Predicted events (Npred) ..: 18640.998 (Nobs - Npred = 0.00164345248049358)
  61 + 2019-04-05T14:18:18: === GModels ===
  62 + 2019-04-05T14:18:18: Number of models ..........: 6
  63 + 2019-04-05T14:18:18: Number of parameters ......: 26
  64 + 2019-04-05T14:18:18: === GCTAModelAeffBackground ===
  65 + 2019-04-05T14:18:18: Name ......................: bkg_23523
  66 + 2019-04-05T14:18:18: Instruments ...............: HESS
  67 + 2019-04-05T14:18:18: Instrument scale factors ..: unity
  68 + 2019-04-05T14:18:18: Observation identifiers ...: 23523
  69 + 2019-04-05T14:18:18: Model type ................: "PowerLaw" * "Constant"
  70 + 2019-04-05T14:18:18: Number of parameters ......: 4
  71 + 2019-04-05T14:18:18: Number of spectral par's ..: 3
  72 + 2019-04-05T14:18:18: Prefactor ................: 2.98790819794319e-13 +/- 4.43400659819813e-15 [1e-16,1e-12] ph/cm2/s/MeV (free,scale=1e-14,gradient)
  73 + 2019-04-05T14:18:18: Index ....................: -3.22702359177827 +/- 0.0221521174108544 [-5,5] (free,scale=1,gradient)
  74 + 2019-04-05T14:18:18: PivotEnergy ..............: 1000000 MeV (fixed,scale=1000000,gradient)
  75 + 2019-04-05T14:18:18: Number of temporal par's ..: 1
  76 + 2019-04-05T14:18:18: Normalization ............: 1 (relative value) (fixed,scale=1,gradient)
  77 + 2019-04-05T14:18:18: === GCTAModelAeffBackground ===
  78 + 2019-04-05T14:18:18: Name ......................: bkg_23526
  79 + 2019-04-05T14:18:18: Instruments ...............: HESS
  80 + 2019-04-05T14:18:18: Instrument scale factors ..: unity
  81 + 2019-04-05T14:18:18: Observation identifiers ...: 23526
  82 + 2019-04-05T14:18:18: Model type ................: "PowerLaw" * "Constant"
  83 + 2019-04-05T14:18:18: Number of parameters ......: 4
  84 + 2019-04-05T14:18:18: Number of spectral par's ..: 3
  85 + 2019-04-05T14:18:18: Prefactor ................: 2.44289795381698e-13 +/- 3.75318255643637e-15 [1e-16,1e-12] ph/cm2/s/MeV (free,scale=1e-14,gradient)
  86 + 2019-04-05T14:18:18: Index ....................: -3.21741632072522 +/- 0.0232805723886611 [-5,5] (free,scale=1,gradient)
  87 + 2019-04-05T14:18:18: PivotEnergy ..............: 1000000 MeV (fixed,scale=1000000,gradient)
  88 + 2019-04-05T14:18:18: Number of temporal par's ..: 1
  89 + 2019-04-05T14:18:18: Normalization ............: 1 (relative value) (fixed,scale=1,gradient)
  90 + 2019-04-05T14:18:18: === GCTAModelAeffBackground ===
  91 + 2019-04-05T14:18:18: Name ......................: bkg_23559
  92 + 2019-04-05T14:18:18: Instruments ...............: HESS
  93 + 2019-04-05T14:18:18: Instrument scale factors ..: unity
  94 + 2019-04-05T14:18:18: Observation identifiers ...: 23559
  95 + 2019-04-05T14:18:18: Model type ................: "PowerLaw" * "Constant"
  96 + 2019-04-05T14:18:18: Number of parameters ......: 4
  97 + 2019-04-05T14:18:18: Number of spectral par's ..: 3
  98 + 2019-04-05T14:18:18: Prefactor ................: 2.26357632227583e-13 +/- 3.56983854227473e-15 [1e-16,1e-12] ph/cm2/s/MeV (free,scale=1e-14,gradient)
  99 + 2019-04-05T14:18:18: Index ....................: -3.25328282552122 +/- 0.0239266148104449 [-5,5] (free,scale=1,gradient)
  100 + 2019-04-05T14:18:18: PivotEnergy ..............: 1000000 MeV (fixed,scale=1000000,gradient)
  101 + 2019-04-05T14:18:18: Number of temporal par's ..: 1
  102 + 2019-04-05T14:18:18: Normalization ............: 1 (relative value) (fixed,scale=1,gradient)
  103 + 2019-04-05T14:18:18: === GCTAModelAeffBackground ===
  104 + 2019-04-05T14:18:18: Name ......................: bkg_23592
  105 + 2019-04-05T14:18:18: Instruments ...............: HESS
  106 + 2019-04-05T14:18:18: Instrument scale factors ..: unity
  107 + 2019-04-05T14:18:18: Observation identifiers ...: 23592
  108 + 2019-04-05T14:18:18: Model type ................: "PowerLaw" * "Constant"
  109 + 2019-04-05T14:18:18: Number of parameters ......: 4
  110 + 2019-04-05T14:18:18: Number of spectral par's ..: 3
  111 + 2019-04-05T14:18:18: Prefactor ................: 2.87721724170504e-13 +/- 4.31204359584396e-15 [1e-16,1e-12] ph/cm2/s/MeV (free,scale=1e-14,gradient)
  112 + 2019-04-05T14:18:18: Index ....................: -3.25393054001656 +/- 0.0227802144778143 [-5,5] (free,scale=1,gradient)
  113 + 2019-04-05T14:18:18: PivotEnergy ..............: 1000000 MeV (fixed,scale=1000000,gradient)
  114 + 2019-04-05T14:18:18: Number of temporal par's ..: 1
  115 + 2019-04-05T14:18:18: Normalization ............: 1 (relative value) (fixed,scale=1,gradient)
  116 + 2019-04-05T14:18:18: === GModelSky ===
  117 + 2019-04-05T14:18:18: Name ......................: Crab
  118 + 2019-04-05T14:18:18: Instruments ...............: all
  119 + 2019-04-05T14:18:18: Instrument scale factors ..: unity
  120 + 2019-04-05T14:18:18: Observation identifiers ...: all
  121 + 2019-04-05T14:18:18: Model type ................: PointSource
  122 + 2019-04-05T14:18:18: Model components ..........: "PointSource" * "PowerLaw" * "Constant"
  123 + 2019-04-05T14:18:18: Number of parameters ......: 6
  124 + 2019-04-05T14:18:18: Number of spatial par's ...: 2
  125 + 2019-04-05T14:18:18: RA .......................: 83.6331 [-360,360] deg (fixed,scale=1)
  126 + 2019-04-05T14:18:18: DEC ......................: 22.0145 [-90,90] deg (fixed,scale=1)
  127 + 2019-04-05T14:18:18: Number of spectral par's ..: 3
  128 + 2019-04-05T14:18:18: Prefactor ................: 1.205595290581e-15 +/- 1.02399444013702e-16 [1e-23,1e-13] ph/cm2/s/MeV (free,scale=1e-16,gradient)
  129 + 2019-04-05T14:18:18: Index ....................: -2.64310065863216 +/- 0.0532926238637032 [-0,-5] (free,scale=-1,gradient)
  130 + 2019-04-05T14:18:18: PivotEnergy ..............: 300000 [10000,1000000000] MeV (fixed,scale=1000000,gradient)
  131 + 2019-04-05T14:18:18: Number of temporal par's ..: 1
  132 + 2019-04-05T14:18:18: Normalization ............: 1 (relative value) (fixed,scale=1,gradient)
  133 + 2019-04-05T14:18:18: === GCTAModelIrfBackground ===
  134 + 2019-04-05T14:18:18: Name ......................: CTABackgroundModel
  135 + 2019-04-05T14:18:18: Instruments ...............: CTA
  136 + 2019-04-05T14:18:18: Instrument scale factors ..: unity
  137 + 2019-04-05T14:18:18: Observation identifiers ...: all
  138 + 2019-04-05T14:18:18: Model type ................: "PowerLaw" * "Constant"
  139 + 2019-04-05T14:18:18: Number of parameters ......: 4
  140 + 2019-04-05T14:18:18: Number of spectral par's ..: 3
  141 + 2019-04-05T14:18:18: Prefactor ................: 1 +/- 0 [0.001,1000] ph/cm2/s/MeV (free,scale=1,gradient)
  142 + 2019-04-05T14:18:18: Index ....................: 0 +/- 0 [-5,5] (free,scale=1,gradient)
  143 + 2019-04-05T14:18:18: PivotEnergy ..............: 1000000 [10000,1000000000] MeV (fixed,scale=1000000,gradient)
  144 + 2019-04-05T14:18:18: Number of temporal par's ..: 1
  145 + 2019-04-05T14:18:18: Normalization ............: 1 (relative value) (fixed,scale=1,gradient)
97 146  
98 147 .. note::
  148 + Using ``inmodel=$CTOOLS/share/models/crab.xml`` will also append the CTA
  149 + background model to the ``models.xml`` file, yet since this model only
  150 + applies to CTA data it is ignored in the fit. This is signalled by
99 151  
100   - By default, :ref:`csfindobs` only selects data of highest quality (i.e. QUALITY=0).
101   - You may overwrite this default by specifying the hidden parameter
102   - ``min_qual``. For example, ``min_qual=1`` selects all data with a
103   - looser quality criteria.
  152 + .. code-block:: none
104 153  
  154 + 2019-04-05T14:18:17: Parameter "Prefactor" has zero curvature. Fix parameter.
  155 + 2019-04-05T14:18:17: Parameter "Index" has zero curvature. Fix parameter.
105 156  
106   -Create an observation list
107   ---------------------------
108   -
109   -As next step, you must convert the runlist ASCII file into an observation
110   -definition XML file. The observation definition XML file will contain the
111   -file names of all files that are needed for the analysis. You do this
112   -conversion with the :ref:`csiactobs` script:
113   -
114   -.. code-block:: bash
115   -
116   - $ csiactobs
117   - Data storage name [] fits-data-name
118   - Input runlist file [runlist.lis]
119   - Number of free parameters per background model [1]
120   - Output model definition XML file [bgmodels.xml]
121   - Output observation definition XML file [obs.xml]
122   -
123   -The :ref:`csiactobs` script will create two output files: the observations
124   -definition XML file ``obs.xml`` and an output model definition XML file
125   -``bgmodels.xml``. To generate ``obs.xml``, :ref:`csiactobs` has used the
126   -IACT data storage and extracted the relevant file names. ``bgmodels.xml`` is
127   -a file that is used for background modeling, where each observation will have
128   -its own independent background model. In the example above, you have set the
129   -number of free parameters per background model to one, hence the normalisation
130   -of the background model for each observation will be a free parameter that is
131   -later adjusted by the maximum likelihood fit.
132   -
133   -There are some further hidden parameters to steer the start parameters for the
134   -effective area and the Gaussian background models. You may have a look at
135   -:ref:`csiactobs` to see the full list of available parameters.
136   -
137   -You may also note that the :ref:`csiactobs` script has created a ``csiactobs.log``
138   -file that logs the complete energy range of the observations that have been
139   -selected. These values may be important for later, in particular if you
140   -intend to do a stacked analysis.
141   -
142   -In case that you have already a model definition XML file that describes a model
143   -of the celestial source distribution (a.k.a. a sky model), you may provide this
144   -sky model to :ref:`csiactobs` using the hidden ``inmodel`` parameter. The
145   -output model definition XML file will then contain the sky model and the
146   -background model, and you can use the model directly in a maximum likelihood
147   -fit.
148   -
149   -.. code-block:: bash
150   -
151   - $ csiactobs inmodel="mymodel.xml"
152   -
153   -Alternatively, you can merged models using the :ref:`csmodelmerge` script:
154   -
155   -.. code-block:: bash
156   -
157   - $ csmodelmerge
158   - Input model definition XML files [mymodel.xml bgmodels.xml]
159   - Output model definition XML file [combined_model.xml]
160   -
161   -.. note::
162   -
163   - The number of files that can be merged with :ref:`csmodelmerge` is not
164   - limited to two. The list of input file names may be either separated by
165   - whitespace or semi-colons, can be specified using wildcards, or can be given
166   - in an ASCII file (see :ref:`csmodelmerge`).
  157 + during the fit, and the corresponding model parameter have an error of
  158 + zero, although they are signalled as free.
167 159  
168 160 .. warning::
169   -
170   - Each model component in the input model definition XML files that are provided
171   - to :ref:`csmodelmerge` must have a different and unique name. Merging for
172   - example the same XML model twice will raise an exception.
173   -
174   -A list of available sky models can be found
175   -`here <http://gammalib.sourceforge.net/user_manual/modules/model.html>`_.
176   -If you are not familiar with creating sky models you should read the section
177   -about :ref:`modelling CTA data <um_models>`.
178   -
179   -
180   -Example XML files
181   ------------------
182   -To get familiar with the XML syntax and format, example files for an observation
183   -definition XML file and a model definition XML file are shown in the following.
184   -
185   -Observation definition XML file
186   -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
187   -
188   -.. code-block:: xml
189   -
190   - <?xml version="1.0" encoding="UTF-8" standalone="no"?>
191   - <observation_list title="observation list">
192   - <observation name="Crab Nebula" id="11111" instrument="HESS">
193   - <parameter name="EventList" file="/path/to/fits/file/events_11111.fits.gz" />
194   - <parameter name="EffectiveArea" file="/path/to/fits/file/aeff_11111.fits.gz" />
195   - <parameter name="PointSpreadFunction" file="/path/to/fits/file/psf_11111.fits.gz" />
196   - <parameter name="EnergyDispersion" file="/path/to/fits/file/edisp_11111.fits.gz" />
197   - <parameter name="Background" file="/path/to/fits/file/bgmodel_11111.fits.gz" />
198   - </observation>
199   - <observation name="Crab Nebula" id="11112" instrument="HESS">
200   - <parameter name="EventList" file="/path/to/fits/file/events_11112.fits.gz" />
201   - <parameter name="EffectiveArea" file="/path/to/fits/file/aeff_11112.fits.gz" />
202   - <parameter name="PointSpreadFunction" file="/path/to/fits/file/psf_11112.fits.gz" />
203   - <parameter name="EnergyDispersion" file="/path/to/fits/file/edisp_11112.fits.gz" />
204   - <parameter name="Background" file="/path/to/fits/file/bgmodel_11112.fits.gz" />
205   - </observation>
206   - <observation name="Crab Nebula" id="11113" instrument="HESS">
207   - <parameter name="EventList" file="/path/to/fits/file/events_11113.fits.gz" />
208   - <parameter name="EffectiveArea" file="/path/to/fits/file/aeff_11113.fits.gz" />
209   - <parameter name="PointSpreadFunction" file="/path/to/fits/file/psf_11113.fits.gz" />
210   - <parameter name="EnergyDispersion" file="/path/to/fits/file/edisp_11113.fits.gz" />
211   - <parameter name="Background" file="/path/to/fits/file/bgmodel_11113.fits.gz" />
212   - </observation>
213   - <observation name="Crab Nebula" id="11114" instrument="HESS">
214   - <parameter name="EventList" file="/path/to/fits/file/events_11114.fits.gz" />
215   - <parameter name="EffectiveArea" file="/path/to/fits/file/aeff_11114.fits.gz" />
216   - <parameter name="PointSpreadFunction" file="/path/to/fits/file/psf_11114.fits.gz" />
217   - <parameter name="EnergyDispersion" file="/path/to/fits/file/edisp_11114.fits.gz" />
218   - <parameter name="Background" file="/path/to/fits/file/bgmodel_11114.fits.gz" />
219   - </observation>
220   - </observation_list>
221   -
222   -
223   -Model definition XML file
224   -^^^^^^^^^^^^^^^^^^^^^^^^^
225   -
226   -.. code-block:: xml
227   -
228   - <?xml version="1.0" encoding="UTF-8" standalone="no"?>
229   - <source_library title="source library">
230   - <source name="bkg_11111" type="CTAIrfBackground" instrument="HESS" id="11111">
231   - <spectrum type="ConstantValue">
232   - <parameter name="Normalization" value="1" error="0" scale="1" min="0.01" max="100" free="1" />
233   - </spectrum>
234   - </source>
235   - <source name="bkg_11112" type="CTAIrfBackground" instrument="HESS" id="11112">
236   - <spectrum type="ConstantValue">
237   - <parameter name="Normalization" value="1" error="0" scale="1" min="0.01" max="100" free="1" />
238   - </spectrum>
239   - </source>
240   - <source name="bkg_11113" type="CTAIrfBackground" instrument="HESS" id="11113">
241   - <spectrum type="ConstantValue">
242   - <parameter name="Normalization" value="1" error="0" scale="1" min="0.01" max="100" free="1" />
243   - </spectrum>
244   - </source>
245   - <source name="bkg_11114" type="CTAIrfBackground" instrument="HESS" id="11114">
246   - <spectrum type="ConstantValue">
247   - <parameter name="Normalization" value="1" error="0" scale="1" min="0.01" max="100" free="1" />
248   - </spectrum>
249   - </source>
250   - <source name="Crab" type="PointSource">
251   - <spectrum type="PowerLaw">
252   - <parameter name="Prefactor" scale="1e-17" value="3.0" min="1e-07" max="1000.0" free="1"/>
253   - <parameter name="Index" scale="-1" value="2.5" min="0.0" max="+5.0" free="1"/>
254   - <parameter name="PivotEnergy" scale="1e6" value="1.0" min="0.01" max="1000.0" free="0"/>
255   - </spectrum>
256   - <spatialModel type="PointSource">
257   - <parameter name="RA" scale="1.0" value="83.6331" min="-360" max="360" free="1"/>
258   - <parameter name="DEC" scale="1.0" value="22.0145" min="-90" max="90" free="1"/>
259   - </spatialModel>
260   - </source>
261   - </source_library>
262   -
263   -.. note::
264   -
265   - It is important to ensure background models are properly linked to their
266   - respective observation. Therefore it is required to keep the attributes
267   - ``instrument`` and ``id`` the same for the observation and the corresponding
268   - background model. The tool :ref:`csiactobs` assures this automatically.
269   -
270   -
271   -Run ctselect
272   -------------
273   -To prepare the data for analysis, cuts have to be applied to the event data.
274   -The selection is performed by :ref:`ctselect`.
275   -This tool writes out selected event lists into the local directory. If the
276   -observation definition XML file contains several runs, it is recommended
277   -to first create a separate folder and specify this folder in the hidden
278   -``prefix`` argument.
279   -
280   -.. code-block:: bash
281   -
282   - $ mkdir selected
283   -
284   -.. code-block:: bash
285   -
286   - $ ctselect usethres=DEFAULT usepnt=yes prefix=selected/selected_
287   - Input event list or observation definition file [events.fits] obs.xml
288   - Lower energy limit (TeV) [0.1]
289   - Upper energy limit (TeV) [100.0]
290   - Radius of ROI (degrees) (0-180) [3.0] 2.5
291   - Start time (CTA MET in seconds) [0.0]
292   - End time (CTA MET in seconds) [0.0]
293   - Output event list or observation definition file [selected_events.fits] selected_obs.xml
294   -
295   -For IACT analysis, it is recommended to use the hidden parameter
296   -``usethres="DEFAULT"``. This instructs :ref:`ctselect` to extract the safe
297   -energy range from the instrument response functions and apply them to the
298   -data. This safe energy range is thus superior to the energy limit passed via
299   -the user parameters. In addition, to analyse the complete field of view, the
300   -parameter ``usepnt=yes`` uses, for each observation, the pointing position as
301   -centre for the selection radius. The radius parameter is dependent on the
302   -intrument, for an instrument with a 5° field of view, a radius of 2.5°
303   -seems reasonable. The time selection is not applied in the above example;
304   -specifying 0 as start and end time skips the time selection. For time-resolved
305   -analysis, it is important to know the MET time that is required to extract.
306   -The result of the selection step is written into the observation XML file
307   -``selected_obs.xml``, which now contains references to the new selected event
308   -FITS files.
309   -
310   -
311   -Unbinned analysis
312   ------------------
313   -Once the data is selected, the easiest way to analyse is an unbinned analysis.
314   -Note that the input model definition XML file must now contain the background
315   -models and source components to describe the field of view.
316   -
317   -.. code-block:: bash
318   -
319   - $ ctlike
320   - Event list, counts cube or observation definition file [events.fits] selected_obs.xml
321   - Source model [$CTOOLS/share/models/crab.xml] crab_models.xml
322   - Source model output file [crab_results.xml]
323   -
324   -The result of the fit was stored in ``crab_results.xml``. Note that fitted
325   -parameters, ``Prefactors`` in particular, typically use MeV as energy unit.
326   -To monitor the progress of the fit on the screen, one can simply run with the
327   -option ``debug=yes``. Alternatively, the logfile ``ctlike.log`` can be inspected
328   -after the fit.
329   -
330   -On default, energy dispersion is not considered in the fit. To switch on the
331   -usage of the energy migration matrix, the hidden parameter ``edisp=yes`` can
332   -be provided. Note that this will cause a significant reduction of the
333   -computing speed.
334   -
335   -
336   -Stacked analysis
337   -----------------
338   -In a stacked analysis the events of all observations are stacked into a single
339   -counts cube. This analysis mode is much faster than unbinned analysis when
340   -having a large dataset (e.g. > 100 hours). For this type of analysis, some
341   -intermediate data products have to be produced. The products are a binned counts
342   -cube, an exposure cube, a point spead function cube, optionally an energy
343   -dispersion cube, and a background cube.
344   -
345   -Bin events
346   -^^^^^^^^^^
347   -
348   -.. code-block:: bash
349   -
350   - $ ctbin
351   - Event list or observation definition file [events.fits] selected_obs.xml
352   - First coordinate of image center in degrees (RA or galactic l) (0-360) [83.63]
353   - Second coordinate of image center in degrees (DEC or galactic b) (-90-90) [22.01]
354   - Projection method (AIT|AZP|CAR|MER|MOL|STG|TAN) [CAR]
355   - Coordinate system (CEL - celestial, GAL - galactic) (CEL|GAL) [CEL]
356   - Image scale (in degrees/pixel) [0.02]
357   - Size of the X axis in pixels [200]
358   - Size of the Y axis in pixels [200]
359   - Algorithm for defining energy bins (FILE|LIN|LOG) [LOG]
360   - Start value for first energy bin in TeV [0.1] 0.5
361   - Stop value for last energy bin in TeV [100.0] 50
362   - Number of energy bins [20]
363   - Output counts cube [cntcube.fits]
364   -
365   -
366   -Create exposure cube
367   -^^^^^^^^^^^^^^^^^^^^
368   -After binning the events into a three-dimensional cube, an exposure cube has to
369   -be computed. The exposure is defined as the effective area times the dead-time
370   -corrected observation time. Each observation from the input container gets
371   -stacked in the resulting cube. The exposure is stored in units of :math:`cm^2 s`.
372   -The exposure cube does not have to contain the same binning as the event cube
373   -but for simplicity, the event cube can be passed to adopt the binning
374   -parameters. Note, however, that the exposure cube is defined in true sky
375   -coordinates and energy while the counts cube is defined in reconstructed sky
376   -coordinates and energy. Consequently, the sky area and energy range covered by
377   -the exposure cube should be slightly larger than that of the counts cube to
378   -accommodate for spill over of events due to the point spread function and energy
379   -dispersion. By computing the exposure cube on the same grid as the counts cube,
380   -the spill over of events from sources at the edge of cube will not be handled
381   -correctly. In the example below, however, no source at the edge of the field of
382   -view is present. Therefore, for simplicity, the count cube is used as input cube
383   -to extract the binning.
384   -
385   -This task of computing the exposure cube is is performed by :ref:`ctexpcube`.
386   -
387   -.. code-block:: bash
388   -
389   - $ ctexpcube
390   - Event list or observation definition file [NONE] selected_obs.xml
391   - Input counts cube file to extract exposure cube definition [NONE] cntcube.fits
392   - Output exposure cube file [expcube.fits]
393   -
394   -Alternatively, the exposure cube can be created with different binning than the
395   -counts cube:
396   -
397   -.. code-block:: bash
398   -
399   - $ ctexpcube
400   - Input event list or observation definition XML file [NONE] selected_obs.xml
401   - Input counts cube file to extract exposure cube definition [NONE]
402   - First coordinate of image center in degrees (RA or galactic l) (0-360) [83.63]
403   - Second coordinate of image center in degrees (DEC or galactic b) (-90-90) [22.01]
404   - Projection method (AIT|AZP|CAR|MER|MOL|STG|TAN) [CAR]
405   - Coordinate system (CEL - celestial, GAL - galactic) (CEL|GAL) [CEL]
406   - Image scale (in degrees/pixel) [0.02] 0.04
407   - Size of the X axis in pixels [200] 100
408   - Size of the Y axis in pixels [200] 100
409   - Lower energy limit (TeV) [0.5]
410   - Upper energy limit (TeV) [50.0]
411   - Number of energy bins [20] 30
412   - Output exposure cube file [expcube.fits]
413   -
414   -Create point spread function cube
415   -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
416   -As a next step for the stacked analysis, a cube containing the point spread
417   -function (PSF) must be computed. Since the PSF cannot be stored by one single
418   -parameter, the PSF cube computed by :ref:`ctpsfcube` has a fourth dimension.
419   -In each bin of the cube, the PSF is stored as a function of offset from source.
420   -The granularity of the PSF histogram is determined by the hidden parameter
421   -``anumbins`` (default: 200). Therefore, when passing the event cube to adopt
422   -the sky binning for the PSF cube, the resulting FITS file can become quite large
423   -due to the fourth dimension. Usually in IACT analysis, the PSF doesn't change
424   -too dramatically across the field of view. Therefore the user can think about
425   -reducing the spatial binning of the PSF cube:
426   -
427   -.. code-block:: bash
428   -
429   - $ ctpsfcube
430   - Input event list or observation definition XML file [NONE] selected_obs.xml
431   - Input counts cube file to extract PSF cube definition [NONE]
432   - First coordinate of image center in degrees (RA or galactic l) (0-360) [83.63]
433   - Second coordinate of image center in degrees (DEC or galactic b) (-90-90) [22.01]
434   - Projection method (AIT|AZP|CAR|MER|MOL|STG|TAN) [CAR]
435   - Coordinate system (CEL - celestial, GAL - galactic) (CEL|GAL) [CEL]
436   - Image scale (in degrees/pixel) [1.0] 0.2
437   - Size of the X axis in pixels [10] 20
438   - Size of the Y axis in pixels [10] 20
439   - Lower energy limit (TeV) [0.1] 0.5
440   - Upper energy limit (TeV) [100.0] 50
441   - Number of energy bins [20]
442   - Output PSF cube file [psfcube.fits]
443   -
444   -Depending on the required PSF precision, one could reduce the number of offset
445   -bins via the hidden parameter ``anumbins``:
446   -
447   -.. code-block:: bash
448   -
449   - $ ctpsfcube anumbins=100
450   -
451   -Create energy dispersion cube (optional)
452   -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
453   -In case energy dispersion should be considered used in the analysis, the tool
454   -:ref:`ctedispcube` computes the energy migration response for the given
455   -observations. Analoguous to the PSF cube, the energy dispersion cube is stored
456   -in a four-dimensional map. In each bin of the cube, the migration value
457   -``E_reco/E_true`` is stored. The granularity of the migration histogram can be
458   -steered via the hidden parameter ``migrabins`` (default: 100). In addition, the
459   -maximum migration value can be set via the hidden parameter ``migramax``
460   -(default: 2.0). Similar to the PSF cube, the energy dispersion cube FITS file
461   -can become quite large. Therefore, since the energy dispersion shouldn't vary
462   -too much across the sky region, one could reduce the spatial binning:
463   -
464   -.. code-block:: bash
465   -
466   - $ ctedispcube
467   - Input event list or observation definition XML file [NONE] selected_obs.xml
468   - Input counts cube file to extract energy dispersion cube definition [NONE]
469   - First coordinate of image center in degrees (RA or galactic l) (0-360) [83.63]
470   - Second coordinate of image center in degrees (DEC or galactic b) (-90-90) [22.01]
471   - Projection method (AIT|AZP|CAR|MER|MOL|STG|TAN) [CAR]
472   - Coordinate system (CEL - celestial, GAL - galactic) (CEL|GAL) [CEL]
473   - Image scale (in degrees/pixel) [1.0] 0.2
474   - Size of the X axis in pixels [10] 20
475   - Size of the Y axis in pixels [10] 20
476   - Lower energy limit (TeV) [0.1] 0.5
477   - Upper energy limit (TeV) [100.0] 50
478   - Number of energy bins [20]
479   - Output energy dispersion cube file [edispcube.fits]
480   -
481   -Create background cube
482   -^^^^^^^^^^^^^^^^^^^^^^
483   -Last but not least, for a stacked IACT analysis a cube containing the background
484   -rate in sky coordinates and reconstructed energy has to be computed. This task
485   -is performed by :ref:`ctbkgcube`. The binning here can also differ from the counts
486   -cube. For simplicity, however, the example below uses the counts cube to adopt
487   -the binning.
488   -
489   -.. code-block:: bash
490   -
491   - $ ctbkgcube debug=yes
492   - Input event list or observation definition XML file [NONE] selected_obs.xml
493   - Input counts cube file to extract background cube definition [NONE] cntcube.fits
494   - Input model XML file [NONE] crab_models.xml
495   - Output background cube file [bkgcube.fits]
496   - Output model XML file [NONE] binned_models.xml
497   -
498   -Note that this tool also requires the parameters of an input and output model.
499   -In the model XML file that came out of :ref:`csiactobs`, one background model
500   -per observation is included. This models get merged and averaged in the background
501   -sky cube. In the output model the background models per obseervation will be
502   -removed. Instead, a global background model for the newly created background
503   -cube is included. Sky models present in the input model XML file will also be
504   -included in the new XML file, which subsequently can be used for stacked
505   -:ref:`ctlike`.
506   -
507   -Example for stacked model XML file
508   -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
509   -The output model of :ref:`ctbkgcube` looks the following:
510   -
511   -.. code-block:: xml
512   -
513   - <?xml version="1.0" encoding="UTF-8" standalone="no"?>
514   - <source_library title="source library">
515   - <source name="BackgroundModel" type="CTACubeBackground" instrument="CTA,HESS,MAGIC,VERITAS">
516   - <spectrum type="PowerLaw">
517   - <parameter name="Prefactor" value="1" error="0" scale="1" min="0.01" max="100" free="1" />
518   - <parameter name="Index" value="0" error="0" scale="1" min="-5" max="5" free="1" />
519   - <parameter name="PivotEnergy" value="1" scale="1e+06" free="0" />
520   - </spectrum>
521   - </source>
522   - <source name="Crab" type="PointSource">
523   - <spectrum type="PowerLaw">
524   - <parameter name="Prefactor" scale="1e-17" value="3.0" min="1e-07" max="1000.0" free="1"/>
525   - <parameter name="Index" scale="-1" value="2.48" min="0.0" max="+5.0" free="1"/>
526   - <parameter name="PivotEnergy" scale="1e6" value="1." min="0.01" max="1000.0" free="0"/>
527   - </spectrum>
528   - <spatialModel type="PointSource">
529   - <parameter name="RA" scale="1.0" value="83.6331" min="-360" max="360" free="0"/>
530   - <parameter name="DEC" scale="1.0" value="22.0145" min="-90" max="90" free="0"/>
531   - </spatialModel>
532   - </source>
533   - </source_library>
534   -
535   -The background model with ``type=CTACubeBackground`` is used to scale the
536   -background cube stored in the FITS file created by :ref:`ctbkgcube`.
537   -
538   -Run ctlike
539   -^^^^^^^^^^
540   -Having all the intermediate data products ready, a stacked analysis can be
541   -conducted using :ref:`ctlike`.
542   -
543   -.. code-block:: bash
544   -
545   - $ ctlike
546   - Input event list, counts cube or observation definition XML file [events.fits] cntcube.fits
547   - Input exposure cube file (only needed for stacked analysis) [NONE] expcube.fits
548   - Input PSF cube file (only needed for stacked analysis) [NONE] psfcube.fits
549   - Input background cube file (only needed for stacked analysis) [NONE] bkgcube.fits
550   - Input model XML file [binned_models.xml]
551   - Output model XML file [binned_results.xml]
552   -
553   -Note that when passing an counts cube to :ref:`ctlike`, the tool behaves
554   -differently than in unbinned mode. It queries directly for the additional
555   -ingredients for the stacked analysis. It is important to pass the background
556   -model generated by :ref:`ctbkgcube` here to ensure the proper modelling of
557   -the background in the fit.
558   -
559   -To consider also the energy dispersion, you have to execute the :ref:`ctlike`
560   -tool with the hidden parameter ``edisp=yes``. The tool will then also query for
561   -the energy dispersion cube:
562   -
563   -.. code-block:: bash
564   -
565   - $ ctlike debug=yes
566   - Input event list, counts cube or observation definition XML file [events.fits] cntcube.fits
567   - Input exposure cube file (only needed for stacked analysis) [NONE] expcube.fits
568   - Input PSF cube file (only needed for stacked analysis) [NONE] psfcube.fits
569   - Input energy dispersion cube file (only needed for stacked analysis) [NONE] edispcube.fits
570   - Input background cube file (only needed for stacked analysis) [NONE] bkgcube.fits
571   - Input model XML file [binned_models.xml]
572   - Output model XML file [binned_results.xml]
  161 + The effective area background model generated by :ref:`csiactobs` is very
  162 + simplistic, and should not be used for a serious data analysis of H.E.S.S.
  163 + data. Please follow the procedure :ref:`hess_dr1` for the generation of
  164 + a more reliable background model.
... ...
doc/source/users/tutorials/howto/iact/copy.rst
... ... @@ -3,66 +3,162 @@
3 3 Copying IACT data
4 4 ==================
5 5  
  6 + .. admonition:: What you will learn
  7 +
  8 + You will learn how copy data from an IACT data store to your disk.
  9 +
6 10 Before you start
7 11 ----------------
8   -Currently the way how IACT data is stored is quite prototypical. The ctools can work
9   -with IACT data that is stored following the `specifications here <http://gamma-astro-data-formats.readthedocs.org/en/latest/data_storage/index.html>`_.
10   -The script :ref:`csiactcopy` can be used to transfer these data from one system to another.
11   -IACT data is the property of collaborations (such as HESS, MAGIC or VERITAS) and not publicly available. If you are member of one of these
12   -collaborations, you should have access to a central place where the FITS data is stored and maintained.
13   -
14   -Mount remote file system
15   -------------------------
16   -Note that :ref:`csiactcopy` does not include functionality to establish a conection to a remote file system.
17   -For the moment, it is capable of copying data from one location to another. Therefore it is necessary to
18   -mount the remote file system where the data is located on the local machine. This can be achieved using e.g.
19   -`sshfs <https://github.com/libfuse/sshfs>`_:
  12 +
  13 +Currently the way how IACT data is stored is quite prototypical.
  14 +The ctools can work with IACT data that is stored following the specifications
  15 +provided in the
  16 +`Data formats for gamma-ray astronomy <http://gamma-astro-data-formats.readthedocs.org/en/latest/data_storage/index.html>`_.
  17 +The script :ref:`csiactcopy` can be used to transfer IACT data from one system
  18 +to another.
  19 +IACT data is the property of collaborations (such as HESS, MAGIC or VERITAS)
  20 +and not publicly available, except for the recent
  21 +`public release of some H.E.S.S. data <https://www.mpi-hd.mpg.de/hfm/HESS/pages/dl3-dr1>`_.
  22 +In this tutorial we use these public data.
  23 +After downloading the data and uncompression the tarball you should have a
  24 +folder named ``hess_dl3_dr1`` in your current workingcdirectory with the
  25 +following structure:
20 26  
21 27 .. code-block:: bash
22 28  
23   - $ sshfs user@remote.server.com:/path/to/remote/fits/data/ /path/to/mountpoint/
  29 + hess_dl3_dr1/
  30 + hess_dl3_dr1/data
  31 + hess_dl3_dr1/hdu-index.fits.gz
  32 + hess_dl3_dr1/obs-index.fits.gz
  33 + hess_dl3_dr1/README.txt
  34 +
  35 +You have to add the file ``master.json`` to this folder with the following
  36 +content:
  37 +
  38 +.. code-block:: json
  39 +
  40 + {
  41 + "datasets": [
  42 + {
  43 + "name": "hess_dl3_dr1",
  44 + "hduindx": "hdu-index.fits.gz",
  45 + "obsindx": "obs-index.fits.gz",
  46 + "Total observations": 106,
  47 + "Total time": "48.6 hours",
  48 + "Total events": 662826,
  49 + "Disk usage": "43 MB",
  50 + "Date created": "2018-09-10"
  51 + }
  52 + ]
  53 + }
  54 +
  55 +.. note::
  56 + If you are member of an IACT collaboration and if the collaboration has
  57 + setup a datastore in the
  58 + `compliant format <http://gamma-astro-data-formats.readthedocs.org/en/latest/data_storage/index.html>`_
  59 + you can also establish a remote connection to this datastore using for
  60 + example
  61 +
  62 + .. code-block:: bash
  63 +
  64 + $ sshfs user@remote.server.com:/path/to/remote/fits/data/ /project-data/hess
  65 +
  66 + The success of this operation can be verified by checking if there is a file
  67 + called ``master.json`` in the mounted directory.
  68 +
  69 +
  70 +Check what data are available
  71 +-----------------------------
  72 +
  73 +Before you start an analysis, it is important that you know what kind of data
  74 +are available for analysis.
  75 +In the `current scheme <http://gamma-astro-data-formats.readthedocs.org/en/latest/data_storage/index.html>`_
  76 +every dataset is represented by a unique name, in our case ``hess_dl3_dr1``.
  77 +You have to pass this string to some set of scripts to find and use the data
  78 +for analysis.
  79 +
  80 +To check what datasets are available you should run the :ref:`csiactdata`
  81 +script:
  82 +
  83 +.. code-block:: bash
  84 +
  85 + $ csiactdata debug=yes
  86 + Path were data is located [] /project-data/hess/hess_dl3_dr1
  87 +
  88 +This script will list the names of available datasets. Here at excerpt of the
  89 +log file ``csiactdata.log``:
  90 +
  91 +.. code-block:: none
  92 +
  93 + 2019-04-05T13:10:35: +======================+
  94 + 2019-04-05T13:10:35: | Data storage entries |
  95 + 2019-04-05T13:10:35: +======================+
  96 + 2019-04-05T13:10:35: Master index file .........: /project-data/hess/hess_dl3_dr1/master.json
  97 + 2019-04-05T13:10:35:
  98 + 2019-04-05T13:10:35: +------------------------+
  99 + 2019-04-05T13:10:35: | Available data configs |
  100 + 2019-04-05T13:10:35: +------------------------+
  101 + 2019-04-05T13:10:35: === hess_dl3_dr1 ===
  102 + 2019-04-05T13:10:35: Name ......................: hess_dl3_dr1
  103 + 2019-04-05T13:10:35: Observation index .........: obs-index.fits.gz
  104 + 2019-04-05T13:10:35: HDU index .................: hdu-index.fits.gz
  105 + 2019-04-05T13:10:35: Date created ..............: 2018-09-10
  106 + 2019-04-05T13:10:35: Total events ..............: 662826
  107 + 2019-04-05T13:10:35: Total time ................: 48.6 hours
  108 + 2019-04-05T13:10:35: Disk usage ................: 43 MB
  109 + 2019-04-05T13:10:35: Total observations ........: 105
24 110  
25   -The success of this operation can be verified by checking if there is a file called ``master.json``
26   -in the mounted directory.
27 111  
28 112 Copy data
29 113 ---------
30   -After mounting the file system to a folder of your choice (``/path/to/mountpoint/`` in this case), the copying of the data can be initiated.
31   -To monitor the progress on the screen, the hidden parameter ``debug=yes`` can be used. To increase the chattiness of the output,
32   -the hidden parameter ``chatter`` can e.g. be set to 3.
33 114  
34   -.. code-block:: bash
35   -
36   - $ csiactcopy debug=yes chatter=3
37   - Location of remote master file [/path/to/mountpoint/master.json]
38   - Name of FITS production to download [iact-fits]
39   - Destination path of FITS data [/path/to/local/fits/data/]
40   -
41   -The name of the FITS production must be available in the remote
42   -``master.json`` file. In case the names of the available productions are unkown
43   -there are two options to determine an appropriate name:
44   -
45   -* Run :ref:`csiactdata` and set as ``datapath`` the mountpoint
46   -* Try any name, :ref:`csiactcopy` will throw a ``RuntimeError`` with the available production names if the input name is not found
47   -
48   -Copy only a subset of the data
49   -------------------------------
50   -If the user is interested only in a small subset of the remote FITS production,
51   -it is also possible to download only parts of the data. The user simply has to provide an
52   -ASCII file containing a list of observation IDs to be downloaded (one ID per line). The hidden
53   -parameter ``runlist`` then needs to be specified.
  115 +You can now copy the data from the data store to your disk. In this example we
  116 +assume that the H.E.S.S. datastore is at ``/project-data/hess/hess_dl3_dr1``.
  117 +To do the copy you should type
54 118  
55 119 .. code-block:: bash
56   -
57   - $ csiactcopy runlist=myrunlist.txt debug=yes
58   -
  120 +
  121 + $ csiactcopy
  122 + Location of remote master file [/path/to/mountpoint/master.json] /project-data/hess/hess_dl3_dr1/master.json
  123 + Name of FITS production to download [iact-fits] hess_dl3_dr1
  124 + Destination path of FITS data [/path/to/local/fits/data/] data
  125 +
  126 +.. note::
  127 + To monitor the progress on the screen you should set the hidden parameter
  128 + ``debug=yes``. You may also increase the chattiness of the output by
  129 + increasing the hidden parameter ``chatter`` from its default value of 2
  130 + to for example 3.
  131 +
  132 +.. note::
  133 + If you are only interested in a subset of the dataset you can also download
  134 + only that subset. For that purpose you have to specify an ASCII file
  135 + containing a list of observation identifiers to be downloaded, e.g.
  136 +
  137 + .. code-block:: none
  138 +
  139 + 20136
  140 + 20137
  141 + 20151
  142 +
  143 + You can then run :ref:`csiactcopy` by specifying the ASCII filename
  144 + for the hidden ``runlist`` parameter:
  145 +
  146 + .. code-block:: bash
  147 +
  148 + $ csiactcopy runlist=runlist.txt
  149 + Location of remote master file [/project-data/hess/hess_dl3_dr1/master.json]
  150 + Name of FITS production to download [hess_dl3_dr1]
  151 + Destination path of FITS data [data]
  152 +
  153 +
59 154 Troubleshooting
60 155 ---------------
61   -In case a download was interupted (e.g. the remote file system was disconnected), the script can simply be rerun.
62   -The index files which list the files available on the system are refreshed after the copying process.
63   -The hidden boolean parameter ``clobber`` (default=yes) specifies if the user wishes that local content should be overwritten with
64   -remote content. In turn, specifying ``clobber=no`` can speed up the process since remote content does not need to be copied
65   -if it is already available on the user machine.
66   -
67   -To verify the copying process was successful, it is recommended to run :ref:`csiactdata` after the process in order to list available productions
68   -on the local machine. The output should now include the copied FITS production name.
  156 +In case a download was interupted (e.g. the remote file system was
  157 +disconnected), the script can simply be rerun.
  158 +The index files, which list the files available on the system, are refreshed
  159 +after the copying process.
  160 +The hidden boolean parameter ``clobber`` (by default ``yes``) specifies if the
  161 +user wishes that local content should be overwritten with remote content.
  162 +In turn, specifying ``clobber=no`` can speed up the process since remote
  163 +content does not need to be copied if it is already available on the user
  164 +machine.
... ...
doc/source/users/tutorials/howto/iact/high_level.rst deleted
1   -.. _sec_iact_high_level:
2   -
3   -High level analysis tools for IACT data
4   -=======================================
5   -
6   -This section describes and explains how to create high level output. The
7   -following scripts and tools are explained:
8   -
9   -* :ref:`csobsinfo` (inspect and observation XML files)
10   -* :ref:`csmodelinfo` (inspect model XML files)
11   -* :ref:`ctulimit` (compute upper limits)
12   -* :ref:`cterror` (compute asymmetric errors)
13   -* :ref:`csspec` (compute spectral points)
14   -* :ref:`cslightcrv` (compute a light curve)
15   -* :ref:`csresmap` (compute a residual map)
16   -* :ref:`cttsmap` (compute a test statistics map)
17   -
18   -Inspecting observation definition files
19   ----------------------------------------
20   -After having created an observation definition file, the user might want to
21   -know a summary of its content. Important quantities usually are the number of
22   -observations, the total livetime, or the mean zenith angle. For this purpose,
23   -:ref:`csobsinfo` inspects an observation XML container for its content.
24   -
25   -.. code-block:: bash
26   -
27   - $ csobsinfo
28   - Event list, counts cube, or observation definition file [obs.xml] selected_obs.xml
29   -
30   -This dumps general information of the observation container in a logfile ``csobsinfo.log``.
31   -For a direct output on the screen, pass the hidden parameter ``debug=yes`` on the command line:
32   -
33   -.. code-block:: bash
34   -
35   - $ csobsinfo debug=yes
36   - Event list, counts cube, or observation definition file [obs.xml] selected_obs.xml
37   - ...
38   - 2016-02-19T15:43:08: +=========+
39   - 2016-02-19T15:43:08: | Summary |
40   - 2016-02-19T15:43:08: +=========+
41   - 2016-02-19T15:43:08: Unbinned observations .....: 4
42   - 2016-02-19T15:43:08: Number of events ..........: 6085
43   - 2016-02-19T15:43:08: Binned observations .......: 0
44   - 2016-02-19T15:43:08: Number of bins ............: 0
45   - 2016-02-19T15:43:08:
46   - 2016-02-19T15:43:08: === Pointings ===
47   - 2016-02-19T15:43:08: Mean zenith angle .........: 46.89
48   - 2016-02-19T15:43:08: Mean azimuth angle ........: 13.63
49   - 2016-02-19T15:43:08:
50   - 2016-02-19T15:43:08: === Energy range ===
51   - 2016-02-19T15:43:08: Emin ......................: 461.48 GeV
52   - 2016-02-19T15:43:08: Emax ......................: 37.8832 TeV
53   - 2016-02-19T15:43:08:
54   - 2016-02-19T15:43:08: === Time range ===
55   - 2016-02-19T15:43:08: Start [MJD] ...............: 53343.9212153
56   - 2016-02-19T15:43:08: Stop [MJD] ................: 53347.9321065
57   - 2016-02-19T15:43:08: Total ontime ..............: 6877.00 s = 114.62 min = 1.91 h
58   - 2016-02-19T15:43:08: Total livetime ............: 6427.52 s = 107.13 min = 1.79 h
59   - ...
60   -
61   -Further options are described on the reference page :ref:`csobsinfo`.
62   -
63   -Inspecting model XML files
64   ---------------------------
65   -Similar to the observation containers, model XML file can be inspected, too.
66   -The tool :ref:`csmodelinfo` gives a summary of the model container. In particular,
67   -the number of free parameters, number of sky and background models and also number of
68   -parameters that are at their limits might be of interest.
69   -
70   -.. code-block:: bash
71   -
72   - $ csmodelinfo debug=yes
73   - Input model definiton file XML file [$CTOOLS/share/models/crab.xml]
74   - Output DS9 region file [ds9.reg]
75   - 2016-02-19T15:50:40: +=========+
76   - 2016-02-19T15:50:40: | Summary |
77   - 2016-02-19T15:50:40: +=========+
78   - 2016-02-19T15:50:40: === Instrument specific models ===
79   - 2016-02-19T15:50:40: All .......................: 1
80   - 2016-02-19T15:50:40: CTA .......................: 1
81   - 2016-02-19T15:50:40: === Model types ===
82   - 2016-02-19T15:50:40: PointSource ...............: 1
83   - 2016-02-19T15:50:40: CTAIrfBackground ..........: 1
84   - 2016-02-19T15:50:40:
85   - 2016-02-19T15:50:40: +=======================+
86   - 2016-02-19T15:50:40: | Parameter information |
87   - 2016-02-19T15:50:40: +=======================+
88   - 2016-02-19T15:50:40: All parameters ............: 10
89   - 2016-02-19T15:50:40: Fixed parameters ..........: 6
90   - 2016-02-19T15:50:40: Free parameters (total) ...: 4
91   - 2016-02-19T15:50:40: Free background parameters : 2
92   - 2016-02-19T15:50:40: Free source parameters ....: 2
93   - 2016-02-19T15:50:40: Free spectral parameters ..: 2
94   - 2016-02-19T15:50:40: Parameters at limit .......: 0
95   -
96   -Similar to :ref:`csobsinfo`, the script also generates a ds9 region file
97   -including the positions of all the sky model components.
98   -
99   -Compute upper limit
100   --------------------
101   -Very often in gamma-ray astronomy sources are at the verge of detection or even not detectable.
102   -In such cases, it is useful to derive an upper limit using :ref:`ctulimit`.
103   -
104   -.. code-block:: bash
105   -
106   - $ ctulimit
107   - Input event list, counts cube or observation definition XML file [selected_obs.xml]
108   - Input model XML file [crab_models.xml]
109   - Source of interest [Crab]
110   -
111   -The upper limit will be stored in the log file. To get the limit printed on screen, use the hidden parameter
112   -``debug=yes``.
113   -
114   -Compute asymmetric errors
115   --------------------------
116   -When an analysis approaches its final state, it makes sense to have asymmetric errors on the parameters of the
117   -source of interest. For this purpose, the tool :ref:`cterror` can be used:
118   -
119   -.. code-block:: bash
120   -
121   - $ cterror debug=yes
122   - Input event list, counts cube or observation definition XML file [selected_obs.xml]
123   - Input model XML file [crab_models.xml]
124   - Source of interest [Crab]
125   - Output model XML file [cterror_results.xml]
126   -
127   -The output model does not contain asymmetric errors yet. The positive and negative uncertainties can be read from the
128   -logfile (or from screen if ``debug=yes`` was specified).
129   -
130   -Compute spectral points
131   ------------------------
132   -A very common task in astronomy is to compute spectral data points. To determine a spectral data point, a small energy
133   -range is considered and the model prefactor and its uncertainty is evaluated. The tool :ref:`csspec` works on both, binned
134   -and unbinned data. The hidden parameter ``edisp=yes`` can be specified in both cases to consider the energy migration
135   -matrix in the fit.
136   -
137   -.. note::
138   -
139   - The resulting spectral points are provided as a function of reconstructed energy.
140   -
141   -Unbinned
142   -^^^^^^^^
143   -
144   -.. code-block:: bash
145   -
146   - $ csspec debug=yes
147   - Input event list, counts cube, or observation definition XML file [events.fits] selected_obs.xml
148   - Input model XML file [$CTOOLS/share/models/crab.xml] crab_models.xml
149   - Source name [Crab]
150   - Algorithm for defining energy bins (FILE|LIN|LOG) [LOG]
151   - Lower energy limit for spectral points (TeV) [0.1] 0.5
152   - Upper energy limit for spectral points (TeV) [100.0] 50.0
153   - Number of spectral points (1-10000) [20] 10
154   - Output spectrum file [spectrum.fits]
155   -
156   -Binned
157   -^^^^^^
158   -
159   -.. code-block:: bash
160   -
161   - $ csspec debug=yes
162   - Input event list, counts cube, or observation definition XML file [cntcube.fits]
163   - Input exposure cube file (only needed for stacked analysis) [expcube.fits]
164   - Input PSF cube file (only needed for stacked analysis) [psfcube.fits]
165   - Input background cube file (only needed for stacked analysis) [bkgcube.fits]
166   - Input model XML file [binned_models.xml]
167   - Source name [Crab]
168   - Number of spectral points (1-10000) [10]
169   - Lower energy limit for spectral points (TeV) [0.5]
170   - Upper energy limit for spectral points (TeV) [50.0]
171   - Output spectrum file [spectrum.fits]
172   -
173   -Plot spectral points
174   -^^^^^^^^^^^^^^^^^^^^
175   -Instead of supporting a plotting library, simple example scripts are available to visualise data products.
176   -Have a look at ``$CTOOLS/examples/show_spectrum.py`` how a spectrum can be plotted using ``matplotlib``.
177   -To have first glance at the above computed spectrum one can use this script in the following way:
178   -
179   -.. code-block:: bash
180   -
181   - $ python $CTOOLS/examples/show_spectrum.py spectrum.fits
182   -
183   -
184   -Compute light curves
185   ---------------------
186   -Euqally common in astronomy are light curves from a time-series analysis. The tool :ref:`cslightcrv` takes as input only
187   -an unbinned observation container (or a single event list). Counts cube are not possible as inout since the time informstion
188   -is lost during the binning procedure. Nevertheless, in case of large time bins, it is possible to require a binned analysis
189   -in each time bin. The tool will accordingly slice the event list and create the data products for the stacked analysis in each
190   -time span.
191   -
192   -.. code-block:: bash
193   -
194   - $ cslightcrv
195   - Input event list, counts cube, or observation definition XML file [selected_obs.xml]
196   - Input model XML file [crab_models.xml]
197   - Source name [Crab]
198   - Algorithm for defining time bins (FILE|LIN|GTI) [GTI]
199   - Number of energy bins per light curve bin (0=unbinned) [0]
200   - Lower energy limit of events (TeV) [0.1] 0.5
201   - Upper energy limit of events (TeV) [100.0] 50
202   - Output light curve file [lightcurve.fits]
203   -
204   -Compute a residual map
205   -----------------------
206   -A frequent means to visually inspect the fitted model with respect to the input data is to create residual maps.
207   -Using :ref:`csresmap` such a map can easily be computed. The tool internally bins the data according to user parameters.
208   -(if data is not already provided in a binned state). Taking into account the corresponding IRFs,
209   -:reF:`csresmap` computes a model map (running :ref:`ctmodel`). Subsequently, there are several choices how data
210   -and model should be compared. There are three options:
211   -
212   -* ``SUB``: the subtraction of the model from the counts. The resulting map will display differences in absolute counts.
213   -* ``SUBDIV``: the subtraction and division by the model. The resulting map will display relative differences of the data with respect to the model.
214   -* ``SUBDIVSQRT``: the subtraction and division by the square root of the model. The resulting map will display an approximation of a residual significance map. In case of sufficient count statistic per bin, the bin value represents the significance.
215   -
216   -Example unbinned observation container
217   -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
218   -
219   -.. code-block:: bash
220   -
221   - $ csresmap
222   - Input event list, counts cube, or observation definition XML file [selected_obs.xml]
223   - Input model XML file [crab_results.xml]
224   - First coordinate of image center in degrees (RA or galactic l) (0-360) [83.63]
225   - Second coordinate of image center in degrees (DEC or galactic b) (-90-90) [22.01]
226   - Coordinate System (CEL|GAL) [CEL]
227   - Projection method (AIT|AZP|CAR|MER|MOL|STG|TAN) [CAR]
228   - Size of the X axis in pixels [100]
229   - Size of the Y axis in pixels [100]
230   - Pixel size (deg/pixel) [0.02]
231   - Residual map computation algorithm (SUB|SUBDIV|SUBDIVSQRT) [SUBDIV]
232   - Output residual map file [resmap.fits]
233   -
234   -Example binned/stacked observation container
235   -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
236   -The tool behaves differently if the input observation parameter is a count cube. It will query for the additional binned reponse components.
237   -
238   -.. code-block:: bash
239   -
240   - $ csresmap
241   - Input event list, counts cube, or observation definition XML file [cntcube.fits]
242   - Input model cube file (generated with ctmodel) [NONE]
243   - Input exposure cube file (only needed for stacked analysis) [expcube.fits]
244   - Input PSF cube file (only needed for stacked analysis) [psfcube.fits]
245   - Input background cube file (only needed for stacked analysis) [bkgcube.fits]
246   - Input model XML file [binned_results.xml]
247   - Residual map computation algorithm (SUB|SUBDIV|SUBDIVSQRT) [SUBDIV]
248   - Output residual map file [resmap.fits]
249   -
250   -Note that the tool also queries for an input model cube file (which we set to ``NONE`` here). This is very convenient in case the model cube has already been precomputed
251   -using :ref:`ctmodel`.
252   -
253   -Example with input model cube
254   -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
255   -In case a model cube can be provided, :ref:`csresmap` simply collapses both cube into one skymap and applies the specified algorithm.
256   -
257   -.. code-block:: bash
258   -
259   - $ csresmap
260   - Input event list, counts cube, or observation definition XML file [cntcube.fits]
261   - Input model cube file (generated with ctmodel) [modcube.fits]
262   - Residual map computation algorithm (SUB|SUBDIV|SUBDIVSQRT) [SUB]
263   - Output residual map file [resmap.fits]
264   -
265   -
266   -Compute a test statistics (TS) map
267   -----------------------------------
268   -The test statistics quantity is a more precise measurement of statistic than the ``SUBDIVSQRT`` option in :ref:`csresmap`.
269   -To get familiar with the concept of this procedure, :ref:`read more about TS maps <howto_ts>`.
270   -
271   -.. code-block:: bash
272   -
273   - $ cttsmap
274   - Input event list, counts cube or observation definition XML file [selected_obs.xml]
275   - Input model XML file [crab_results.xml]
276   - Test source name [Crab]
277   - First coordinate of image center in degrees (RA or galactic l) (0-360) [83.63]
278   - Second coordinate of image center in degrees (DEC or galactic b) (-90-90) [22.01]
279   - Projection method (AIT|AZP|CAR|MER|MOL|STG|TAN) [CAR]
280   - Coordinate system (CEL - celestial, GAL - galactic) (CEL|GAL) [CEL]
281   - Image scale (in degrees/pixel) [0.05]
282   - Size of the X axis in pixels [20]
283   - Size of the Y axis in pixels [20]
284   - Output Test Statistic map file [tsmap.fits]
285   -
286   -The output file will contain a sky map holding the TS map. In complementary extensions, for each free parameter of the source model, a map displaying the parameter
287   -value is added to the output file. For instance a map of the prefactor can be visualised with `DS9 <http://ds9.si.edu/>`_ as follows:
288   -
289   -.. code-block:: bash
290   -
291   - $ ds9 tsmap.fits[Prefactor]
292   -
293   -Since :ref:`cttsmap` has to run a parameter optimisation in each skymap bin, it is very time comsuming to
294   -compute a fine-granulated TS sky map. Read here how to split up the :ref:`computation into several jobs <sec_iact_tips>`.
295   -Similar as many other tools :ref:`cttsmap` can also work on binned observation input:
296   -
297   -.. code-block:: bash
298   -
299   - $ cttsmap debug=yes
300   - Input event list, counts cube or observation definition XML file [cntcube.fits]
301   - Input exposure cube file (only needed for stacked analysis) [expcube.fits]
302   - Input PSF cube file (only needed for stacked analysis) [psfcube.fits]
303   - Input background cube file (only needed for stacked analysis) [bkgcube.fits]
304   - Input model XML file [binned_results.xml]
305   - Test source name [Crab]
306   - First coordinate of image center in degrees (RA or galactic l) (0-360) [83.63]
307   - Second coordinate of image center in degrees (DEC or galactic b) (-90-90) [22.01]
308   - Projection method (AIT|AZP|CAR|MER|MOL|STG|TAN) [CAR]
309   - Coordinate system (CEL - celestial, GAL - galactic) (CEL|GAL) [CEL]
310   - Image scale (in degrees/pixel) [0.05]
311   - Size of the X axis in pixels [20]
312   - Size of the Y axis in pixels [20]
313   - Output Test Statistic map file [tsmap.fits]
314   -
315   -
doc/source/users/tutorials/howto/iact/index.rst
1 1 .. _sec_iact:
2 2  
3   -Analysing current IACT data
4   -===========================
  3 +Analysing data from existing IACTs
  4 +==================================
  5 +
  6 +This tutorial is specifically written for the analysis of H.E.S.S. data but
  7 +it may also be applicable to VERITAS and MAGIC, provided that these instruments
  8 +set up a data store that is compliant with the format described in
  9 +`Data formats for gamma-ray astronomy <http://gamma-astro-data-formats.readthedocs.org/en/latest/data_storage/index.html>`_.
5 10  
6 11 .. toctree::
7 12 :maxdepth: 1
8 13  
9 14 copy
  15 + select
10 16 analysis
11   - high_level
12   - simulation
13   - tips
14 17  
... ...
doc/source/users/tutorials/howto/iact/select.rst 0 → 100755
  1 +.. _sec_iact_selection:
  2 +
  3 +Selecting IACT observations
  4 +===========================
  5 +
  6 + .. admonition:: What you will learn
  7 +
  8 + You will learn how to select observations from a dataset.
  9 +
  10 +As next step you have to select observations from a dataset and to build
  11 +an observation definition XML file for the analysis.
  12 +
  13 +
  14 +Selecting observations
  15 +----------------------
  16 +
  17 +First run :ref:`csfindobs` to select the observations of interest.
  18 +The most common search for observations is by pointing direction on the sky,
  19 +and you do this by typing
  20 +
  21 +.. code-block:: bash
  22 +
  23 + $ csfindobs
  24 + Path were data are located (NONE uses $VHEFITS environment variable) [NONE] data
  25 + Name of FITS production (Run csiactdata to view your options) [prod-name] hess_dl3_dr1
  26 + Right ascension of selection region centre (deg) [83.63]
  27 + Declination of selection region centre (deg) [22.51]
  28 + Search radius of selection region (deg) [2.5]
  29 + Output runlist file [runlist.lis]
  30 +
  31 +This creates an output ASCII file named ``runlist.lis`` that contains the
  32 +identifiers for all observations with pointing directions located within 2.5 deg
  33 +around the position of the Crab nebula. The content of the file is
  34 +
  35 +.. code-block:: none
  36 +
  37 + 23523
  38 + 23526
  39 + 23559
  40 + 23592
  41 +
  42 +hence four observations were found in the production. An excerpt of the log
  43 +file ``csfindobs.log`` of the run is shown below:
  44 +
  45 +.. code-block:: none
  46 +
  47 + 2019-04-05T13:22:08: +===================+
  48 + 2019-04-05T13:22:08: | Find observations |
  49 + 2019-04-05T13:22:08: +===================+
  50 + 2019-04-05T13:22:08: Expression ................: ANGSEP(83.63,22.51,RA_PNT,DEC_PNT)<=2.5&&QUALITY<=0
  51 + 2019-04-05T13:22:08: Observations ..............: 4
  52 + 2019-04-05T13:22:08: Observation 1 .............: 23523
  53 + 2019-04-05T13:22:08: Observation 2 .............: 23526
  54 + 2019-04-05T13:22:08: Observation 3 .............: 23559
  55 + 2019-04-05T13:22:08: Observation 4 .............: 23592
  56 +
  57 +.. note::
  58 + You may add further selection criteria using the hidden ``expression``
  59 + parameter.
  60 + You may use any FITS selection expression that is
  61 + `supported by the cfitsio row filtering specification <https://heasarc.gsfc.nasa.gov/docs/software/fitsio/c/c_user/node97.html>`_.
  62 + Column names that can be used in the expression can be found
  63 + `here <http://gamma-astro-data-formats.readthedocs.org/en/latest/data_storage/obs_index/index.html>`_
  64 + or by simply browsing the observation index file with a FITS viewer.
  65 + For example
  66 +
  67 + .. code-block:: bash
  68 +
  69 + $ csfindobs expression="ZEN_PNT < 30 && LIVETIME > 1600"
  70 + Path were data are located (NONE uses $VHEFITS environment variable) [data]
  71 + Name of FITS production (Run csiactdata to view your options) [hess_dl3_dr1]
  72 + Right ascension of selection region centre (deg) [83.63] NONE
  73 + Output runlist file [runlist.lis]
  74 +
  75 + will select all observations that have a zenith angle less than 30 deg and a
  76 + livetime larger than 1600 seconds. Note that pointing selection was omitted
  77 + by specifying ``NONE`` for the Right Ascension.
  78 +
  79 +.. note::
  80 + By default, :ref:`csfindobs` only selects data of highest quality
  81 + (i.e. ``QUALITY=0``).
  82 + You may overwrite this default by specifying the hidden parameter
  83 + ``min_qual``. For example, ``min_qual=1`` selects all data with a
  84 + looser quality criteria.
  85 +
  86 +
  87 +Create an observation list
  88 +--------------------------
  89 +
  90 +Now you can convert the runlist into an observation definition XML file.
  91 +You do this conversion with the :ref:`csiactobs` script:
  92 +
  93 +.. code-block:: bash
  94 +
  95 + $ csiactobs
  96 + Path were data are located (NONE uses $VHEFITS environment variable) [NONE] data
  97 + Data storage name [fits-prod-name] hess_dl3_dr1
  98 + Input runlist file [runlist.lis]
  99 + Number of free parameters per background model [1] 2
  100 + Output model definition XML file [bkgmodels.xml]
  101 + Output observation definition XML file [obs.xml]
  102 +
  103 +The :ref:`csiactobs` script will create two output files: the observation
  104 +definition XML file ``obs.xml`` and an output model definition XML file
  105 +``bkgmodels.xml``. To generate ``obs.xml``, :ref:`csiactobs` has used the
  106 +IACT data storage and extracted the relevant file names. ``bgmodels.xml`` is
  107 +a file that is used for background modeling, where each observation will have
  108 +its own independent background model. In the example above, you have set the
  109 +number of free parameters per background model to one, hence the normalisation
  110 +of the background model for each observation will be a free parameter that is
  111 +later adjusted by the maximum likelihood fit.
  112 +
  113 +Here the observation definition file that was created
  114 +
  115 +.. code-block:: xml
  116 +
  117 + <?xml version="1.0" encoding="UTF-8" standalone="no"?>
  118 + <observation_list title="observation list">
  119 + <observation name="Crab Nebula" id="23523" instrument="HESS">
  120 + <parameter name="EventList" file="data/data/hess_dl3_dr1_obs_id_023523.fits.gz[events]" />
  121 + <parameter name="EffectiveArea" file="data/data/hess_dl3_dr1_obs_id_023523.fits.gz[aeff]" />
  122 + <parameter name="PointSpreadFunction" file="data/data/hess_dl3_dr1_obs_id_023523.fits.gz[psf]" />
  123 + <parameter name="EnergyDispersion" file="data/data/hess_dl3_dr1_obs_id_023523.fits.gz[edisp]" />
  124 + <parameter name="Background" file="" />
  125 + </observation>
  126 + <observation name="Crab Nebula" id="23526" instrument="HESS">
  127 + <parameter name="EventList" file="data/data/hess_dl3_dr1_obs_id_023526.fits.gz[events]" />
  128 + <parameter name="EffectiveArea" file="data/data/hess_dl3_dr1_obs_id_023526.fits.gz[aeff]" />
  129 + <parameter name="PointSpreadFunction" file="data/data/hess_dl3_dr1_obs_id_023526.fits.gz[psf]" />
  130 + <parameter name="EnergyDispersion" file="data/data/hess_dl3_dr1_obs_id_023526.fits.gz[edisp]" />
  131 + <parameter name="Background" file="" />
  132 + </observation>
  133 + <observation name="Crab Nebula" id="23559" instrument="HESS">
  134 + <parameter name="EventList" file="data/data/hess_dl3_dr1_obs_id_023559.fits.gz[events]" />
  135 + <parameter name="EffectiveArea" file="data/data/hess_dl3_dr1_obs_id_023559.fits.gz[aeff]" />
  136 + <parameter name="PointSpreadFunction" file="data/data/hess_dl3_dr1_obs_id_023559.fits.gz[psf]" />
  137 + <parameter name="EnergyDispersion" file="data/data/hess_dl3_dr1_obs_id_023559.fits.gz[edisp]" />
  138 + <parameter name="Background" file="" />
  139 + </observation>
  140 + <observation name="Crab Nebula" id="23592" instrument="HESS">
  141 + <parameter name="EventList" file="data/data/hess_dl3_dr1_obs_id_023592.fits.gz[events]" />
  142 + <parameter name="EffectiveArea" file="data/data/hess_dl3_dr1_obs_id_023592.fits.gz[aeff]" />
  143 + <parameter name="PointSpreadFunction" file="data/data/hess_dl3_dr1_obs_id_023592.fits.gz[psf]" />
  144 + <parameter name="EnergyDispersion" file="data/data/hess_dl3_dr1_obs_id_023592.fits.gz[edisp]" />
  145 + <parameter name="Background" file="" />
  146 + </observation>
  147 + </observation_list>
  148 + <observation_list title="observation list" />
  149 +
  150 +and here the model definition file
  151 +
  152 +.. code-block:: xml
  153 +
  154 + <?xml version="1.0" encoding="UTF-8" standalone="no"?>
  155 + <source_library title="source library">
  156 + <source name="bkg_23523" type="CTAAeffBackground" instrument="HESS" id="23523" tscalc="0">
  157 + <spectrum type="PowerLaw">
  158 + <parameter name="Prefactor" value="1" error="0" scale="1e-14" min="0.01" max="100" free="1" />
  159 + <parameter name="Index" value="-2" scale="1" min="-5" max="5" free="1" />
  160 + <parameter name="PivotEnergy" value="1" scale="1000000" free="0" />
  161 + </spectrum>
  162 + </source>
  163 + <source name="bkg_23526" type="CTAAeffBackground" instrument="HESS" id="23526" tscalc="0">
  164 + <spectrum type="PowerLaw">
  165 + <parameter name="Prefactor" value="1" error="0" scale="1e-14" min="0.01" max="100" free="1" />
  166 + <parameter name="Index" value="-2" scale="1" min="-5" max="5" free="1" />
  167 + <parameter name="PivotEnergy" value="1" scale="1000000" free="0" />
  168 + </spectrum>
  169 + </source>
  170 + <source name="bkg_23559" type="CTAAeffBackground" instrument="HESS" id="23559" tscalc="0">
  171 + <spectrum type="PowerLaw">
  172 + <parameter name="Prefactor" value="1" error="0" scale="1e-14" min="0.01" max="100" free="1" />
  173 + <parameter name="Index" value="-2" scale="1" min="-5" max="5" free="1" />
  174 + <parameter name="PivotEnergy" value="1" scale="1000000" free="0" />
  175 + </spectrum>
  176 + </source>
  177 + <source name="bkg_23592" type="CTAAeffBackground" instrument="HESS" id="23592" tscalc="0">
  178 + <spectrum type="PowerLaw">
  179 + <parameter name="Prefactor" value="1" error="0" scale="1e-14" min="0.01" max="100" free="1" />
  180 + <parameter name="Index" value="-2" scale="1" min="-5" max="5" free="1" />
  181 + <parameter name="PivotEnergy" value="1" scale="1000000" free="0" />
  182 + </spectrum>
  183 + </source>
  184 + </source_library>
  185 +
  186 +.. note::
  187 + In case that you have already a model definition XML file that describes a
  188 + model of the celestial source distribution (a.k.a. a sky model), you may
  189 + provide this sky model to :ref:`csiactobs` using the hidden ``inmodel``
  190 + parameter. The output model definition XML file will then contain the sky
  191 + model and the background model, and you can use the model directly in a
  192 + maximum likelihood fit.
  193 +
  194 + .. code-block:: bash
  195 +
  196 + $ csiactobs inmodel=$CTOOLS/share/models/crab.xml
  197 + Path were data are located (NONE uses $VHEFITS environment variable) [data]
  198 + Data storage name [hess_dl3_dr1]
  199 + Input runlist file [runlist.lis]
  200 + Number of free parameters per background model [2]
  201 + Output model definition XML file [bkgmodels.xml] models.xml
  202 + Output observation definition XML file [obs.xml]
  203 +
  204 +
  205 +Select events
  206 +-------------
  207 +
  208 +The final step needed is the selection of events, to define in particular
  209 +the region of interest and energy range for the analysis.
  210 +You do this by typing
  211 +
  212 +.. code-block:: bash
  213 +
  214 + $ ctselect
  215 + Input event list or observation definition XML file [events.fits] obs.xml
  216 + Radius of ROI around pointing or specified RA/DEC (degrees) (0-180) [3.0] 2.0
  217 + Start time (UTC string, JD, MJD or MET in seconds) [NONE]
  218 + Lower energy limit (TeV) [0.1] 0.5
  219 + Upper energy limit (TeV) [100.0] 10.0
  220 + Output event list or observation definition XML file [selected_events.fits] obs_selected.xml
  221 +
  222 +which will select all events within a radius of 2 degrees around the pointing
  223 +direction with energies between 500 GeV and 10 TeV.
  224 +
... ...
doc/source/users/tutorials/howto/iact/simulation.rst deleted
1   -.. _sec_iact_simulation:
2   -
3   -Simulating IACT data
4   -====================
5   -This section describes how to run simulations with current IACT data. In particular, a guide how to simulate data for a given set of IRFs, or a given set of
6   -observations is provided. In addition, tools to compute sensitivity curves or pull distributions are presented.
7   -All tools are based on the event list simulator :ref:`ctobssim`.
8   -
9   -Simulations for a specific observation container
10   -------------------------------------------------
11   -On default, :ref:`ctobssim` builds a new observation container by querying relevant user parameters. In case
12   -an observation XML container is already in place, the hidden parameter ``inobs`` can be specified (:ref:`see here how to create an
13   -observation container <sec_iact_analysis>`). It might be useful to run :ref:`ctselect` beforehand to ensure that the simulation knows about
14   -energy thresholds, field of view, etc. If an unselected observation container is used, the simulation tool will query for global energy
15   -boundaries and a global RoI radius. The following example illustrates the usage of an already selected observation container.
16   -
17   -.. code-block:: bash
18   -
19   - $ ctobssim inobs=selected_obs.xml
20   - Input model XML file [crab_models.xml]
21   - Output event data file or observation definition XML file [sim_obs.xml]
22   -
23   -Note that :ref:`ctobssim` writes a new event list file per observation into the current directory. If the observation container
24   -is large, it might be convenient to create a new folder and specify the ``prefix`` keyword (similar to :ref:`ctselect`):
25   -
26   -.. code-block:: bash
27   -
28   - $ mkdir simulated
29   - $ ctobssim inobs=selected_obs.xml prefix=simulated/sim_events_
30   - ...
31   -
32   -The newly created observation container can be used in the same way as a normal observation container, i.e. it can be analysed with
33   -:ref:`ctlike` etc.
34   -
35   -.. note::
36   -
37   - On default, :ref:`ctobssim` uses the same seed in every execution, i.e. the tool will always return the same simulated event list, regardless
38   - when, where and how it was executed. To steer this the user can provide the hidden integer parameter ``seed``:
39   -
40   - .. code-block:: bash
41   -
42   - $ ctobssim inobs=selected_obs.xml seed=2
43   -
44   - This might be important if simulations are run in parallel.
45   -
46   -Simulating observation time for a given set of IRFs
47   ----------------------------------------------------
48   -There are other use cases to run simulations, for instance when there is no observation container available yet. This might be the case e.g
49   -for an observation proposal: one would want to use a certain set of IRFs to simulate e.g. 50 hours of data to check whether a source
50   -of interest can be detected. For this purpose it is necessary to first find a set of suitable IRFs to be used for
51   -simulations. This can be easily done using :ref:`csfindobs` and using the ``expression`` parameter. The following example extracts
52   -observations with at a certain zenith angle range (20-25 degrees) and azimuth angle range (90-100 degrees):
53   -
54   -.. code-block:: bash
55   -
56   - $ csfindobs expression="AZ_PNT>90.0&&AZ_PNT<100.0&&ZEN_PNT>20.0&&ZEN_PNT<25.0"
57   - Name of FITS production (Run csiactdata to view your options) [fits-prod-name]
58   - Right ascension [83.6331] NONE
59   - Runlist outfile [zen20_az90.lis]
60   -
61   -Of course, further details could be specified, e.g. one might want to check for a certain range of muon efficieny, or any other parameter which impacts IRFs.
62   -The output runlist subsequently has to be converted to an observation XML file using :ref:`csiactobs` (:ref:`see here how to execute this tool <sec_iact_analysis>`).
63   -In the following it is assumed that the observation XML file of the selected representative observations is produced and called ``myobs.xml``.
64   -
65   -Create caldb entry
66   -^^^^^^^^^^^^^^^^^^
67   -Now, one of the observations in this container have to be added to the local calibration database. This database is used
68   -by ctools to extract IRFs for simulations.
69   -
70   -.. code-block:: bash
71   -
72   - $ csobs2caldb
73   - Input observation definition file [myobs.xml]
74   - Response output name (e.g. Zenith50) [zen20_az90]
75   -
76   -Note that the response name has been specified to "zen20_az90". Any name can be chosen which represents the observation which was inserted into the database.
77   -On default, :ref:`csobs2caldb` uses the first observation in the provided XML file. This can be modifyied by specifying the hidden integer parameter index, e.g.:
78   -
79   -.. code-block:: bash
80   -
81   - $ csobs2caldb index=1
82   -
83   -This would use the second entry instead of the first from the observation container to extract the IRFs.
84   -
85   -.. note::
86   -
87   - The ``index`` parameter starts counting from zero.
88   -
89   -Check caldb entries
90   -^^^^^^^^^^^^^^^^^^^
91   -In order to check if the calibration database has successfully been added, the tool :ref:`cscaldb` can be used to inspect the calibration database:
92   -
93   -.. code-block:: bash
94   -
95   - $ cscaldb debug=yes
96   - ...
97   - 2016-02-25T15:57:11: +==============+
98   - 2016-02-25T15:57:11: | Mission: cta |
99   - 2016-02-25T15:57:11: +==============+
100   - 2016-02-25T15:57:11: === Response functions in database "hess" ===
101   - 2016-02-25T15:57:11: zen20_az90
102   - 2016-02-25T15:57:11:
103   - 2016-02-25T15:57:11: === Response functions in database "prod2" ===
104   - 2016-02-25T15:57:11: North_0.5h
105   - 2016-02-25T15:57:11: North_50h
106   - 2016-02-25T15:57:11: North_5h
107   - 2016-02-25T15:57:11: South_0.5h
108   - 2016-02-25T15:57:11: South_50h
109   - 2016-02-25T15:57:11: South_5h
110   - ...
111   -
112   -Note that the observation of the actual instrument ("hess" in this example) was added to the CTA mission. This is mainly for structural and simplicity reasons
113   -and might be improved eventually. Nevertheless, this caldb entry can now be used to simulate any observation duration.
114   -
115   -Simulate
116   -^^^^^^^^
117   -
118   -.. code-block:: bash
119   -
120   - $ ctobssim
121   - RA of pointing (degrees) (0-360) [83.63]
122   - Dec of pointing (degrees) (-90-90) [22.01]
123   - Radius of FOV (degrees) (0-180) [2.5]
124   - Start time (MET in s) [0.0]
125   - End time (MET in s) [18000.0]
126   - Lower energy limit (TeV) [0.5]
127   - Upper energy limit (TeV) [50]
128   - Calibration database [hess]
129   - Instrument response function [zen20_az90]
130   - Input model XML file [input_model.xml]
131   - Output event data file or observation definition XML file [simulated_events.fits]
132   -
133   -Here, a total duration of 5 hours at the crab position was simulated using the newly created caldb entry. Note that the choice of the energy range is up to the user.
134   -One should bear in mind that simulations only make sense in energy ranges where the IRFs are well-defined. The simulation
135   -assumes one observation of defined duration here. Accordingly, only one FITS event list is written as output. The input XML model
136   -must now contain only one background model and (of course) all sky models that are needed. Here is an example of an input model
137   -e.g. for the Crab Nebula:
138   -
139   -.. code-block:: xml
140   -
141   - <?xml version="1.0" standalone="no"?>
142   - <source_library title="source library">
143   - <source name="Crab" type="PointSource">
144   - <spectrum type="PowerLaw">
145   - <parameter name="Prefactor" scale="1e-17" value="3.5" min="1e-07" max="1000.0" free="1"/>
146   - <parameter name="Index" scale="-1" value="2.5" min="0.0" max="+5.0" free="1"/>
147   - <parameter name="PivotEnergy" scale="1e6" value="1.0" min="0.01" max="1000.0" free="0"/>
148   - </spectrum>
149   - <spatialModel type="PointSource">
150   - <parameter name="RA" scale="1.0" value="83.6331" min="-360" max="360" free="0"/>
151   - <parameter name="DEC" scale="1.0" value="22.0145" min="-90" max="90" free="0"/>
152   - </spatialModel>
153   - </source>
154   - <source name="CTABackgroundModel" type="CTAIrfBackground" instrument="CTA">
155   - <spectrum type="PowerLaw">
156   - <parameter name="Prefactor" scale="1.0" value="1.0" min="1e-3" max="1e+3" free="1"/>
157   - <parameter name="Index" scale="1.0" value="0.0" min="-5.0" max="+5.0" free="1"/>
158   - <parameter name="PivotEnergy" scale="1e6" value="1.0" min="0.01" max="1000.0" free="0"/>
159   - </spectrum>
160   - </source>
161   - </source_library>
162   -
163   -Note that since the caldb entry (:ref:`csobs2caldb`) has been produced under the mission "CTA", the event list is now considered a CTA observation (regardless of the input instrument).
164   -Therefore, it is important to have "CTA" specified in the instrument attribute of the background model XML component. Of course the addition of any sky model can be
165   -done without a problem.
166   -
167   -The output of this kind of simulation is a single FITS event list file that can be used for further analysis.
168   -For instance one could run :ref:`ctlike` in the following way:
169   -
170   -.. code-block:: bash
171   -
172   - $ ctlike
173   - Input event list, counts cube or observation definition XML file [simulated_events.fits]
174   - Calibration database [hess]
175   - Instrument response function [zen20_az90]
176   - Input model XML file [input_model.xml]
177   - Output model XML file [simulated_fit_results.xml]
178   -
179   -Note that :ref:`ctlike` automatically queries for the IRF database in case no full observation container is provided. Any other tool taking
180   -unbinned observations as input will behave the same way. Now all means to run simulations with current IACTs are at hand.
181   -The following sections will describe tools for running higher level simulation analysis.
182   -
183   -Create pull distributions
184   --------------------------
185   -Pull distributions inspect the capability of the science tools to find source parameters in the data that were injected into the simulations.
186   -The tool :ref:`cspull` will run several times the sequence of simulating and fitting the data. After each sequence, the pull value of the
187   -fitted parameters is written into an ASCII file. The pull value ``p`` of parameter ``X`` is defined as follows:
188   -
189   -.. math::
190   - p = \frac{X_{\rm fitted} - X_{\rm true}}{\Delta X_{\rm fitted}}
191   -
192   -Where :math:`\Delta X_{\rm fitted}` is the derived error of the fitted parameter :math:`X_{\rm fitted}`. After several trials, the pull value should follow
193   -a normal distribution. Knowing this, the tool may help to distinguish if a source is detectable and can be normally reconstructed.
194   -The tool can either work on an input observation container or on the caldb entry that was created above. The number of trials is queried by the tool, too.
195   -
196   -Example: observation container
197   -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
198   -
199   -.. code-block:: bash
200   -
201   - $ cspull inobs=selected_obs.xml
202   - Number of energy bins (0=unbinned) [0]
203   - Input model XML file [crab_results.xml]
204   - Output pull distribution file [pull.dat]
205   - Number of trials [100]
206   -
207   -Note that the tool asks for the energy binning. It will use an unbinned analysis if zero bins are provided. Otherwise
208   -it will query for binning parameters and will convert the simulated observations into a binned/stacked observation by running
209   -:ref:`ctbin`, :ref:`ctexpcube`, :ref:`ctpsfcube` and :ref:`ctbkgcube` in order before fitting the simulated data with :ref:`ctlike`.
210   -
211   -Example: caldb entry
212   -^^^^^^^^^^^^^^^^^^^^
213   -
214   -.. code-block:: bash
215   -
216   - $ cspull
217   - RA of pointing (deg) (0-360) [83.6331]
218   - Dec of pointing (deg) (-90-90) [22.0145]
219   - Duration (in s) [18000.0]
220   - Lower energy limit (TeV) [0.5]
221   - Upper energy limit (TeV) [50]
222   - Calibration database [hess]
223   - Instrument response function [zen20_az90]
224   - Number of energy bins (0=unbinned) [0]
225   - Input model XML file [input_models.xml]
226   - Output pull distribution file [pull.dat]
227   - Number of trials [100]
228   -
229   -Again, the caldb entry ``zen20_az90`` which was created above is used to simulate 5 hours of data. The data is simulated and analysed 100
230   -times in a row and the output is written into the file ``pull.dat``.
231   -
232   -Visualise pull distributions
233   -----------------------------
234   -The output ASCII file of :ref:`cspull` can be inspected by the user using own scripts. As a starting point, the ``example`` folder
235   -contains a script that can be used to visualise the pull distribution histogram:
236   -
237   -.. code-block:: bash
238   -
239   - $ python $CTOOLS/examples/show_pull_histogram.py pull.dat Pull_Crab_Prefactor 50
240   -
241   -Run the script without arguments to see the usage. In case a wrong parameter name was provided, the tool will print available parameters on the screen.
242   -The last integer parameter defines the binning of the histogram. In addition, the tool shows the standard normal distribution which should
243   -match the pull distribution for a high number of trials if everything went well.
244   -
245   -Create sensitivity curve
246   -------------------------
247   -A very important means to illustrate if a source can be detected with a certain significance is a sensitivity curve.
248   -The tool :ref:`cssens` can be used to compute either an integral, or a differential sensitivity curve for a given observation container.
249   -Analogous to :ref:`cspull`, the tool works either on an unbinned observation container or using a caldb entry.
250   -The script :ref:`cssens` computes the differential or integrated CTA sensitivity using maximum likelihood fitting of a test source.
251   -The differential sensitivity is determined for a number of energy bins, the integral sensitivity is determined for a number
252   -of energy thresholds. The test source is fitted to simulated data using ctlike to determine it’s detection significance as a
253   -function of source flux. The source flux is then varied until the source significance achieves a given level, specified by the
254   -(hidden) significance parameter ``sigma``. To damp variations between individual Monte Carlo simulations, a sliding average is
255   -applied in the significance computation (controlled by hidden ``num_avg parameter``). Note that this procedure might take a while to compute.
256   -To switch from diffferential to integral sensitivity, the hidden parameter ``type="Integral"`` has to be specified. It should be noted that
257   -the results are written to a file ``sensitivity.dat``. To change the output file name, the hidden parameter ``outfile=mysensitivity.dat``
258   -can be provided.
259   -
260   -
261   -Example: observation container
262   -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
263   -
264   -.. code-block:: bash
265   -
266   - $ cssens inobs=selected_obs.xml
267   - Input model XML file [crab_results.xml]
268   - Source name [Crab]
269   - Lower energy limit (TeV) [0.5]
270   - Upper energy limit (TeV) [50]
271   - Number of energy bins for differential sensitivity computation [21]
272   -
273   -Example: caldb entry
274   -^^^^^^^^^^^^^^^^^^^^
275   -
276   -.. code-block:: bash
277   -
278   - $ cssens
279   - Calibration database [hess]
280   - Instrument response function [zen20_az90]
281   - Effective exposure time (s) [18000.0]
282   - Radius of ROI (deg) [2.5]
283   - Input model XML file [input_models.xml]
284   - Source name [Crab]
285   - Lower energy limit (TeV) [0.5]
286   - Upper energy limit (TeV) [50]
287   - Number of energy bins for differential sensitivity computation [21]
288   -
289   -Note that the input model in the latter case only requires to contain one background model (see section above). Using this setup,
290   -the user can specify the observation time for the sensitivity curve. In this example, again, 5 hours are used.
291   -
292   -Visualise sensitivity curves
293   -----------------------------
294   -In order to show a sensitivity curve, there is also a script in the example folder:
295   -
296   -.. code-block:: bash
297   -
298   - $ python $CTOOLS/examples/show_sensitivity.py sensitivity.dat
299   -
300   -This will display a sensitivity curve for the applied settings.
doc/source/users/tutorials/howto/iact/tips.rst deleted
1   -.. _sec_iact_tips:
2   -
3   -Tips, Tricks and FAQs
4   -=====================
5   -This section shows some of the wrinkles that could ease your life when
6   -running IACT analyses.
7   -
8   -
9   -Visualise observations
10   -----------------------
11   -A very common task is to inspect the observation container and visualise several
12   -quantities.
13   -
14   -Create a skymap of counts
15   -^^^^^^^^^^^^^^^^^^^^^^^^^
16   -The tool :ref:`ctskymap` allows to create a single map of counts contained
17   -inside an observation container. In contrast to :ref:`ctbin`, ctskymap takes
18   -into account all events that fall into the skymap energy range and spatial
19   -range. Since :ref:`ctbin` is intended for stacked analysis, it respects energy
20   -thresholds, RoI selections, etc. In contrast, :ref:`ctskymap` may be used for
21   -a simple visualisation of the counts.
22   -
23   -.. code-block:: bash
24   -
25   - $ ctskymap debug=yes
26   - Input event list or observation definition XML file [events.fits] obs.xml
27   - Lower energy limit (TeV) [0.1]
28   - Upper energy limit (TeV) [100.0]
29   - First coordinate of image center in degrees (RA or galactic l) (0-360) [83.63]
30   - Second coordinate of image center in degrees (DEC or galactic b) (-90-90) [22.01]
31   - Projection method (AIT|AZP|CAR|MER|MOL|STG|TAN) [CAR]
32   - Coordinate system (CEL - celestial, GAL - galactic) (CEL|GAL) [CEL]
33   - Image scale (in degrees/pixel) [0.02]
34   - Size of the X axis in pixels [200]
35   - Size of the Y axis in pixels [200]
36   - Output skymap file [skymap.fits]
37   -
38   -Alternatively, you may start from a counts cube and collapse the maps into one
39   -single map via Python:
40   -
41   -.. code-block:: python
42   -
43   - >>> import gammalib
44   - >>> map = gammalib.GSkyMap("myCountCube.fits") # Load counts cube
45   - >>> map.stack_maps() # Stack all cube maps into a single map
46   - >>> map.save("myCountMap.fits") # Save map to file
47   -
48   -Create region file of pointings
49   -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
50   -You may want to know how the pointings of the observation container distribute
51   -across the sky. For this purpose, :ref:`csobsinfo` provides the hidden parameter
52   -``ds9file`` which instructs the script produce a region file that contains the
53   -pointing information:
54   -
55   -.. code-block:: bash
56   -
57   - $ csobsinfo ds9file=pointings.reg debug=yes
58   - Event list, counts cube, or observation definition file [obs.xml]
59   -
60   -Together with the previously produced skymap, you can plot the pointings using
61   -for example ds9:
62   -
63   -.. code-block:: bash
64   -
65   - $ ds9 skymap.fits -regions pointings.reg
66   -
67   -
68   -Plot zenith angle distribution of observations
69   -----------------------------------------------
70   -The script :ref:`csobsinfo` logs information about an observation container
71   -into a log file. You may access the information through dedicated class methods
72   -and plot the information using for example matplotlib:
73   -
74   -.. code-block:: python
75   -
76   - >>> # Run script
77   - >>> import cscripts
78   - >>> info = cscripts.csobsinfo()
79   - >>> info["inobs"] = "selected_obs.xml"
80   - >>> info.run()
81   -
82   - >>> # Plot data
83   - >>> import matplotlib.pyplot as plt
84   - >>> plt.hist(info.zeniths(), bins=10, range=[0,90])
85   - >>> plt.show()
86   -
87   -This example will plot a histogram showing the zenith angle distribution of the
88   -observation container. In addition there are more methods to access information
89   -in Python:
90   -
91   -.. code-block:: python
92   -
93   - >>> info.azimuths() # list of azimuth values
94   - >>> info.offsets() # list of offset values (only computed if hidden parameter offset=yes was specified)
95   - >>> info.ebounds() # gammalib.GEbounds object of energy ranges
96   - >>> info.gti() # gammalib.GGti object containing good time intervals
97   -
98   -There is also an example script to plot information about an observation
99   -container which might be useful for a quick visualisation:
100   -
101   -.. code-block:: bash
102   -
103   - $ python $CTOOLS/examples/show_obs.py selected_obs.xml
104   -
105   -
106   -Visualise models
107   -----------------
108   -Similar to :ref:`csobsinfo`, there is the :ref:`csmodelinfo` script to find out
109   -more about the content of a model XML file. In order to show the position and
110   -sizes of the model on top of a skymap, this tool has the hidden parameter
111   -``ds9file``, too.
112   -
113   -.. code-block:: bash
114   -
115   - $ csmodelinfo ds9file=models.reg debug=yes
116   - Input model XML file [$CTOOLS/share/models/crab.xml]
117   -
118   -There are several options regarding the color, text and other attributes of the
119   -region file. To see a full list, :ref:`visit the reference page <csmodelinfo>`.
120   -Analogous to the pointings of the observation container, the models can be
121   -visualised using for example ds9:
122   -
123   -.. code-block:: bash
124   -
125   - $ ds9 skymap.fits -regions models.reg
126   -
127   -
128   -Manipulating models in python
129   ------------------------------
130   -This example will give you some guidance on how to work with model XML files in
131   -Python and how to manipulate their content:
132   -
133   -.. code-block:: python
134   -
135   - >>> import gammalib
136   -
137   - >>> # Open model file
138   - >>> models = gammalib.GModels("$CTOOLS/share/models/crab.xml")
139   -
140   - >>> # Access a model component
141   - >>> src = models["Crab"]
142   - >>> print(src)
143   -
144   - >>> # Retrieve the spectral or spatial component
145   - >>> print(src.spectral())
146   - >>> print(src.spatial())
147   -
148   - >>> # Print the spectral parameter "Prefactor"
149   - >>> prefactor = src.spectral()["Prefactor"]
150   - >>> print("Prefactor: "+str(prefactor.value())+" +- "+str(prefactor.error()))
151   -
152   - >>> # Set the prefactor value
153   - >>> prefactor.value(3.5e-17)
154   -
155   - >>> # Loop over models and fix all parameters
156   - >>> for model in models:
157   - >>> for par in model: # Loop over model parameters
158   - >>> par.fix() # Fix parameter
159   -
160   - >>> # Release one specific parameter
161   - >>> models["Crab"]["Prefactor"].free()
162   -
163   - >>> # Set the parameter range
164   - >>> models["Crab"]["Prefactor"].min(1e-18)
165   - >>> models["Crab"]["Prefactor"].max(1e-16)
166   -
167   - >>> # ... or in one step:
168   - >>> models["Crab"]["Prefactor"].range(1e-18, 1e-16)
169   -
170   - >>> # Remove model from container
171   - >>> models.remove("Crab")
172   -
173   - >>> # Save model to another XML file
174   - >>> models.save("mymodels.xml")
175   -
176   -
177   -Retrieve likelihood values from ctlike
178   ---------------------------------------
179   -For some purposes, it might be useful to retrieve the fitted likelihood value
180   -and other results of the fit with :ref:`ctlike`.
181   -
182   -.. code-block:: python
183   -
184   - >>> import gammalib
185   - >>> import ctools
186   -
187   - >>> # Create and run ctlike
188   - >>> like = ctools.ctlike()
189   - >>> like["inobs"] = "selected_obs.xml"
190   - >>> like["inmodel"] = "mymodels.xml"
191   - >>> like.run()
192   -
193   - >>> # Get optimizer from ctlike
194   - >>> opt = like.opt()
195   - >>> print(opt)
196   -
197   - >>> # Get likelihood value from optimizer
198   - >>> print(opt.value())
199   -
200   - >>> # Get fit status:
201   - >>> print(opt.status())
202   - >>> # 0: converged
203   - >>> # 1: stalled
204   - >>> # 2: singular curvature matrix encountered
205   - >>> # 3: curvature matrix not positive definite
206   - >>> # 4: errors are inaccurate
207   -
208   -
209   -Speed up analysis
210   ------------------
211   -In some cases it may occur that the analysis takes very long. There are several
212   -reasons that can slow down the analysis quite severly. Not all issues can be
213   -tackled. Nevertheless, here is a list of actions that have proven to speed up
214   -the fit.
215   -
216   -* Consider switching to stacked analysis if the observation time exceeds 50-100
217   - hours.
218   -* Diffuse and extended sources in the RoI are computational expensive. However,
219   - in unbinned analysis they impact the speed much more.
220   -* Usage of energy dispersion could also cause a long fit. Try switching it off
221   - if speed is important. Keep in mind this will have an effect on the results.
222   -* Switch to a machine that supports OpenMP. This allows to parallelise the fit
223   - onto several cores.
224   -
225   -
226   -Compute excess maps
227   --------------------
228   -A very important means to visualise images of sources are excess maps. The script
229   -:ref:`csresmap` can also be used to create such a map. The user, however, has to
230   -manually remove the source(s) of interest from the input model XML file.
231   -
232   -.. code-block:: python
233   -
234   - >>> import gammalib
235   - >>> models = gammalib.GModels("crab_models.xml")
236   - >>> models.remove("Crab")
237   - >>> models.save("models_without_crab.xml")
238   -
239   -Of course, this can also be done by editing the XML file with an editor.
240   -
241   -Subsequently, :ref:`csresmap` can be executed using the ``algorithm=SUB``
242   -parameter:
243   -
244   -.. code-block:: bash
245   -
246   - $ csresmap
247   - Input event list, counts cube, or observation definition XML file [selected_obs.xml]
248   - Input model XML file [models_without_crab.xml]
249   - First coordinate of image center in degrees (RA or galactic l) (0-360) [83.63]
250   - Second coordinate of image center in degrees (DEC or galactic b) (-90-90) [22.01]
251   - Coordinate System (CEL|GAL) [CEL]
252   - Projection method (AIT|AZP|CAR|MER|MOL|STG|TAN) [CAR]
253   - Size of the X axis in pixels [100]
254   - Size of the Y axis in pixels [100]
255   - Pixel size (deg/pixel) [0.02]
256   - Residual map computation algorithm (SUB|SUBDIV|SUBDIVSQRT) [SUB]
257   - Output residual map file [excessmap.fits]
258   -
259   -
260   -Compute flux maps
261   ------------------
262   -The creation of a flux map is much more expensive in terms of computing than an
263   -excess map. In the section about :ref:`high level analysis tools <sec_iact_high_level>`,
264   -the usage of :ref:`cttsmap` was demonstrated. This tool can also be used to
265   -generate flux maps. For this, the user has to incorporate a test source in the
266   -XML model file. This source should consist of a spectral component using a
267   -``PowerLawPhotonFlux`` model:
268   -
269   -.. code-block:: xml
270   -
271   - <source name="TestSource" type="PointSource">
272   - <spectrum type="PowerLawPhotonFlux">
273   - <parameter scale="1e-07" name="PhotonFlux" min="1e-07" max="1000.0" value="1.0" free="1"/>
274   - <parameter scale="1.0" name="Index" min="-5.0" max="+5.0" value="-2.0" free="1"/>
275   - <parameter scale="1.0" name="LowerLimit" min="10.0" max="1000000.0" value="100.0" free="0"/>
276   - <parameter scale="1.0" name="UpperLimit" min="10.0" max="1000000.0" value="500000.0" free="0"/>
277   - </spectrum>
278   - <spatialModel type="PointSource">
279   - <parameter free="0" max="360" min="-360" name="RA" scale="1" value="83.6331" />
280   - <parameter free="0" max="90" min="-90" name="DEC" scale="1" value="22.0145" />
281   - </spatialModel>
282   - </source>
283   -
284   -The further content of the rest of the XML model file depends on the user
285   -requirements:
286   -
287   -* For an absolute flux map, all other sky models should be removed
288   -* For a residual flux map, all other sky models should be kept
289   -
290   -The tool :ref:`cttsmap` will create one skymap per free spectral parameter in
291   -the model. In the result FITS file, there will be an extension called ``Integral``
292   -that contains the flux map. For the above XML example, the source name
293   -``TestSource`` should be specified to :ref:`cttsmap`.
294   -
295   -
296   -Speed up TS map computation
297   ----------------------------
298   -Since the computation of a TS map can be extremely time consuming, the option of
299   -splitting the computation into several jobs is supported. This might be of
300   -particular interest if the user has access to a batch farm.
301   -
302   -Split TS map computation
303   -^^^^^^^^^^^^^^^^^^^^^^^^
304   -For the purpose of job splitting, the hidden parameter ``binmin`` and ``binmax``
305   -were included in the tool. These are integer parameter that specify which bins
306   -should be computed. For instance, if the map should consist of 30x30(=900) pixels,
307   -the user could for example execute
308   -
309   -.. code-block:: bash
310   -
311   - $ cttsmap binmin=0 binmax=299 outmap=tsmap_0_299.fits
312   - ...
313   - $ cttsmap binmin=300 binmax=599 outmap=tsmap_300_599.fits
314   - ...
315   - $ cttsmap binmin=600 binmax=899 outmap=tsmap_600_899.fits
316   - ...
317   -
318   -Each command could run on a different machine.
319   -
320   -.. note::
321   -
322   - The output file name of individual jobs should be different. Otherwise files
323   - could overwrite each other. The naming of the individual slices is up to the
324   - user.
325   -
326   -The script :ref:`cstsmapsplit` will take care of this bookkeeping. It creates an
327   -ASCII file containing all the commands according to the user input. The following
328   -example will create split the computation of a TS map into 2000 separate task.
329   -Each task will only compute 5 bins on its own. This is very useful if the
330   -observation container is large or the fit simply takes a long time.
331   -
332   -.. code-block:: bash
333   -
334   - $ cstsmapsplit
335   - Input event list, counts cube or observation definition XML file [selected_obs.xml]
336   - Input model definition XML file [$CTOOLS/share/models/crab.xml]
337   - First coordinate of image center in degrees (RA or galactic l) (0-360) [83.63]
338   - Second coordinate of image center in degrees (DEC or galactic b) (-90-90) [22.01]
339   - Projection method (AIT|AZP|CAR|MER|MOL|STG|TAN) [CAR]
340   - Coordinate system (CEL - celestial, GAL - galactic) (CEL|GAL) [CEL]
341   - Image scale (in degrees/pixel) [0.02]
342   - Size of the X axis in pixels [100]
343   - Size of the Y axis in pixels [100]
344   - Test source name [Crab]
345   - Output Test Statistic map file [tsmap.fits]
346   - Number of TS map bins per task [5]
347   - Compute null hypothesis first? [yes]
348   - ASCII file containing all commands [commands.dat]
349   -
350   -.. note::
351   -
352   - One can decide if we want to compute the null hypothesis first. This way, a
353   - parameter optimisation will be performed without the test source. The obtained
354   - likelihood value will then be passed to each individual task of :ref:`cttsmap`.
355   - If ``compute_null=no``, each task has to compute the null hypothesis itself.
356   -
357   -Merge splitted TS maps
358   -^^^^^^^^^^^^^^^^^^^^^^
359   -The script :ref:`cstsmapmerge` is intended to take care of merging the files
360   -that were produced while splitting the TS map computation. There are several
361   -options to pass the files to be merged as arguments:
362   -
363   -* a space-separated list of file names (e.g. tsmap1.fits tsmap2.fits)
364   -* a comma-separated list of file names (e.g. tsmap1.fits,tsmap2.fits)
365   -* a wildcard string (e.g. tsmap*.xml)
366   -* an ASCII file containing the file names, one file per line (e.g. @mymaps.txt)
367   -
368   -In this example the ASCII file method is presented:
369   -
370   -.. code-block:: bash
371   -
372   - # Put slice files into an ascii file
373   - $ ls tsmap_*.fits > tsmapfiles.txt
374   -
375   - # Run cstsmapmerge
376   - $ cstsmapmerge
377   - Input TS map FITS files [@tsmapfiles.txt]
378   - Output TS map FITS file [mytsmap.fits]
379   -
380   -
381   -Creating a python analysis pipeline
382   ------------------------------------
383   -It is easily possible to build an own analysis workflow with a simple python
384   -script. The following source code example shows a python script running from
385   -gathering observations until fitting spectral points without storing intermediate
386   -data products on disk. It assumes that the environment variable ``$VHEFITS`` is
387   -set to the path where IACT FITS data is located.
388   -
389   -.. code-block:: python
390   -
391   - >>> import gammalib
392   - >>> import ctools
393   - >>> import cscripts
394   -
395   - >>> # Set debug flag
396   - >>> debug = True
397   -
398   - >>> # Set flag to use energy dispersion
399   - >>> edisp = False
400   -
401   - >>> # Set inmodel file name
402   - >>> inmodel = "$GAMMALIB/test/data/model_point_plaw.xml"
403   -
404   - >>> # Expand environment variable
405   - >>> inmodel = gammalib.expand_env(inmodel)
406   -
407   - >>> # Extract coordinates and model properties
408   - >>> models = gammalib.GModels(inmodel)
409   - >>> srcname = models[0].name()
410   - >>> ra = models[0]["RA"].value()
411   - >>> dec = models[0]["DEC"].value()
412   -
413   - >>> # Find FITS production name
414   - >>> iactdata = cscripts.csiactdata()
415   - >>> iactdata["debug"] = debug
416   - >>> iactdata.run()
417   -
418   - >>> # Use first available production
419   - >>> prodname = iactdata.names()[0]
420   -
421   - >>> # Run csfindobs
422   - >>> findobs = cscripts.csfindobs()
423   - >>> findobs["ra"] = ra
424   - >>> findobs["dec"] = dec
425   - >>> findobs["rad"] = 2.5
426   - >>> findobs["prodname"] = prodname
427   - >>> findobs["debug"] = debug
428   - >>> findobs["outfile"] = "NONE"
429   - >>> findobs.run()
430   -
431   - >>> # Retrieve obervation IDs (runlist)
432   - >>> obs_ids = findobs.obs_ids()
433   -
434   - >>> # Build observation container
435   - >>> iactobs = cscripts.csiactobs()
436   - >>> iactobs["prodname"] = prodname
437   - >>> iactobs["inmodel"] = inmodel
438   - >>> iactobs["bkgpars"] = 1
439   - >>> iactobs["outobs"] = "NONE"
440   - >>> iactobs["outmodel"] = "NONE"
441   - >>> iactobs["debug"] = debug
442   - >>> iactobs.runlist(obs_ids)
443   - >>> iactobs.run()
444   -
445   - >>> # Retrieve observation container and energy boundaries
446   - >>> obs = iactobs.obs()
447   - >>> ebounds = iactobs.ebounds()
448   -
449   - >>> # Run ctselect
450   - >>> select = ctools.ctselect(obs)
451   - >>> select["usepnt"] = True
452   - >>> select["rad"] = 2.5
453   - >>> select["usethres"] = "DEFAULT"
454   - >>> select["tmin"] = 0.0
455   - >>> select["tmax"] = 0.0
456   - >>> select["emin"] = 0.1
457   - >>> select["emax"] = 100.0
458   - >>> select["debug"] = debug
459   - >>> select.run()
460   -
461   - >>> # Pass selected observations to ctlike
462   - >>> like = ctools.ctlike(select.obs())
463   - >>> like["debug"] = debug
464   - >>> like["edisp"] = edisp
465   - >>> like.run()
466   -
467   - >>> # Compute a spectrum and save
468   - >>> spec = cscripts.csspec(like.obs())
469   - >>> spec["srcname"] = srcname
470   - >>> spec["emin"] = ebounds.emin().TeV()
471   - >>> spec["emax"] = ebounds.emax().TeV()
472   - >>> spec["enumbins"] = 10
473   - >>> spec["edisp"] = edisp
474   - >>> spec["ebinalg"] = "LOG"
475   - >>> spec["debug"] = debug
476   - >>> spec["outfile"] = "spectrum.fits"
477   - >>> spec.execute()
478   -
479   - >>> # Remove model for excess map computation
480   - >>> like.obs().models().remove(srcname)
481   -
482   - >>> # Compute an excess map
483   - >>> resmap = cscripts.csresmap(like.obs())
484   - >>> resmap["xref"] = ra
485   - >>> resmap["yref"] = dec
486   - >>> resmap["proj"] = "CAR"
487   - >>> resmap["coordsys"] = "CEL"
488   - >>> resmap["emin"] = ebounds.emin().TeV()
489   - >>> resmap["emax"] = ebounds.emax().TeV()
490   - >>> resmap["nxpix"] = 100
491   - >>> resmap["nypix"] = 100
492   - >>> resmap["binsz"] = 0.02
493   - >>> resmap["outmap"] = "excessmap.fits"
494   - >>> resmap.execute()
495   -
496   -The results of this analysis workflow can be inspected using an example Python
497   -script and for example ds9:
498   -
499   -.. code-block:: bash
500   -
501   - $ python $CTOOLS/examples/show_spectrum.py spectrum.fits
502   - $ ds9 excessmap.fits
503   -
doc/source/users/tutorials/howto/index.rst
... ... @@ -46,5 +46,5 @@ Analysis of data from other instruments
46 46  
47 47 How to analyse Fermi-LAT data with ctools? <fermi/index>
48 48 How to analyse COMPTEL data with ctools? <comptel/index>
49   - How to analyse H.E.S.S. data with ctools? <iact/index>
  49 + How to analyse data from existing IACTs (H.E.S.S., VERITAS or MAGIC) with ctools? <iact/index>
50 50 howto_combine_instruments
... ...