Simon Bonnefoy / ctools

New Features and Important Changes in ctools 1.6.0

3 November 2018


Introduction
------------
This document briefly summarizes the changes that have been made in this
ctools release with respect to the precedent release.


Configuration
-------------
A Code Generator was added to the development directory that can be started
using dev/codegen.py and that requires GammaLib to be installed. The code
generator allows adding ctools and cscripts, including unit tests and code
documentation, that are fully functional. The developer then only needs to
fill the skeleton with the relevant code (#2709).

Keep existing library path in installation script on Mac OS X (#2420).


Python interface
----------------
ctools and cscripts can now be pickled from Python (#1938).


Support model
-------------
None


Base classes
------------
The ctool base class now loads the application parameters from the file provided
in the "syspfiles" folder and updates parameter values using the file found in
the user's "pfiles" folder, so that any change to the parameter file structure
is handled transparently and no longer leads to an exception (#2513).

The ctool::time_reference constant was removed since it is not required
anymore. The time reference is now transparently handled by the tools, and
if there is a choice, a "mjdref" parameter was added to the tools (#2054).

The ctool::create_cta_obs() method now queries the optional "mjdref" parameter
so that a MJD reference time can be specified by the tool. This functionality
is now used by ctobssim (#2054).

The ctools::get_gti() method now takes GTimeReference as argument to enable
tools to retrieve Mission Elapsed Time (MET) in the time reference of the
observation (#2054).

The ctools::is_valid_filename() method was removed since the validity check
of the filename is now done by the GApplicationPar class.


ctbin - CTA event binning
-------------------------
ctbin now accouts for varying energy thresholds when computing the weight
array (#2685).

Add hidden "nthreads" parameter to control parallel computations (#2421).

Copy the time reference from the first input observation (#2054).


ctbkgcube - CTA background cube generation
------------------------------------------
Add ctbkgcube::cntcube() method so that a counts cube be specified as
instance before running the tool. This avoids storing the counts cube on
disk if the counts cube definition should be used to generate a background
cube. In addition, the background cube values are now devided by the
weight values of the counts cube since later, during model evaluation,
the model is multiplied by the weight values. This makes sure that the
normalisation is correct (#2685).


ctbutterfly - Generate butterfly diagrams
-----------------------------------------
Add hidden "like_accuracy" parameter to allow control of the accuracy of
the maximum log-likelikelihood value (#2662).


ctcubemask - CTA cube filtering
-------------------------------
None


ctedispcube - CTA energy dispersion cube generation
---------------------------------------------------
None


cterror - Likelihood profile error computation
----------------------------------------------
Add hidden "like_accuracy" parameter to allow control of the accuracy of
the maximum log-likelikelihood value (#2662).

Add hidden "nthreads" parameter to control parallel computations (#2421).


ctexpcube - CTA exposure cube generation
----------------------------------------
None


ctlike - Maximum likelihood fitting
-----------------------------------
Add hidden "like_accuracy" parameter to allow control of the accuracy of
the maximum log-likelikelihood value (#2662).

Add hidden "nthreads" parameter to control parallel computations (#2421).


ctmapcube - CTA map cube generation
-----------------------------------
None


ctmodel - CTA counts cube modelling
-----------------------------------
The tools now computes DETX and DETY for every event bin if they are not
available so that model that rely on the existence of these parameters can
be handled (#2694).

Add hidden "nthreads" parameter to control parallel computations (#2421).


ctobssim - CTA observation simulation
-------------------------------------
Add hidden "nthreads" parameter to control parallel computations (#2421).

Add "mjdref" parameter to allow specification of the MJD reference time
for the simulation (#2054).


ctphase - Event phase computation
---------------------------------
None


ctprob - Event probability computation
--------------------------------------
None


ctpsfcube - CTA point spread function cube generation
-----------------------------------------------------
None


ctselect - CTA event selection
------------------------------
Do no longer set missing energy and RoI boundaries in ctool::get_observations
and ctool::setup_observations since this led to a problem in ctselect. For
H.E.S.S. data, where the boundaries are not set, the boundaries were preset
with the User parameters, which made event selection with the usepnt=no option
impossible. It is not clear why the setting of missing boundaries was introduced
but it looks like a dangerous thing to do. The setting has therefore been
removed (#2686).

Avoid an error in case that the RoI radius is identical to the existing RoI
radius (#2682).

Interpret MET specified by "tmin" and "tmax" parameters in the time reference
of the input observations (#2054). 


ctskymap - CTA sky mapping
--------------------------
Add the "inmap", "iterations" and "threshold" parameters. The hidden
"inmap" parameter allows specifying a sky map file containing a "COUNTS"
and "ACCEPTANCE" extension which avoids looping over event lists in an
observation container. The "iterations" parameter allows the iterative
determination of an exclusion map from the resulting significance map for
the RING background method. The exclusion map will flag all pixels with
significances above the "threshold" value so that they will be excluded
in the subsequent RING background determination. Since this will impact
the pixel significances, the procedure needs to be repeated a couple of
times to stabilise (#2311).

Accelerate the IRF and RING background methods. For the RING background
method the computation is either performed using a FFT or using a direct
computation. The FFT computation is faster, but assumed Euclidian
distances for the sky map grid, the direct computation is slower, but
accurately computes the distances between sky map pixels (#2309).


cttsmap - Generation of Test Statistic map
------------------------------------------
Add hidden "like_accuracy" parameter to allow control of the accuracy of
the maximum log-likelikelihood value (#2662).


ctulimit - Compute upper limit
------------------------------
Add hidden "like_accuracy" parameter to allow control of the accuracy of
the maximum log-likelikelihood value (#2662).

Add "parname" parameter to allow upper limit computations also for a node
function spectral model. In addition, it is now checked that the parameter
values actually bracket the upper limit (#2233).


support - Support classes
-------------------------
None


csbkgmodel - Generate background model for 3D analysis
------------------------------------------------------
Introduced the script that generates a background for a 3D analysis. This
script is particularly useful to generate a background model for the analysis
of the H.E.S.S. public data release (#2694).


cscaldb - Inspect calibration database
--------------------------------------
None


csebins - Generate energy boundaries
------------------------------------
None


csfindobs - Find IACT observations
----------------------------------
None


csiactcopy - Copy IACT observations
-----------------------------------
None


csiactdata - Inspect IACT data store
------------------------------------
None


csiactobs - Inspect IACT observation definition XML file
--------------------------------------------------------
None


cslightcrv - Generate light curve
---------------------------------
Add hidden "use_irf_bkg" parameter for forwarding to csphagen.

Add hidden "nthreads" parameter and implement parallelisation of light
curve computation (#2421).

Add hidden "mjdref" parameter for the specification of the MET reference
(#2054).


csmodelinfo - Return model information
--------------------------------------
None


csmodelmerge - Merge models
---------------------------
None


csmodelselect - Select models
-----------------------------
None


csmodelsois - Puts selected sources in diffuse model cube
---------------------------------------------------------
None


csobs2caldb - Generate calibration database entry for an IACT observation
-------------------------------------------------------------------------
Change background template table name from "BGD" to "BKG" (#2343).


csobsdef - Generate observation definition XML file
---------------------------------------------------
Add hidden "mjdref" parameter for the specification of the MET reference
(#2054).


csobsinfo - Return observation definition information
-----------------------------------------------------
Add ras() and decs() methods (#2313)


csobsselect - Select observations
---------------------------------
Use time reference of input observations if time selection is given in MET
(#2054).


csphagen - Generate On/Off observations
---------------------------------------
Add hidden "use_irf_bkg" parameter. By default the parameter is set to "yes",
which means that the background acceptance provided in the IRF is used for
the computation of the alpha parameter and the background rate, stored in the
"BACKSCAL" column of the On spectrum and the "BACKRESP" column of the Off
spectrum, respectively. If the parameter is set to "no" it is assumed that the
background rate in the On and Off regions are the same, and the IRF background
template is not used. In that case, the "BACKSCAL" column gives the ratio of
the solid angles between On and Off regions, and the "BACKRESP" column is set
to zero. The resulting data can only be fitted using the "wstat" statistic
(#2659).

Add hidden "nthreads" parameter and implement parallelisation of computations
(#2421).

The true number of energy bins is now computed using
   int(bins_per_decade * num_of_decades + 0.5)
making sure in addition that the result is at least 1. This provides better
control over the number of energy bins (#2490).

The script now produces data that are compliant with Xspec. An Xspec howto
tutorial was added to the user documentation (#2404).


csphasecrv - Generate phase curve
---------------------------------
Add hidden "nthreads" parameter and implement parallelisation of computations
(#2421).


cspull - Generate pull distributions
------------------------------------
Add hidden "nthreads" parameter and implement parallelisation of computations
(#2421).

Add hidden "mjdref" parameter for the specification of the MET reference
(#2054).


csresmap - Generate residual map
--------------------------------
None


csresspec - Generate residual spectrum
--------------------------------------
An exception occured when using a pre-computed model cube via the modcube
parameter due to a bug in the code. This bug was fixed (#2448).


csroot2caldb - Generates calibration database from ROOT file
------------------------------------------------------------
Change energy dispersion energy axis name from "ETRUE" to "ENERG" (#2670).

Change background template table name from "BGD" to "BKG" (#2343).

The ROOT Python module is now imported in the run method and not globally so
that no conflit arises with the "--help" option (#2652).


cssens - Computes sensitivity curve
-----------------------------------
Add hidden "nthreads" parameter and implement parallelisation of computations
(#2421).


csspec - Generates Spectral Energy Distribution
-----------------------------------------------
The script no longer skips energy bins with zero counts (#2671).

The full true energy range of the RMF is now used in an On/Off analysis to
prevent cutting off the tails of the energy dispersion, which happened before
when the script was used for the analysis of H.E.S.S. data (#2656).

Add hidden "nthreads" parameter and implement parallelisation of computations
(#2421).

csspec now properly computes the Test Statistic for an On/Off analysis using
the "wstat" statistic in the case that the input model definition XML file
did not contain a background model (#2312).

csspec now computes upper limits also for the NODES method by fixing all node
intensities and varying each node until the log-likelihood increases by a
certain amount (#2233).


cssrcdetect - Detects sources in sky map
----------------------------------------
The script was made more robust by optionally convolving the input sky map
with a smoothing kernel with a specified correlation radius (options are a
"DISK" or a "GAUSSIAN" kernel, default is now "DISK"). In addition, the mean
and standard deviation of the pixel values are now computed locally over a
given radius, controlled through the "avgrad" parameter. This accomodates for
counts variations in the field of view (#2498).


cstsdist - Computes Test Statistic distribution
-----------------------------------------------
Add hidden "nthreads" parameter and implement parallelisation of computations
(#2421).


cstsmapmerge - Merge Test Statistic maps
----------------------------------------
None


cstsmapsplit - Prepare split Test Statistic computation
-------------------------------------------------------
Add hidden "like_accuracy" parameter to allow control of the accuracy of
the maximum log-likelikelihood value (#2662).


csviscube - Generate visibility cube
------------------------------------
Add hidden "mjdref" parameter for the specification of the MET reference
(#2054).


csworkflow - Execute workflow
-----------------------------
None


calutils - Calibration utilities
--------------------------------
None


ioutils - Input/Output utilities
--------------------------------
None


modutils - Model utilities
--------------------------
None


mputils - Multiprocessing utilities
-----------------------------------
Introduce module (#2421).


obsutils - Observation utilities
--------------------------------
Add "mjdref" parameter to set_obs() function (#2054).


Examples
--------
Handle zero model values for SUBVID method in show_response.py script (#2456).

Added show_rmf.py script.


caldb
-----
None


models
------
None