Authors: Hans Wenzel, Soon Yung Jun, Krzysztof Genser (Fermilab)

Note this is a draft (not yet linked anywhere). Once Geant4.11 is released we will replace the links to github.


electron shower in liquid Argon

CaTS logoCaTS (Calorimeter and Tracker Simulation) is a flexible and extend-able Geant4 based simulation framework. CaTS is also used to demonstrate the  G4Opticks hybrid workflow calling Opticks from within Geant4 for the creation and propagation of optical photons on GPU's. Opticks is an open source project. It translates Geant4 optical physics to NVIDIA® CUDA, OptiXTM to run on GPU's and the Geant4 geometry into a geometry optimized for ray tracing on GPU's. Opticks only handles optical photons while Geant4 (running on the CPU/host) handles all other particles.  

G4Opticks is part of Opticks and provides the interface between Geant4 and Opticks. In many ways the generation and tracing of optical photons is the ideal application to be ported to GPU’s:

  •  there is only one particle involved (optical photon), but depending on the use case there can be many of them  (10s of millions in case or a liquid Argon TPC for a typical neutrino events).  This allows for massive parallelism (low latency, no big fluctuations in computing time.).
  •  Only a few simple physics processes need to be implemented on the GPU. The processes are :
    • G4Cerenkov (generate),
    • G4Scintillation (generate),
    • G4OpAbsorption,
    • G4OpRayleigh,
    • G4OpBoundaryProcess (only a few surface types),
    • G4OpWLS (not yet implemented),
    • G4OpMieHG (not needed),
    • Reemission (Opticks specific, no equivalent in Geant4).
  • These processes don’t need a lot of input data (the so called Gensteps for the Cerenkov (C) and Scintillation (S) processes) and don't produce alot od output. Only few photons reach the Photodetectors and produce a Hit. So the amount of data data to be transferred  between host(CPU) and device (GPU)  (Gensteps --> Device,  PhotonHits --> Host) is small.
  • Optical ray tracing is a well established field so we  benefit from available efficient algorithms (e.g., OptiXTM).
  • use NVIDIA® hardware (some with RTX: raytracing hardware acceleration) and software (CUDA, OptiX).
  • G4Opticks is a runtime/build time option. One can switch between Opticks and the use of the Geant4 native optical processes for one to one comparison.

G4Opticks hybrid workflow

CaTS was developed to support R&D projects where it is necessary to easily change the geometry, physics configurations etc. at run-time without recompiling.  CaTS allows to simulate Detector setups ranging from a single Detector (e.g. single crystal) to complex setups like test beam experiments consisting of many different detector elements.

The components of CaTS are:

  • Detector Description: described in gdml input file (e.g. crystalcal.gdml) (Geometry, Materials, optical properties, sensitive detector). We use extensions to the gdml schema to attach sensitive detectors to logical Volumes, assign step limits toe specific Volumes and to apply visualization attributes (e.g. color, transparency..) to volumes.
  • (needs work) plug and play. making use of predefined components. automated just to attach a sensitive detector to a logical volume  in the gdml file hits are added to the root based output and properly names.
  • Input modules:
    • GPS
    • Particle Gun
  • Physics: CaTS uses G4PhysListFactoryAlt to define and configure physics. One can choose between all Geant Reference Physics Lists and can extend it  by adding various physics constructors like the optical physics processes (Cerenkov, Rayleigh, Scintillation etc.), step limiter, neutron limiters etc. The physics configuration can be specified via a command line option.
    ./CaTS -g simpleLArTPC.gdml -pl 'FTFP_BERT+OPTICAL+STEPLIMIT' -m time.mac -t 4
    • where:
      • -g specifies the gdml file to use (mandatory argument)
      • -pl specifies the physics configuration in this case we select the FTFP_BERT reference ohysics list and extend it with the OPTICAL and STEPLIMIT and physics constructors. The default is "FTFP_BERT+OPTICAL+STEPLIMIT"
      • -m specifies the Geant4 macro to use. If no macro is provided interactive processing is assumed.
      • -t specifies the number of threads. If the number of threads is not given it's set to the number of CPU cores available.
  • Sensitive Detectors: (+ corresponding Hit classes)
    • TrackerSD(Hit)  registers the step points of charged particles.
    • CalorimeterSD(Hit) registering energy deposit
    • DRCalorimeterSD(Hit) besides registering energy deposit counts produced Cerenkov photons
    • MsCSD(Hit) used to study multiple scattering on a thin layer
    • PhotonSD(Hit) sensitive detector that registers optical photons.
    • lArTPCSD(Hit) sensitive detector that registers ionization and collects Gensteps (Scintillation and Cerenkov) to be processed by Opticks.
    • RadiatorSD is special since no Hits are created. This sensitive detector just collects the  Gensteps  (Scintillation and Cerenkov) to be processed by Opticks.
  • CaTS uses Root IO to provide persistency for Hits.
  • Using of G4Opticks doesn't require changes to Geant4! CaTS only makes use of provided Geant4 user interfaces: UserActions, Sensitive Detectors...
  • When G4Opticks is used for the generation and tracing of optical photons we still use the Geant4 Cerenkov and Scintillation (C/S) processes to calculate the number of optical photons to be generated at a given step and to provide all necessary quantities to generate the photons on the GPU. The information collected is the so called GenStep which is different for Cerenkov and Scintillation (C/S). The harvesting is done in sensitive Detectors(SD) (RadiatorSD/lArTPCSD). The PhotonHits are collected in PhotonSD (see the CaTS workflow below). Note even when G4Opticks is used the Geant4 Cerenkov and Scintillation processes are still required to calculate the number of photons to be generated at each given step. The number of photons generated by Opticks is constrained to the number calculated by the equivalent Geant4 Cerenkov/Scintillation process.


CaTS workflow


Opticks resources:

EPJ Web of Conferences 214, 02027 (2019), Simon Blyth: “Opticks : GPU Optical Photon Simulation for Particle Physics using NVIDIA® OptiXTM “

Detector geometry in Opticks: Documentation:

Code repositories: : main development repository     : used for snapshots and tagged releases.

CaTS resources:


Main git repository: used for development.
Instructions how to build Opticks and how to install all necessary software:

CaTS examples (examples consist of: gdml file, Geant4 macro and an application that makes histograms from the hits collections):


Papers based on CaTS-simulation:

  • Space-Time Development of Electromagnetic and Hadronic Showers and Perspectives for Novel Calorimetric Techniques. Andrea Benaglia(CERN), Etiennette Auffray, Paul Lecoq, Hans Wenzel, Adam Para(Fermilab) (Apr, 2016), Published in: IEEE Trans.Nucl.Sci. 63 (2016) 2, 574-579 , Contribution to: SCINT 2015, 574-579
  • Physics of hadron shower development and the implications for calorimetric resolution. Hans Wenzel(Fermilab), Adam Para(Fermilab), Andrea Delgado(Texas A-M) (2013), Contribution to: CHEF 2013, 265-271