Multi-job execution#

GATE can split one simulation into several child jobs, run those jobs, and merge their output again. This is useful in two main situations:

local execution on a workstation, where several jobs can run in parallel on the same machine
server execution, where the split jobs are submitted to a scheduler such as Slurm or HTCondor

This section focuses on the current user-facing workflow in GATE 10.

Local multi-job execution#

For local execution, the recommended entry point is sim.run(...) with the number_of_jobs argument. GATE then:

resolves and splits the simulation into child jobs
runs the jobs through a local process pool
optionally merges the result back into the live simulation object

Run split, execution, and merge in one go#

If the goal is to run locally and immediately continue with postprocessing in the same script, use merge_after_run=True together with wait_for_result=True.

import opengate as gate

sim = gate.Simulation()
sim.output_dir = "output"

# ... configure geometry, physics, sources, and actors ...

controller = sim.run(
    number_of_jobs=4,
    wait_for_result=True,
    merge_after_run=True,
)

# At this point, the merged output is already available in sim.output_dir
# and the live simulation object can be used for postprocessing.

In the example, controller is a SplitRunMergeController object that can be inspected. It is always returned by sim.run(...) when the simulation is run in multiple jobs.

Local pooled execution always uses one spawned worker process per child job.

Note

number_of_jobs=1 does not create a split campaign. It maps to an ordinary simulation run in a new process, i.e. it is equivalent to sim.run(start_new_process=True)

Split and run now, merge later#

Sometimes it is preferable to separate execution and merging. A typical example is a long local run that should finish first, while merging and postprocessing happen only afterwards, possibly in another script.

The first script can launch the split jobs and stop after execution:

import opengate as gate

sim = gate.Simulation()
sim.output_dir = "output"

# ... configure geometry, physics, sources, and actors ...

controller = sim.run(
    number_of_jobs=4,
    wait_for_result=False,
    merge_after_run=False,
    campaign_dir="campaign",
)

print(controller.stage)         # typically "submitted"
print(controller.campaign_dir) # folder containing simulation.json and job0001, job0002, ...

Later, another script can re-use the campaign folder and merge the finished jobs:

import opengate as gate

merge_manager = gate.jobs_merge("campaign")
sim = merge_manager.master_simulation
gate.jobs_clean("campaign")  # this removes the job folders and other metadata about the split run

merge_manager.print_merge_summary()
# ... post-process sim, actors, and merged output ...

In this scenario, merging recreates a simulation from the files in the campaign folder and writes merged output to the configured output directory of that simulation.

Low-level split, run, and merge#

The lower-level functions can also be called explicitly. jobs_split(...) returns a JobsSplitManager object; use jobs_split_manager.campaign_dir when passing the campaign folder to the next stage.

import opengate as gate

sim = gate.Simulation()
sim.output_dir = "output"

# ... configure geometry, physics, sources, and actors ...

jobs_split_manager = gate.jobs_split(
    simulation=sim,
    campaign_dir="campaign",
    number_of_jobs=4,
    policy="split_in_time_total",
)

run_summary = gate.jobs_run(
    jobs_split_manager.campaign_dir,
    backend="local_pool",
)

merge_manager = gate.jobs_merge(jobs_split_manager.campaign_dir)
merged_sim = merge_manager.master_simulation

Split policies#

GATE currently provides two time-based split policies.

split_in_time_total: This is the default policy. It splits the total active simulation time into several consecutive jobs of similar duration. A child job may therefore bridge across two original run timing intervals. This is usually the best starting point for local acceleration and for general split campaigns.
split_in_time_per_run: This policy splits each original run timing interval separately. The number of jobs must then be a multiple of the number of original runs. Use this when the split should remain more directly aligned with the original run structure.

Choose the policy explicitly with the split_policy argument:

import opengate as gate

sim = gate.Simulation()

# ... configure geometry, sources, actors, and run_timing_intervals ...

controller_total = sim.run(
    number_of_jobs=4,
    split_policy="split_in_time_total",
    wait_for_result=True,
    merge_after_run=True,
)

controller_per_run = sim.run(
    number_of_jobs=4,
    split_policy="split_in_time_per_run",
    wait_for_result=True,
    merge_after_run=True,
)

In practice, split_in_time_total is the recommended default unless the simulation logic or the validation strategy specifically benefits from staying closer to the original run partition.

Tip

If you use split_in_time_per_run and set the number of jobs equal to the number of run timing intervals in your simulation, you will get one job per run timing interval.

Server-based execution#

Server-based multi-job execution is intended to be file-based:

prepare a campaign folder containing simulation.json and the required input files, probably locally
transfer the campaign folder to the server
split the simulation on the server
submit the child jobs through a scheduler
inspect job status
merge the finished jobs
optionally clean temporary split artifacts
transfer the campaign folder with the merged output back to your local machine

Warning

Server-based multi-job execution is currently untested. The workflow and command-line tools are available, but they should still be treated as experimental until broader validation has been completed.

Suggested server workflow#

On a large server, execution of simulations is usually handled via command-line tools.

Assume a campaign folder called campaign containing:

simulation.json
any input files needed by the simulation

Then the workflow is:

opengate_jobs_split campaign --number-of-jobs 100
opengate_jobs_run campaign --backend htcondor --backend-options-json htcondor_options.json
opengate_jobs_status campaign
opengate_jobs_merge campaign
opengate_jobs_clean campaign

The campaign folder is the authoritative container throughout the workflow. It contains:

the master simulation.json
optionally simulation_resolved.json
the child folders job0001, job0002, …
the manifest and status files used by the jobs tools
the merged output

Backend options on a server#

The opengate_jobs_run command accepts the backend explicitly and can read backend options from a JSON file:

opengate_jobs_run campaign --backend htcondor --backend-options-json htcondor_options.json

As a convenience, the command also looks for a default file named jobs_backend_options.json inside the campaign folder when no --backend-options-json or --backend argument is provided.

Example: HTCondor#

A possible HTCondor backend options file could look like:

{
  "backend": "htcondor",
  "backend_options": {
    "submit_file_commands": {
      "request_memory": "8 GB",
      "request_cpus": "4"
    },
    "command_line_args": ["-batch-name", "gate_jobs"]
  }
}

The actual submission requirements are often site-dependent. The backend options should therefore be adapted to the target server.

Command-line tools#

GATE ships with multi-job command-line tools mainly intended for handling simulations on servers:

opengate_jobs_split: split a campaign folder into child jobs
opengate_jobs_run: launch a split campaign through a backend
opengate_jobs_status: inspect the current campaign status
opengate_jobs_merge: merge the finished jobs
opengate_jobs_clean: remove temporary split-job artifacts
opengate_job_runner: execute one individual child job folder

The single-job runner is mainly useful for scheduler payload commands. End users will usually work with the campaign-level commands instead.