Stability simulations with continuation
For single-bunch PyPARIS simulations, use a DAGMan node that runs one part at a time. The worker script downloads the inputs and last completed state, runs one simulation part, and uploads its checkpoint. Read the storage and environment setup first.
Configure simulation parts
In Simulation_parameters.py, for example:
N_turns = 100
N_turns_target = 1000
check_for_resubmit = True
submission_system = 'HTCondor'
resubmit_command = None
Choose each part to finish comfortably within the job time limit, including
input and output transfers. PyPARIS returns 177 when a completed part needs
continuation. A normal final exit is 0; inspect the simulation logs to
distinguish reaching the target from an intentional early stop. Other codes
represent failures. See saving and restarting.
Submit description
Save as htcondor.sub. Keep this file, pyparis_executable.sh, the POST
script, DAG, and logs/ on AFS. Put the simulation parameters and
pyecloud_config in an existing EOS run directory. This example transfers
LHC_chm_ver.mat separately; replace its URL with your geometry file and make
sure the simulation configuration refers to the transferred filename.
universe = vanilla
executable = pyparis_executable.sh
arguments = $(request_cpus)
output = logs/$(ClusterId).$(ProcId).out
error = logs/$(ClusterId).$(ProcId).err
log = logs/$(ClusterId).log
should_transfer_files = YES
transfer_input_files = root://eosproject-e.cern.ch//eos/project/e/ecloud-simulations/YOURDIRECTORY/stability/LHC_chm_ver.mat
when_to_transfer_output = ON_EXIT
transfer_output_files = ""
request_cpus = 8
request_memory = 16GB
request_disk = 20GB
+JobFlavour = "tomorrow"
queue 1
Adjust CPU, memory, and disk requests to your simulation.
Passing request_cpus to the script makes PyPARIS use the allocated number
of processes. Simulation data travels to EOS from
the worker, so no simulation filenames appear in transfer_output_files.
Worker script
Download pyparis_executable.sh
and save it beside htcondor.sub. Set ROOT_URL, SIM_PATH, and the EOS
environment path. The script normalizes trailing slashes in ROOT_URL and
SIM_PATH. See
Python environment for setup instructions.
This script follows the example in
examples/HTCondor_templates/PyPARIS_singlebunch_checkpoint_job/.
#!/bin/bash
set -uo pipefail
ROOT_URL="root://eosproject-e.cern.ch/"
SIM_PATH="/eos/project/e/ecloud-simulations/YOURDIRECTORY/stability/run_000/"
# Normalize to one trailing slash before joining URLs and filenames.
while [[ "$ROOT_URL" == */ ]]; do ROOT_URL="${ROOT_URL%/}"; done
ROOT_URL="$ROOT_URL/"
while [[ "$SIM_PATH" == */ ]]; do SIM_PATH="${SIM_PATH%/}"; done
SIM_PATH="$SIM_PATH/"
previous_bunch_state=""
JOB_PWD="${_CONDOR_SCRATCH_DIR:?}"
cpus=${1:?Missing CPU count}
xrdcp_opts=(--retry 3)
cd "$JOB_PWD" || exit 1
run_python() {
python "$@"
}
checkpoint_files() {
local part_num part
part_num=$(sed -n 's/^present_simulation_part = \([0-9][0-9]*\)$/\1/p' simulation_status.sta) || return 1
[[ "$part_num" =~ ^[0-9]+$ ]] || return 1
part=$(printf '%02d' "$((10#$part_num))") || return 1
files=("bunch_evolution_${part}.h5" "slice_evolution_${part}.h5"
"bunch_status_part${part}.h5" pyparislog.txt sim_param.pkl
envinfo.txt stdout.txt stderr.txt)
}
transfer_inputs() {
local listing file
# A failed listing is an error, not evidence of a fresh simulation.
listing=$(xrdfs "$ROOT_URL" ls "${SIM_PATH%/}") || return 1
if grep -Fxq "${SIM_PATH}simulation_status.sta" <<< "$listing"; then
xrdcp "${xrdcp_opts[@]}" "${ROOT_URL}${SIM_PATH}simulation_status.sta" ./ || return 1
checkpoint_files || return 1
for file in "${files[@]}"; do
xrdcp "${xrdcp_opts[@]}" "${ROOT_URL}${SIM_PATH}${file}" ./ || return 1
done
previous_bunch_state="${files[2]}"
for file in multigrid_config_dip.pkl multigrid_config_dip.txt; do
if grep -Fxq "${SIM_PATH}${file}" <<< "$listing"; then
xrdcp "${xrdcp_opts[@]}" "${ROOT_URL}${SIM_PATH}${file}" ./ || return 1
fi
done
else
echo "No simulation status in the accessible run directory; starting fresh."
fi
xrdcp "${xrdcp_opts[@]}" "${ROOT_URL}${SIM_PATH}Simulation_parameters.py" ./ || return 1
xrdcp "${xrdcp_opts[@]}" -r "${ROOT_URL}${SIM_PATH}pyecloud_config" ./ || return 1
}
transfer_outputs() {
local file
grep -Fxq 'present_part_done = True' simulation_status.sta || return 1
grep -Fxq 'present_part_running = False' simulation_status.sta || return 1
checkpoint_files || return 1
# Check all required files before starting publication.
for file in "${files[@]}"; do
[[ -f "$file" ]] || { echo "Missing checkpoint file: $file" >&2; return 1; }
done
for file in "${files[@]}"; do
xrdcp "${xrdcp_opts[@]}" -f "$file" "${ROOT_URL}${SIM_PATH}${file}" || return 1
done
for file in multigrid_config_dip.pkl multigrid_config_dip.txt; do
if [[ -f "$file" ]]; then
xrdcp "${xrdcp_opts[@]}" -f "$file" "${ROOT_URL}${SIM_PATH}${file}" || return 1
fi
done
# Publish the status only after every associated file has arrived.
xrdcp "${xrdcp_opts[@]}" -f simulation_status.sta \
"${ROOT_URL}${SIM_PATH}simulation_status.sta" || return 1
# Delete only the state we resumed from, after the new checkpoint is saved.
if [[ -n "$previous_bunch_state" && "$previous_bunch_state" != "${files[2]}" ]]; then
if ! xrdfs "$ROOT_URL" rm "${SIM_PATH}${previous_bunch_state}"; then
echo "Warning: checkpoint saved, but could not remove $previous_bunch_state" >&2
fi
fi
return 0
}
transfer_inputs || exit 1
# Remove if using container -------------------------------------------------
# Batch shells do not run 'conda init'; load Conda's shell functions first.
eval "$(conda shell.bash hook)" || exit 1
conda activate /eos/user/{yourinitial}/{yourusername}/pyecloud-env || exit 1
# ---------------------------------------------------------------------------
# Append one record per simulation part: start time and software version.
date >> envinfo.txt
run_python -c 'import os, PyECLOUD; key = "ECLOUD_CONTAINER_VERSION"; print(f"{key}={os.environ[key]}" if key in os.environ else f"PyECLOUD={PyECLOUD.__version__}")' >> envinfo.txt
run_python -m PyPARIS.multiprocexec -n "$cpus" \
sim_class=PyPARIS_sim_class.Simulation.Simulation >> stdout.txt 2>> stderr.txt
script_exit=$?
echo "Simulation exited with code: $script_exit"
if (( script_exit == 0 || script_exit == 177 )); then
transfer_outputs || exit 1
else
echo "Simulation failed; completed checkpoint not published." >&2
tail -n 100 stderr.txt >&2
fi
exit "$script_exit"
Checkpoint files
The same file set is downloaded on continuation and uploaded after each
completed part. NN is present_simulation_part formatted with at least two
digits (00, 01, …).
File |
Purpose |
|---|---|
|
Part number and completion state; uploaded last |
|
Saved bunch state |
|
Evolution monitors for the part |
|
Saved simulation parameters |
|
Simulation log |
|
Start time and software version of every part, one record appended per part |
|
Worker-captured Python output |
|
Optional saved dipole multigrid configuration, copied when present |
Simulation_parameters.py and pyecloud_config are fetched for every part.
Keep these inputs and the software environment consistent throughout the run.
The run directory must exist before the first job. A successful directory
listing without a status file means a fresh run; a failed listing aborts the
job. Never remove the status file to recover a failed continuation.
Alternative: use the maintained container
Replace run_python with the CVMFS container wrapper
and remove the two Conda lines (eval and conda activate). No environment activation is needed
when using the container.
The exact container path is:
/cvmfs/unpacked.cern.ch/ghcr.io/ekatralis/ecloud-containers:latest/
Input and output transfers always run in the host worker shell.
Exit codes and checkpoint transfers
The script captures Python’s exit code immediately. It uploads the checkpoint
on 0 or 177, then returns that code. Input or output transfer failures return
1. Do not add an unguarded set -e around Python: 177 is expected and the
checkpoint must be uploaded before exiting.
The script checks the required files and transfer results, then uploads
simulation_status.sta last. Only after all uploads succeed does it delete
the previous bunch_status_partNN.h5 used to resume the simulation. The new
bunch state and evolution files are kept. If deletion fails, the script logs
a warning and preserves the simulation exit code so continuation can proceed;
the old file can be removed manually.
An interrupted upload can leave files from different parts on EOS. Inspect
the checkpoint before restarting. Run only one workflow per EOS simulation
directory.
DAGMan continuation
Save as workflow.dag:
JOB simulation htcondor.sub
SCRIPT POST simulation post_check.sh $RETURN
RETRY simulation 999 UNLESS-EXIT 0
ABORT-DAG-ON simulation 11 RETURN 1
Save as post_check.sh:
#!/bin/bash
case "${1:-missing}" in
0) exit 0 ;;
177) exit 177 ;;
*) exit 11 ;;
esac
The POST script runs on the submission side. It maps the worker return code
to a DAG decision: 0 completes the node, 177 requests another part, and
any other result becomes 11, aborting the DAG. The retry count bounds the
number of additional parts; choose it to cover your planned run. See
DAGMan retry and abort behavior.
mkdir -p logs
chmod +x pyparis_executable.sh post_check.sh
condor_submit_dag workflow.dag
A job held during output transfer can delay the POST script and prevent
continuation. Inspect condor_q -hold, the scheduler event log, and the DAGMan
output when continuation stops. This workflow continues completed parts;
it does not automatically recover a worker killed mid-part. Resolve incomplete
state or transfer failures before restarting the workflow.