# Buildup simulation scans

Use one independent HTCondor job per scan point. Each job receives common files
and its own input archive, runs in scratch, and returns one explicitly named
archive to EOS. Read the [storage and environment setup](index.md) first.

## Prepare the scan

Keep this submission layout on AFS:

```text
scan/
  htcondor.sub
  job.sh
  sims.txt
  logs/
  point_000/inputs.tgz
  point_001/inputs.tgz
```

`sims.txt` contains one relative directory name per line (`point_000`,
`point_001`, ...), without spaces. Put each point's `.input` and `.beam` files
at the top level of its `inputs.tgz`. Put `run_sim.py`, common geometry, and
other shared files at the top level of `jobfiles.tgz` on EOS. Paths in the
simulation inputs must resolve within the extracted working directory.

## Submit description

Save as `htcondor.sub`. Resource requests below are illustrative; allow scratch
space for inputs, unpacked results, and the compressed archive simultaneously.

```text
universe = vanilla
arguments = ""
executable = job.sh
output = htcondor.out
error = htcondor.err
log = logs/$(ClusterId).log

should_transfer_files = YES
transfer_input_files = $(folder)/inputs.tgz, root://eosproject-e.cern.ch//eos/project/e/ecloud-simulations/YOURDIRECTORY/scan/jobfiles.tgz
when_to_transfer_output = ON_EXIT
transfer_output_files = output.tgz
output_destination = root://eosproject-e.cern.ch//eos/project/e/ecloud-simulations/YOURDIRECTORY/scan/results/$(folder)/
MY.XRDCP_CREATE_DIR = True

+JobFlavour = "tomorrow"
queue folder from sims.txt
```

Each scan point has one EOS directory containing `output.tgz`. Resubmitting
a point reuses that destination. The event log stays on AFS.

## Worker script

Save as `job.sh`. Set `OUTPUT_MAT` to the filename configured by
`filen_main_outp` in your simulation inputs, including the `.mat` extension.
The default option set by PyECLOUD is `Pyecltest.mat`.

The script archives the output, log, and progress files by name, so in
`simulation_parameters.input` set them to plain filenames, without directories:

```python
filen_main_outp = 'Pyecltest.mat'   # must match OUTPUT_MAT
logfile_path = 'logfile.txt'
progress_path = 'progress.txt'
```

These files are then written to the job's scratch directory, where the script
expects them. A path such as `'/eos/.../logfile.txt'` or `'output/logfile.txt'`
writes them elsewhere, and archiving fails because they are missing. If you
choose other names, change them in the script's `tar` command as well.

```bash
#!/bin/bash
set -e

OUTPUT_MAT="Pyecltest.mat"

# Keep an archive available if a later command fails.
printf 'Job did not complete; see the HTCondor logs.\n' > job-status.txt
tar -czf output.tgz job-status.txt

tar -xzf jobfiles.tgz
rm jobfiles.tgz
tar -xzf inputs.tgz
rm inputs.tgz

# Remove if using container and replace with container command -----
# Batch shells do not run 'conda init'; load Conda's shell functions first.
eval "$(conda shell.bash hook)"
conda activate /eos/user/{yourinitial}/{yourusername}/pyecloud-env
run_python() {
    python "$@"
}
# -------------------------------------------------------------------

# Record the start time and software version.
date > envinfo.txt
run_python -c 'import os, PyECLOUD; key = "ECLOUD_CONTAINER_VERSION"; print(f"{key}={os.environ[key]}" if key in os.environ else f"PyECLOUD={PyECLOUD.__version__}")' >> envinfo.txt
run_python run_sim.py

printf 'Simulation completed successfully.\n' > job-status.txt
optional_files=()
if [[ -f stop.txt ]]; then optional_files+=(stop.txt); fi

tar -czvf output.tmp.tgz "$OUTPUT_MAT" progress.txt logfile.txt \
    envinfo.txt job-status.txt "${optional_files[@]}" *.input *.beam
mv output.tmp.tgz output.tgz
```

The simulation runs directly in `_CONDOR_SCRATCH_DIR`. `set -e` stops the
script if extraction, activation, Python, or archiving fails. The final archive
contains the named result and log files, the inputs, and `stop.txt` when present.
Required files must exist for archiving to succeed.

A small fallback archive is created before the simulation. The final archive
replaces it only after `tar` succeeds, so an ordinary failure still leaves
`output.tgz` available for HTCondor to transfer. Forced termination or disk
failure can still prevent delivery. Check the job exit code and logs before
using the results.

### Using the container

Remove the two Conda lines (`eval` and `conda activate`) and replace `run_python` with the
[CVMFS container wrapper](index.md#alternative-a-cvmfs-container). No environment activation is needed for
the container.

## Submit

```bash
mkdir -p logs
condor_submit htcondor.sub
```

## Alternative: transfer individual files from the worker

To avoid archiving, set `transfer_output_files = ""`, remove
`output_destination`, and explicitly upload selected outputs with `xrdcp`
after Python exits. Capture Python's exit code before transferring. Enumerate
existing local files in the shell and check **every** transfer's exit code;
do not pass wildcards to `transfer_output_files`. Fail the job if a required
result is missing or its upload fails. Use the scan point's EOS directory and retain logs for failed simulations too.