Job queue
When you run squadron job queue, your job goes into a queue. Within about a minute it will be assigned to an available machine and start running there.
How jobs get assigned
The system checks for pending jobs roughly every 60 seconds. When it finds one, it will pick the least-loaded machine that meets the job's requirements, and that machine will start the job within a few seconds.
A machine can run at most 1 job at once. If all machines are busy, your job will wait until one is free.
GPU matching
If your manifest sets cuda in the [job] section (e.g. cuda = "13.0"), your job will only run on machines whose GPU driver supports that CUDA version or newer. This is a scheduling constraint, not something that installs a driver in the container. If cuda is omitted (the default), the job can run anywhere.
Memory matching
The memory field in the [job] section of manifest.toml controls how much memory your job gets. Note that its units are binary, so MiB and GiB rather than MB or GB.
squadron project test will apply this memory limit as long as you don't pass --skip-limits, so run that to ensure you have enough.
Running your own jobs while busy
When you're actively using your machine, it's marked BUSY. Other people's jobs won't get assigned to it. But your own jobs still can, by default, since you probably don't want your own work to stall just because you're at your desk.
Idle machines are always preferred. Your job will only be assigned to a busy machine (that you own) if no idle machines anywhere are available.
To opt out of this behavior entirely, run squadron daemon forbid-busy-assign. See Daemon for details.
What happens when a machine goes down
Jobs don't get stuck. If a machine goes offline, runs too long, or becomes busy while running someone else's job, that job will go back into the queue and be reassigned to a different machine on the next pass.
When the new machine picks up the job, the files the previous run wrote to /artifacts are there again. Nothing transfers up front: each file is pulled from storage on demand, so a large checkpoint only downloads when your code actually reads it. Your code can resume where it left off, as long as you write checkpoints to /artifacts and check for them at startup.
Write your program so it can resume from /artifacts. Save checkpoints there periodically, and on startup, look for existing checkpoints to continue from. Skip this and a reassigned job will restart from scratch every time it moves to a new machine. See Adapting Your Code for details.
Job states
| Status | Meaning |
|---|---|
ACTIVE | Queued or currently running |
PAUSED | You paused it; the system skips it |
SUCCESS | The container exited with code 0 |
FAILURE | The container exited with a non-zero code |
Pausing and resuming
You can pause a job from the CLI (squadron job pause) or from the Jobs page. A paused job stays in the system but won't be assigned. Resume it to put it back in the queue.
Admins and Owners can pause or resume anyone's jobs. Regular members can only pause their own.
Email notifications
If the manifest sets email_on_success or email_on_failure to true, you'll get an email when the job finishes. The email goes to the address set in Settings, by default the email associated with your login. You can also override these flags per-job when queuing from the CLI.
Completed jobs in the dashboard
The Jobs page shows completed jobs from the last 30 days, capped at 100. Older jobs are still accessible via squadron job ls.