Daemon
The Squadron daemon runs in the background on every registered machine. When the machine is idle, it pulls queued jobs and runs them.
What "idle" means
The daemon considers a machine available only when all of these are true:
| Metric | Condition |
|---|---|
| Not manually paused | squadron daemon pause hasn't been run recently |
| CPU usage | Non-job usage ≤ 5% (averaged across all cores) |
| RAM usage | Non-job usage ≤ 16 GiB |
| GPU compute utilization | Non-job SM utilization ≤ 5% (NVIDIA only) |
| GPU memory usage | Non-job VRAM usage ≤ 2 GiB (NVIDIA only) |
| Screen inhibit | No screen inhibit active (e.g. YouTube keeping the screen awake) |
| User inactivity | No keyboard or mouse input for at least 300 seconds |
"Non-job" means the daemon subtracts the resources its own containers are using before comparing against these numbers. A machine running a job does not count as busy on that basis alone.
If any condition fails, the daemon will wait and re-check before picking up a job. If a machine isn't taking jobs you expect it to, start with this table.
Machines that stop reporting
The daemon sends a heartbeat every few seconds. If the server stops hearing from a machine, it will mark that machine offline and return any jobs it was running to the queue for reassignment. Once the daemon reaches the server again, the machine comes back on its own, so you don't have to do anything to recover a laptop that slept or a workstation that dropped off the network.
Pausing the daemon
To temporarily stop the daemon from picking up new jobs without deregistering the machine:
# Pause for 8 hours (default)
squadron daemon pause
# Pause for a specific duration
squadron daemon pause 2hResume early with:
squadron daemon resumeResource limits
Every job container runs with memory and process limits to protect the host machine.
| Limit | Value |
|---|---|
| Memory | memory from the [job] section of the manifest |
Shared memory (/dev/shm) | the same as memory |
| Swap | none |
| Max processes | 4096 |
/dev/shm gets the same allowance as the container itself, so shared-memory-hungry code like PyTorch DataLoaders has the full budget to work with.
If your job exceeds the memory limit, the container will be killed and the job marked as failed. The logs will include an out-of-memory message so you can tell what happened. If it hits the PID limit instead, new calls to fork() will start failing.
Testing locally with squadron project test, you can pass --skip-limits to lift the memory and PID caps. There is no equivalent for the daemon: jobs on real machines always run with the limits in place.
Running your own jobs on your busy machines
Busy self-assignment is on by default, which means your own jobs can land on your own machines even while they're marked BUSY (that is, while you're actively using them). Other people's jobs will still wait for the machine to go idle.
The scheduler prefers idle machines over busy ones. Your busy machine will only be used when no idle machine can take the job.
If you want your machines to only run your jobs when fully idle, disable it:
squadron daemon forbid-busy-assignTo re-enable:
squadron daemon allow-busy-assign