Troubleshooting
Job is stuck
Section titled “Job is stuck”Check that the job tags match tags exposed by the RealHW project runner.
tags: - saas-linux-medium-amd64If the job has a hardware marker, confirm your project has access to that hardware class.
Job runs on the wrong shape
Section titled “Job runs on the wrong shape”Use one GitLab-compatible SaaS tag for the generic shape:
tags: - saas-linux-large-amd64Avoid mixing multiple generic size tags unless you intentionally allow any of those sizes.
Hardware job waits
Section titled “Hardware job waits”Hardware markers describe host capability, not immediate availability. A valid GPU, NIC, or SR-IOV request can wait if matching hardware is busy.
Job fails before container start
Section titled “Job fails before container start”The request may be unsatisfiable or blocked by policy. Check unsupported markers, unavailable project entitlements, excessive HW_CPU or HW_MEMORY, and unsupported HW_* variables.
Disk capability probe fails
Section titled “Disk capability probe fails”Compare the reported free space with the job’s floor on the build filesystem.
A script that prints “infra failure” and exits still produces GitLab
script_failure; the message alone does not change the failure category.
RealHW disk rejection before execution uses runner_system_failure.
Ask the fleet operator to inspect disk capacity, drain the affected host, and reclaim unused managed caches or expand capacity before retrying. RealHW admission reserves a default 10 GiB floor plus 4 GiB of growth headroom per active job. A workload can require more; this check is not a disk quota.
KVM is missing or unusable
Section titled “KVM is missing or unusable”Request the kvm capability in addition to the desired size or architecture.
A SaaS-compatible size tag describes a shape and does not guarantee KVM.
The host must provide usable KVM, and the executor must expose /dev/kvm
to the job container. Check both layers when the probe fails.
RealHW supports owned KVM hosts and a dedicated GCE N2 nested-KVM pool. The GCE worker creates a KVM VM during bootstrap before enrolling, but the job must still verify that its container can use the device.
QEMU starts but the guest times out
Section titled “QEMU starts but the guest times out”Once disk and KVM probes pass, inspect the serial log for the expected boot markers and the first panic. A QEMU timeout is not evidence of the original disk failure. Preserve the job revision and artifacts so kernel failures can be investigated separately from runner-capacity problems.