Runner Hardening Best Practices

Runner hardening is one of the highest-leverage CI/CD security improvements you can make: runners (build agents) routinely execute attacker-influenced code (PRs, dependencies, build scripts) while holding the network access and credentials needed to build and ship software. This guide covers practical runner hardening controls—ephemeral isolation, least privilege, secrets protection, patching, egress controls, and monitoring—so a compromised job can’t easily become a compromise of your org.

How it works

Hardening a runner is about constraining what pipeline code can do and what it can access, even if the pipeline itself is malicious (e.g., a compromised dependency, a poisoned PR, or a malicious maintainer).

1) Choose the safest execution model: ephemeral > persistent

• Ephemeral runners (recommended): A fresh VM/container per job, destroyed afterward. This prevents cross-build persistence, reduces data remanence, and limits lateral movement.
• Persistent runners: Convenient but riskier. Attackers can plant backdoors, modify toolchains, or exfiltrate cached credentials across jobs.

Practitioner checklist - Use auto-scaling runner pools (VMs or containers) that are rebuilt from a golden image. - Ensure teardown deletes disks/snapshots or encrypts and rotates keys.

2) Strong isolation boundaries

Isolation should prevent a build from escaping into the host or neighboring builds.

• Prefer VM isolation for untrusted workloads (fork PRs, external contributors).
• If using containers, run them rootless where possible and avoid privileged Docker access.

Key pitfalls - Giving a job access to the Docker socket (/var/run/docker.sock) effectively grants root on the host. - Running “privileged” containers or mounting host paths into build containers expands the blast radius.

3) Least privilege for runner identity and tokens

Runners often authenticate to: - Source control (clone, fetch, status updates) - Artifact registries - Cloud providers (deploy, infrastructure) - Secrets stores

Hardening means: - Scope credentials per repository/environment (not org-wide). - Use short-lived, federated credentials (OIDC) instead of long-lived static keys. - Separate roles for build vs deploy. A build runner should not have production deploy permissions.

4) Secrets: minimize exposure and prevent exfiltration

Treat secrets as “radioactive”—pipeline code will try to print or export them.

• Use masked secrets, but don’t rely on masking alone (encoding/partial leaks bypass masking).
• Avoid injecting secrets into jobs that run on untrusted code paths (e.g., fork PRs).
• Prefer dynamic credentials (OIDC → cloud STS tokens) with short TTLs and tight conditions.

Practical rules - No production secrets in PR validation pipelines. - Use environment approvals for deploy stages. - Restrict secrets access to protected branches/tags. - Require strong credentials for any human “break glass” accounts that can modify runners/pipelines (see how do i create a strong password).

Optional tooling (helps, but doesn’t replace hardening): a password manager can reduce secret sprawl and improve rotation hygiene for human-operated CI/CD admin accounts. If you already use one, consider consolidating privileged credentials in a vault like 1Password: Try 1Password →.

5) Lock down network egress and internal access

Runners frequently have broad outbound internet access and sometimes flat access to internal networks.

• Restrict egress to required destinations (SCM, artifact registries, package mirrors).
• Deny direct access to sensitive internal services (databases, metadata endpoints, admin panels).
• Use private package mirrors and allow-list them.

Common high-impact issue: cloud instance metadata access (e.g., IMDS) from builds can allow credential theft if not restricted.

6) Supply-chain hygiene inside the runner

Runners fetch dependencies and tooling—prime targets for poisoning.

• Pin dependencies (lockfiles), pin container base images by digest.
• Verify provenance where available (signed artifacts, attestations).
• Keep build tools updated; remove compilers/interpreters you don’t need on deploy-only runners.

7) Logging, detection, and integrity monitoring

You need evidence when something goes wrong.

• Centralize runner logs (OS, auth, process, network).
• Alert on suspicious behavior: unexpected outbound connections, new SSH keys, cron jobs, new privileged users, tampering with build toolchains.
• Maintain runner image integrity: build from IaC; detect drift from baseline.

When you’ll encounter it

Runner hardening becomes essential when any of these are true:

• You accept external contributions (public repos, open-source, vendor PRs). Untrusted PR code can execute in CI if misconfigured.
• Your runners can reach production or sensitive networks (VPN, VPC peering, on-prem).
• Your pipelines use powerful credentials (cloud admin keys, deploy tokens, registry push tokens).
• You use self-hosted runners for performance, custom tooling, or network access.
• You build artifacts used by customers (software supply chain risk). A runner compromise can become a downstream compromise.

Typical scenarios: - “We need access to an internal artifact registry, so we added self-hosted runners” → now your build environment is inside your trust boundary. - “Fork PR checks are slow, so we run them on our main runners” → risk of secret leakage or host compromise.

Technical Notes (practical hardening snippets)

OS baseline: patching, services, and filesystem checks (Linux)

Keep the runner host minimal and patched; detect suspicious persistence.

# Patch baseline (example for Debian/Ubuntu)
sudo apt-get update && sudo apt-get -y upgrade

# Quick inventory of risky services and listening ports
sudo ss -lntup
sudo systemctl --type=service --state=running

# Look for unexpected scheduled persistence
sudo crontab -l
sudo ls -la /etc/cron.* /var/spool/cron || true

# New/unknown privileged users
getent passwd | awk -F: '$3==0 {print $1}'
sudo grep -R "sudo" /etc/sudoers /etc/sudoers.d -n

Block “Docker socket = root” patterns

If your job uses Docker, avoid mounting the host socket; use isolated builders (VM-based runners, rootless, or remote build service).

Anti-pattern to eliminate:

# If a job can do this, it can usually control the host:
docker -H unix:///var/run/docker.sock run --privileged -v /:/host alpine chroot /host /bin/sh

Detection idea (host):

# Find which processes have the Docker socket open
sudo lsof /var/run/docker.sock 2>/dev/null | head -n 50

Restrict outbound network (host-based example)

At minimum, start with “default deny egress” on sensitive runner pools and allow only required endpoints.

# Example using UFW (adjust for your environment)
sudo ufw default deny outgoing
sudo ufw allow out 443/tcp   # HTTPS to SCM/artifact endpoints (prefer allow-list by IP/FQDN via proxy)
sudo ufw allow out 53        # DNS (or force through your resolver)
sudo ufw enable
sudo ufw status verbose

If you can’t allow-list by FQDN at the firewall, route runner traffic through an authenticated proxy with URL filtering and logging.

Protect secrets in logs and environment

Minimize secrets exposure and scan logs/artifacts for accidental leaks.

# Spot common leak patterns in job logs/artifacts (example heuristic)
grep -RInE "(BEGIN RSA PRIVATE KEY|AKIA[0-9A-Z]{16}|xox[baprs]-|ghp_[A-Za-z0-9]{36})" ./artifacts ./logs 2>/dev/null | head

Also ensure builds don’t dump environment variables by default. In many incidents, the root cause is a debug flag or env output in a failing job.

Monitor for suspicious network/process behavior

Forward OS logs and add lightweight detections: - Outbound connections to paste sites, file-sharing, unknown VPS IPs - curl | bash execution patterns - New SSH authorized keys

# Quick triage: recent outbound connections (requires audit/EDR for completeness)
sudo journalctl -S "1 hour ago" | grep -iE "(curl|wget|nc|ncat|ssh|scp|base64|python -c|perl -e)" | head -n 100

# Check for unauthorized SSH keys (runner hosts should typically not accept SSH)
sudo find /home /root -maxdepth 3 -name authorized_keys -type f -print -exec sudo sed -n '1,5p' {} \;