feat(dind): multi-queue support, Docker mode fixes, and gVisor sandbox support#876
Conversation
eca8932 to
33125ec
Compare
|
This syntax seems awkward to operate (easier to get wrong than to get right for a new customer's junior admin) queues:
- name: batches
replicaCount: 4
env:
EXECUTOR_QUEUE_NAME:
value: batches
- name: codeintel
replicaCount: 4
env:
EXECUTOR_QUEUE_NAME:
value: codeintelIf possible, this would be simpler: queues:
batches:
replicaCount: 4
codeintel:
replicaCount: 4Or this, if either queue can use the same number of replicas queues:
batches:
codeintel: |
|
Actually, to help reduce the transition cost of moving customers from the k8s Helm chart to this one, it'd be ideal if this Helm chart could read the same values from an override file which was successful with the k8s Helm chart @michaellzc |
|
Some examples here: https://sourcegraph.sourcegraph.com/search?q=repo:%5Egithub%5C.com/sourcegraph/customer-assets$+lang:YAML+%5Eexecutor:$&patternType=regexp&sm=0 Also to note, it's common to use one Helm values override.yaml file when deploying multiple of our Helm charts, so the values which get read from multiple charts don't need to be maintained in duplicate in separate override files. |
marcleblanc2
left a comment
There was a problem hiding this comment.
Left comments in the PR thread re: ergonomics of using the added YAML syntax; it looks easier to get wrong than it is to get right, especially for some of our self-hosted customer admins.
- Add `queues` list to values: when set, renders one Deployment + Service per queue instead of the single executor Deployment - Each queue merges with global executor.env (queue overrides global) and supports replicaCount and resources overrides - Add global executor.resources and per-queue resources override - Fix readiness probe path: /ready does not exist, use /healthz - Set EXECUTOR_USE_KUBERNETES=false to prevent the executor from auto-detecting Kubernetes mode via KUBERNETES_SERVICE_HOST/PORT (which are always injected into pods); dind chart runs in Docker mode using the dind sidecar - Set enableServiceLinks: false to reduce noise in pod env - Fix private docker registry image tag (registry:3) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
--registry-mirror was pointing at http://executor:5000 which doesn't resolve; the private registry service is private-docker-registry:5000. Also fix ConfigMap condition to render in both single and multi-queue modes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
StatefulSet gives stable pod identity and manages its own PVC via volumeClaimTemplates, replacing the separate PersistentVolumeClaim. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add first-class values for options that map directly to executor env vars, mirroring the k8s chart interface: - executor.frontendUrl/frontendPassword/frontendExistingSecret - executor.queueName/queueNames (single-deployment mode) - executor.maximumNumJobs/maximumRuntimePerJob - executor.log.level/format - executor.dockerAddHostGateway - executor.debug.keepWorkspaces In multi-queue mode, each queue's EXECUTOR_QUEUE_NAME is set automatically from its name field. Add executor.validateEnv helper that fails at render time if executor.env or any queue.env contains a managed env var name. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- executor.enabled had no meaningful use; the chart always deploys an executor, so the flag is removed - dind.daemonConfig is now a values map rendered as daemon.json, allowing arbitrary Docker daemon configuration via values overrides Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extract the full Deployment body into executor.deployment in _helpers.tpl. executor.Deployment.yaml is now pure dispatch — resolves the per-queue vs single-deployment args and delegates to the shared template. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Helm trims whitespace between range iterations, causing the --- inside executor.deployment to land on the same line as the prior document's last character. Both deployments ended up in a single YAML document, making helm track them incorrectly. Move the separator to the call site. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Each queue now supports explicit queueName (EXECUTOR_QUEUE_NAME) and queueNames (EXECUTOR_QUEUE_NAMES) fields, mirroring the global executor.queueName/queueNames. The deployment name still comes from queue.name; queueName defaults to name when omitted. The two env vars are mutually exclusive: queueNames takes precedence. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Change --host=tcp://0.0.0.0:2375 to 127.0.0.1:2375 so the unauthenticated Docker API is not reachable from other cluster pods - Add dind.gVisor.enabled (default false): sets runtimeClassName: gvisor and replaces privileged: true with explicit capabilities per https://gvisor.dev/docs/tutorials/docker-in-gke-sandbox/ gVisor intercepts these capabilities in-sandbox; no host privileges are granted regardless of the capability list Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gVisor's kernel does not implement the ioctls required by overlay2, aufs, or devicemapper — all three fail with "invalid argument" or "not found". vfs is the only storage driver that works inside the gVisor sandbox. Add --storage-driver=vfs to the dockerd command when dind.gVisor.enabled is true so the daemon starts successfully on gVisor nodes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gVisor's virtual network layer exposes docker0 as a plain interface, not a bridge, causing dockerd to fail at network controller initialization. --iptables=false bypasses the bridge/netfilter setup path that requires kernel modules (bridge, br_netfilter, nf_conntrack) unavailable in gVisor. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Docker 29 + gVisor requires specific setup before dockerd can start: - ip_forward must be enabled manually (gVisor doesn't auto-enable it) - iptables-legacy NAT POSTROUTING rules must be set up before dockerd since --iptables=false prevents dockerd from managing them - docker0 must be deleted before restart to avoid "existing interface docker0 is not a bridge" crash on CrashLoopBackOff recovery - --feature containerd-snapshotter=false is required for Docker 29+ on gVisor since the containerd image store needs tmpfs as upper layer - --ip6tables=false required alongside --iptables=false per gVisor docs Also bumps dind liveness/readiness initialDelaySeconds (5/10 → 15/20) to give the setup script time to run before the probe fires. References: https://gvisor.dev/docs/tutorials/docker-in-gvisor/ Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs: 1. `ip link del docker0` in gVisor silently fails (or docker0 is pre-existing and can't be deleted), leaving a non-bridge docker0 when dockerd starts. Fix: explicitly create docker0 as a bridge (del, add type bridge, assign 172.17.0.1/16, bring up) before starting dockerd so Docker finds a proper bridge interface. 2. `sed '\s'` is not portable on Alpine's busybox sed — the \s escape is not recognized, causing the device-name variable to capture all ip-route output lines. Fix: use awk to extract the first 'dev' token from the default route. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ating gVisor's netlink implementation does not report IFLA_INFO_KIND="bridge" when querying an interface created with `ip link add type bridge`. This causes Docker's bridge driver to fail the pre-existing interface type check (link.Type() != "bridge"). The fix: don't pre-create docker0. When Docker creates it via Go's netlink library (bridgeAlreadyExists=false), the type check is skipped entirely. Keep `ip link del docker0` for restart resilience — on CrashLoopBackOff restarts, docker0 from the previous run is deleted before dockerd starts, so Docker always sees it as a fresh create. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tcpSocket probes connect from the kubelet using the pod IP, not 127.0.0.1, so they always fail when dockerd is bound to localhost only. After failureThreshold (5 × 5s = 25s past initialDelay), kubelet kills dind — leaving docker0 in the network namespace and causing the "existing interface docker0 is not a bridge" error on the next restart. Switch to exec probe using `docker -H tcp://127.0.0.1:2375 version`, which runs inside the container and can reach the loopback address. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…erwrite Instead of appending gVisor-specific flags to the dockerd command, merge them into daemon.json via the ConfigMap template using mergeOverwrite. - Add dind.gVisor.daemonConfig with the gVisor defaults: storage-driver, iptables, ip6tables, features.containerd-snapshotter - ConfigMap deep-merges gVisor.daemonConfig into dind.daemonConfig when gVisor is enabled, so users can override individual keys if needed - Exec line is now identical for both gVisor and non-gVisor paths Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… body
Move the {{- if gVisor }} inside the command: block instead of
duplicating the whole block, eliminating the repeated dockerd
arguments.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tls, mtu, hosts, registry-mirrors are now in dind.daemonConfig so the command is just [dockerd] (non-gVisor) or [/bin/sh, -c, setup+exec dockerd] (gVisor). No flags passed on the command line. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
0aae929 to
365f89a
Compare
filiphaftek
left a comment
There was a problem hiding this comment.
@michaellzc the changes are very good and makes executore more mature, I only wonder about backward compatibility - I confirmed with DS that this might break existing executors, do we fix this for customers?
@filiphaftek I'm not aware of any customers currently using the DinD deployment method for Executors, and haven't found anything while searching for it in Slack. I'm not sure if we can decipher this via telemetry, or if a new telemetry field needs to be added. |
Extract the hardcoded capabilities list and shell setup script out of the template and into dind.gVisor.securityContext and dind.gVisor.setupScript, following the same pattern as daemonConfig. Both fields are fully overridable in user values files. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ommand Add dind.command (default: [dockerd]) and dind.gVisor.command (the gVisor shell wrapper) as plain YAML arrays in values. Template renders whichever is active via toYaml — no string interpolation needed. Removes the setupScript field. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
there is no backward compatibility concern. the chart has been broken from day one. for example, the executor deployment manifest has been missing |
Replace `docker version` with `wget -qO- http://127.0.0.1:2375/_ping` — lighter check against Docker's purpose-built ping endpoint, using busybox wget which is always present in the Alpine-based dind image. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
some suggestion has been addressed
|
The backport to To backport this PR manually, you can either: Via the sg toolUse the sg backport -r 7.3.x -p 876Via your terminalTo backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-7.3.x 7.3.x
# Navigate to the new working tree
cd .worktrees/backport-7.3.x
# Create a new branch
git switch --create backport-876-to-7.3.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 09972342271f0863de1ed98be81ca25327d86933
# Push it to GitHub
git push --set-upstream origin backport-876-to-7.3.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-7.3.xIf you encouter conflict, first resolve the conflict and stage all files, then run the commands below: git cherry-pick --continue
# Push it to GitHub
git push --set-upstream origin backport-876-to-7.3.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-7.3.x
Once the pull request has been created, please ensure the following:
|
…and gVisor sandbox support (#879) Backport 0997234 from #876 related sourcegraph/sourcegraph#12881 ## Summary ### Multi-queue & executor improvements - Add `queues` list to values: when set, renders one Deployment per queue instead of the single executor Deployment; each queue merges with global `executor.env` (queue overrides global) and supports `replicaCount` and `resources` overrides - Add structured `executor` fields (`frontendUrl`, `frontendPassword`, `queueName`/`queueNames`, `log`, etc.) with validation that blocks managed env vars from `env` - Per-queue `queueName`/`queueNames` fields (deployment name suffix vs queue env var are now independent) ### Security - Bind dockerd to `127.0.0.1:2375` instead of `0.0.0.0` — prevents unauthenticated Docker API access from other cluster pods ### gVisor support (`dind.gVisor.enabled: true`) Enables running executor pods on GKE node pools with `sandbox-type=gvisor`, providing stronger container-to-host isolation. See: https://gvisor.dev/docs/tutorials/docker-in-gvisor/ ## Test plan - [x] Deployed to dev cluster with both `batches` and `codeintel` queues - [x] Verified executors are running in Docker mode (no RBAC errors) - [x] Verified gVisor pods reach `2/2 Running` on `sandbox.gke.io/runtime=gvisor` nodes with 0 restarts - [x] `helm template` renders correctly for both multi-queue and single-deployment modes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Michael Lin <mlzc@hey.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
related https://github.com/sourcegraph/sourcegraph/pull/12881
Summary
Multi-queue & executor improvements
queueslist to values: when set, renders one Deployment per queue instead of the single executor Deployment; each queue merges with globalexecutor.env(queue overrides global) and supportsreplicaCountandresourcesoverridesexecutorfields (frontendUrl,frontendPassword,queueName/queueNames,log, etc.) with validation that blocks managed env vars fromenvqueueName/queueNamesfields (deployment name suffix vs queue env var are now independent)Security
127.0.0.1:2375instead of0.0.0.0— prevents unauthenticated Docker API access from other cluster podsgVisor support (
dind.gVisor.enabled: true)Enables running executor pods on GKE node pools with
sandbox-type=gvisor, providing stronger container-to-host isolation.See: https://gvisor.dev/docs/tutorials/docker-in-gvisor/
Test plan
batchesandcodeintelqueues2/2 Runningonsandbox.gke.io/runtime=gvisornodes with 0 restartshelm templaterenders correctly for both multi-queue and single-deployment modes🤖 Generated with Claude Code