Help us shape the future of Go! Take the 2026 Go Developer Survey

hotrestart

package
v0.0.0-...-9b256ed Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 27, 2026 License: Apache-2.0 Imports: 39 Imported by: 0

Documentation

Overview

Package hotrestart implements a supervisor that manages the aether-proxy Envoy process and performs Envoy hot restarts across restart epochs, replicating the behavior of Envoy's hot-restarter.py in Go (see docs/proposals/001_proxy-hot-restart.md).

Model: the supervisor is the proxy container's entrypoint (PID 1). It forks an Envoy child with --restart-epoch 0 and a fixed --base-id. On a hot-restart trigger (a watched bootstrap-config change or SIGHUP) it forks a new Envoy with the next epoch; Envoy's own shared-memory + abstract-domain-socket IPC transfers the listen-socket FDs and stats to the new process, the old process drains, and after --parent-shutdown-time-s the supervisor terminates it.

The handoff also works ACROSS the pod boundary during a surge upgrade: the overlapping old and new aether-proxy pods share the node's network namespace, /dev/shm (a hostPath) and the same --base-id, so the new pod's Envoy hot-restarts from the old pod's. The supervisors coordinate the per-node restart epoch through a heartbeat file on the shared StateDir and gate pod readiness (ReadyMarkerPath) so the DaemonSet keeps the predecessor until the successor has taken over.

Index

Constants

View Source
const (
	// DefaultStallSampleInterval is the --stall-sample-interval default: the
	// resolution of the state/wchan sampling (schedstat is cumulative, so CPU
	// and runqueue time are exact regardless). Per tick it is two or three
	// small file reads per sampled thread — five threads per epoch at Envoy's
	// default --concurrency on the 4-core reference-cluster workers.
	DefaultStallSampleInterval = 100 * time.Millisecond
	// DefaultStallThreshold is the --stall-threshold default, matching Envoy's
	// default worker watchdog miss_timeout.
	DefaultStallThreshold = 200 * time.Millisecond
)
View Source
const (
	// DefaultStallTopConsumers is how many consumers a [starved] line names per
	// source (--stall-top-consumers).
	DefaultStallTopConsumers = 5
)
View Source
const RestartEpochEnv = "AETHER_RESTART_EPOCH"

RestartEpochEnv is the environment variable every Envoy child is started with: the --restart-epoch value of that child, the same number Envoy exports as server.hot_restart_epoch. It is there for an Envoy format string to read: %ENVIRONMENT(AETHER_RESTART_EPOCH)% in an access-log format makes each row name the generation that wrote it (issue #1333). During a hot-restart overlap that is the only per-generation signal available: the parent stops flushing stats to sinks the moment the child starts.

The exported number is always the child's effective epoch. --restart-epoch cannot be given a second time through ExtraArgs to disagree with it: Envoy refuses a repeated flag ("Argument already set!"), so such a child would never start, and CheckExtraArgs refuses that ExtraArgs before the first fork (issue #1376).

Variables

This section is empty.

Functions

func CheckExtraArgs

func CheckExtraArgs(args []string) error

CheckExtraArgs refuses a Config.ExtraArgs (the supervisor's --envoy-arg) that cannot give a working Envoy. Either the pinned Envoy rejects the command line on every fork, or it starts and the next handoff breaks.

The list is read the way the pinned Envoy reads it (parseEnvoyArgs, issue #1443): a flag, then its value when the flag takes one. So a value is never mistaken for a flag (a --service-node named "-c" is a node name), and a flag is found wherever it stands and however it is spelled ("--socket-path @x" in one item included). Refused, in this order:

  • a flag in reservedEnvoyFlags: one the supervisor passes itself, or one that defeats what the supervisor controls (issues #1376, #1409);
  • any flag given more than once, except --stats-tag (#1375). The flags the chart passes (onceEnvoyFlags) are therefore allowed once and no more;
  • then the first argument, in order, that Envoy does not take: a spelling it does not parse ("--flag=value", "-f=value", "-fvalue"; #1407), a flag it does not have, an argument that is no flag and no flag's value, a flag whose value is missing, and a --concurrency value it refuses or reads as a count the supervisor cannot know (#1408).

It is meant to run once, before the first fork, so the mistake is a startup error naming the argument instead of a supervisor that fails every fork.

What it does not check is what a value means: a log level Envoy does not know, a number that is not one for a flag other than --concurrency. Those still fail at the fork.

func EnvoyDefaultConcurrency

func EnvoyDefaultConcurrency(environ []string) (int, error)

EnvoyDefaultConcurrency is the worker count the pinned Envoy runs when it is started without --concurrency by this process, with environ as its environment. See the comment at the top of this file for the rule.

Types

type Config

type Config struct {
	// EnvoyPath is the path to the Envoy binary.
	EnvoyPath string
	// ConfigPath is the Envoy bootstrap config (-c). A change to this file (when
	// WatchConfig is set) triggers a hot restart.
	ConfigPath string
	// BaseID is Envoy's --base-id, pinned so successive epochs find the same
	// shared-memory segment. Must be stable for the life of the container.
	BaseID uint32
	// DrainTime maps to Envoy --drain-time-s: how long the draining (old) epoch
	// takes to gracefully close connections.
	DrainTime time.Duration
	// ParentShutdownTime maps to Envoy --parent-shutdown-time-s and gates when the
	// supervisor SIGTERMs the previous epoch. Must exceed DrainTime. It also sets
	// the admin re-verify budget: the epoch-identity probe re-confirms on a fresh
	// connection every ParentShutdownTime/adminReverifyDivisor, so a cross-pod
	// takeover is diagnosed while the draining parent still lives (see
	// adminprobe.go).
	ParentShutdownTime time.Duration
	// ExtraArgs are appended to every Envoy invocation (e.g. -l, --service-cluster,
	// --service-node, --service-zone, --concurrency). A --concurrency that differs
	// from a live predecessor's is not hot-restarted: the predecessor is drained
	// and a fresh lineage started (issue #1136, concurrencychange.go).
	ExtraArgs []string
	// WatchConfig enables an fsnotify watch on ConfigPath's directory that
	// self-triggers a hot restart when the bootstrap config changes (e.g. a
	// ConfigMap update propagated by the kubelet).
	WatchConfig bool
	// StateDir is the shared-hostPath dir holding the per-node epoch heartbeat
	// file. A surging successor pod reads it to start at (live predecessor epoch +
	// 1) and hot-restart across the pod boundary. Required.
	StateDir string
	// ReadyMarkerPath is the pod-local readiness marker the supervisor keeps
	// present only while the node's Envoy admin reports LIVE at this supervisor's
	// newest epoch. An exec readiness probe checks the marker so the DaemonSet
	// keeps the old pod until the new one has taken over. Required.
	ReadyMarkerPath string
	// AdminAddress is the Envoy admin host:port used for the readiness check.
	// The supervisor probes it two ways: /ready on a pooled connection for the
	// per-second liveness watchdog, and /server_info on a fresh connection
	// whenever the answer must identify the epoch (see adminprobe.go).
	AdminAddress string
	// HandoffDeadline overrides defaultHandoffDeadline (0 = default). It is
	// measured from the successor's fork or from its admin's last answer at its
	// own epoch, whichever is later (issue #1085): a successor still answering
	// is waiting on xDS, not wedged. Must be comfortably larger than the
	// successor's worst-case admin silence while it loads its first batch.
	HandoffDeadline time.Duration
	// AdminUnresponsiveDeadline overrides defaultAdminUnresponsiveDeadline
	// (0 = default).
	AdminUnresponsiveDeadline time.Duration
	// TerminationGrace is this pod's terminationGracePeriodSeconds, i.e. how
	// long after SIGTERM the kubelet SIGKILLs the container. The supervisor
	// cannot read it from the API, so the chart passes its own value. It bounds
	// the mid-handoff wait for a successor so a termination with no possible
	// successor still drains Envoy (see successorWaitBudget, issue #771).
	// 0 = unknown: wait indefinitely, the pre-#771 behavior.
	TerminationGrace time.Duration
	// ShutdownDrainImmediately skips the bounded wait for a surge successor on
	// SIGTERM and drains Envoy's listeners straight away (issue #795). It is an
	// escape hatch for a deployment where no replacement can overlap this pod —
	// a DaemonSet without maxSurge, a single-instance proxy — where the wait can
	// only ever burn the grace period before falling back to the same drain.
	//
	// Leave it false wherever a surge replacement exists: the wait is what makes
	// pod delete, node drain, eviction and preemption hitless, because the
	// replacement's Envoy hot-restarts ours instead of finding the node empty.
	ShutdownDrainImmediately bool
	// StallSampleInterval enables the Envoy thread-stall sampler (issue #1093,
	// stallsampler.go) at this sampling interval; 0 disables it.
	StallSampleInterval time.Duration
	// StallThreshold is how long a thread may wait for a CPU, stay blocked, or
	// stay on a CPU within a one-second window before the sampler reports it
	// (0 = DefaultStallThreshold, Envoy's worker watchdog miss threshold).
	StallThreshold time.Duration
	// StallTopConsumers is how many CPU consumers (cgroups, and processes
	// where /proc shows the node's) a [starved] stall line names (#1392,
	// cpuconsumers.go), at most 20. 0 turns the consumer sampling off.
	StallTopConsumers int
	// PodName is this proxy pod's name (optional). It is folded into the
	// per-supervisor admin identity (see adminidentity.go) only so a log line
	// about a FOREIGN Envoy answering the node-shared admin address can name
	// the pod it belongs to; the identity's uniqueness comes from a random
	// nonce, not from this.
	PodName string
	// HotRestartOnConcurrencyChange forces a hot restart from a live
	// predecessor whose Envoy worker count differs from the one the next Envoy
	// would run with. By default (false) such a predecessor is drained and a
	// fresh Envoy lineage started instead, because a hot restart across a
	// worker-count change resets about half of the predecessor's live QUIC
	// connections (issue #1136, concurrencychange.go).
	HotRestartOnConcurrencyChange bool
}

Config configures the Envoy hot-restart supervisor.

type EnvoyFlag

type EnvoyFlag struct {
	// Long is the flag with its two dashes: "--log-level".
	Long string
	// Short is the one-letter spelling with its dash ("-l"), or empty. Envoy
	// counts a short and a long spelling as the same flag.
	Short string
	Kind  EnvoyFlagKind
}

EnvoyFlag is one command-line flag of the pinned Envoy.

func EnvoyFlags

func EnvoyFlags() []EnvoyFlag

EnvoyFlags returns the pinned Envoy's command-line flags as the --envoy-arg check knows them. //agent/test/envoyargs holds the list against the binary.

type EnvoyFlagKind

type EnvoyFlagKind int

EnvoyFlagKind is how the pinned Envoy's command-line parser reads one flag.

const (
	// EnvoySwitch takes no value (a TCLAP::SwitchArg): the flag is one
	// argument, and the argument after it is the next flag.
	EnvoySwitch EnvoyFlagKind = iota
	// EnvoyValue takes one value and may be given once (a TCLAP::ValueArg).
	// The value is the next argument, whatever it looks like.
	EnvoyValue
	// EnvoyMultiValue takes one value per occurrence and may be repeated (a
	// TCLAP::MultiArg). --stats-tag is the only one.
	EnvoyMultiValue
)

type Supervisor

type Supervisor struct {
	// contains filtered or unexported fields
}

Supervisor owns the Envoy process lifecycle and performs hot restarts.

func New

func New(cfg Config, log *slog.Logger, metrics *SupervisorMetrics) *Supervisor

New creates a Supervisor. metrics may be nil to disable instrumentation.

func (*Supervisor) Run

func (s *Supervisor) Run(ctx context.Context) error

Run starts Envoy at epoch 0 and supervises it until ctx is canceled (SIGTERM/ SIGINT, which controller-runtime's signal handler maps to ctx.Done) or the newest epoch exits unexpectedly. A watched-config change or SIGHUP triggers a hot restart; SIGUSR1 is forwarded to the current child for log reopen.

type SupervisorMetrics

type SupervisorMetrics struct {
	// contains filtered or unexported fields
}

SupervisorMetrics holds the hot-restart lifecycle instruments. All methods are nil-receiver-safe so the supervisor runs unchanged without telemetry.

These exist for post-mortems of wedged or crashed proxy pods: the epoch gauge and handoff histogram show how far a handoff got and how long it took, the wedge counter says why the watchdog killed the pod, and the child-exit counter distinguishes expected drains from crashes.

func NewSupervisorMetrics

func NewSupervisorMetrics(meter metric.Meter) (*SupervisorMetrics, error)

NewSupervisorMetrics registers the supervisor instruments on the given meter.

type Telemetry

type Telemetry struct {
	// contains filtered or unexported fields
}

Telemetry is the supervisor's metrics pipeline: a plain OTel SDK MeterProvider pushing to the collector over OTLP. No Prometheus registry, no scrape endpoint.

func NewTelemetry

func NewTelemetry(ctx context.Context, cfg TelemetryConfig) (*Telemetry, error)

NewTelemetry builds the supervisor metrics pipeline.

func (*Telemetry) Meter

func (t *Telemetry) Meter() metric.Meter

Meter returns the meter to register supervisor instruments on.

func (*Telemetry) Shutdown

func (t *Telemetry) Shutdown() error

Shutdown flushes and stops the provider. Called on every supervisor exit path — including the wedge watchdog's fatal exit, where the final flush is the crash forensics.

type TelemetryConfig

type TelemetryConfig struct {
	// OTLPEndpoint is the OTLP gRPC collector metrics are pushed to
	// (host:port, insecure). Required: the supervisor is push-only — it runs
	// outside the controller-runtime manager (no shared Prometheus registry to
	// bridge into) and in the host netns, where a scrape port would collide
	// between surge predecessor and successor pods.
	OTLPEndpoint string
	// ServiceVersion is the build version recorded on the OTel resource.
	ServiceVersion string
}

TelemetryConfig configures the supervisor's metrics pipeline.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL