Enterprise · Edge resilience

How edge resilience works

Your edge servers keep serving flags when the backend is down, start again after a restart, and tell you how old their flags are.

Builds on:Clients and edge server

Video coming soon

Before you start

You run one or more edge servers. Clients and edge server shows how to start one.
You know which team each edge serves. An edge serves the flags of the team that owns its client.
You plan how your apps behave when the backend is unreachable. This chapter tells you what keeps working and what does not.

The problem

FluxGate has a control plane and a data plane.

The backend is the control plane. It stores flags, users and approvals in PostgreSQL. You change flags there.
The edge server is the data plane. Your apps evaluate flags on an edge, over OFREP or /evaluate. The edge holds the flags of its team in memory and answers from that copy.

The edge refreshes its copy from a gRPC stream from the backend. A backend outage, a database failure behind the backend, a network split or a deploy of the backend all close that stream. Edge resilience makes sure your apps keep getting answers while the stream is gone, and that an edge that restarts during an outage comes back with flags.

How flags reach an edge

When the backend is up, every edge follows the same cycle. On start and on every reconnect, the edge asks the backend which team its client belongs to, opens the stream with its client ID and API key, and receives one message per feature of the team, followed by a snapshot-complete marker. The edge then drops any cached feature that the snapshot did not contain.

After that, every saved change follows one path: the backend commits the change to PostgreSQL first, then pushes the full updated feature to every edge of the team. Each edge replaces the feature in its cache and clears cached results for it.

1 · Connect on start and on every reconnect

Edge server to Backend
Look up the client’s team
which team does this client belong to?
Edge server to Backend
Open the stream
signs in with the client ID and API key
Backend to Edge server
Snapshot of every team feature
one message per feature
Backend to Edge server
Snapshot complete
the edge drops features that are gone

2 · Live change every saved change

Admin UI to Backend
Save a change
for example criteria, stage, approval, delete
Backend
Commit to PostgreSQL
before anything is pushed
Backend to Edge server
Push the full feature
to every edge of the team
Edge server
Update the cache
replace it, clear cached results

The next evaluation on that edge uses the new rules, and OFREP responses carry a new ETag. The edge sends a heartbeat every 30 seconds and the backend answers it.

If the stream drops

The edge keeps answering from its last copy and stays ready while it holds flags. It reconnects after 1 second, doubling the wait up to 30 seconds, and loads a fresh snapshot. See Edge resilience.

If an edge falls behind

When an edge cannot keep up with the changes, the backend tells it so. The edge keeps its copy, reconnects and loads a full snapshot, which then removes the flags the backend no longer has.

Edge resilience adds what happens when this cycle breaks.

What you get

Capability	What it does	Edition
Ready during an outage	The edge starts and stays ready while the backend is unreachable, as long as it holds flags.	Core
Stale client credentials	The edge accepts SDK keys it verified earlier, for `client_stale_if_error_secs` (default 86400, 24 hours). `0` turns this off.	Core
Local snapshot	The edge writes its flags to disk and reads them at startup.	Core
Config source and age	Every response, the health endpoints and the metrics show where the flags came from and how old they are.	Core
gRPC TLS	The link between edge and backend is encrypted.	Core
Fleet view	Settings, Edge fleet lists every edge with its role, config source and age.	Core
Signed snapshot export and bootstrap	A new edge with an empty disk starts from a file that the backend signed.	Enterprise
Relay edges	One edge serves other edges, so a region needs one stream to the backend.	Enterprise

Enterprise features need the edge_resilience entitlement in your license. Only two actions check it: creating the first snapshot signing key, and registering a relay. An edge never checks a license. No gRPC method checks one, and neither does the export. A license change, a downgrade or a lapsed license can never stop an edge from serving flags.

The states of an edge

An edge is always in one of six states. Two facts describe each state: whether the edge is ready (its readiness probe answers 200), and its config source, the place its flags came from.

Starting
No flags, not ready (503). A valid local file, a complete stream snapshot or an accepted bootstrap file moves it on.
Serving a local file
Ready, source local. A newer signed bootstrap file or the stream replaces it.
Serving a bootstrap file
Ready, source bootstrap. Read only when the stream has not delivered after startup_wait_secs and is disconnected.
Live on the stream
Ready, source stream, age 0. Every saved change arrives here.
Stream lost
Still ready. Serves its cache, the age grows. A reconnect with a full snapshot makes it live again.
Too old
Only with ready_max_age_secs: not ready (503) once the age passes the limit, until the backend is back.

The stream always wins: whatever the edge serves, a complete stream snapshot replaces it and the source becomes stream. Readiness depends on holding flags, not on the stream, so a backend outage does not take edges out of a load balancer unless you set ready_max_age_secs.

State	Ready	`FluxGate-Config-Source`	`FluxGate-Config-Age`
Starting	No, `503`	none (header absent)	absent
Serving a local file	Yes	`local`	seconds since the file was written
Serving a bootstrap file	Yes	`bootstrap`	seconds since the backend generated the export
Live on the stream	Yes	`stream`	`0`
Stream lost, serving its cache	Yes	unchanged	grows every second
Too old (only with `ready_max_age_secs`)	No, `503`	unchanged	above the limit

Three rules decide every move between these states:

The stream always wins. A complete stream snapshot replaces a local or bootstrap file, and the source becomes stream. A file never replaces flags that a stream is still delivering.
Readiness follows flags, not the stream. An edge that holds a complete set of flags is ready, with or without a backend. This keeps edges in your load balancer during an outage. Set ready_max_age_secs only if you want old edges removed.
Old never replaces new. An edge refuses a bootstrap file that is not newer than what it already serves. This is the rollback floor, described in Signed bootstrap snapshots.

What an edge does at startup

Startup is where the states matter most. A new edge, or an edge that restarts during an outage, walks through these steps.

1
Read the configuration
The edge checks trusted_keys before anything else. A malformed key, a key ID that does not match its key, a duplicate, or bootstrap without keys stops the edge here.
2
Is there a usable local file?
<snapshot.dir>/local-<client_id>.fgs, signed with a key derived from the edge's own client secret.
YesFill the cache from it before HTTP starts. Ready, source local.
NoStart HTTP with an empty cache. Not ready yet. A damaged or foreign file logs one error line and is ignored.
3
Open the stream in the background
Look up the client's team, then subscribe. Reconnects wait 1 second, doubling up to 30 seconds ([retry]).
4
Did the stream deliver a snapshot within startup_wait_secs?
Default 5 seconds.
YesSource stream, age 0. The bootstrap file is never read.
NoGo on, but only while the stream is disconnected. A stream that is open and still sending its snapshot is never overwritten.
5
Fetch the bootstrap file
A path, or an https:// URL: no redirects, 10 second timeout, at most 64 MiB, an HTML answer is refused.
6
Check the file, in this order
1. signed with Ed25519 by a key in trusted_keys
2. names a team and a generation time
3. same team as the edge already knows
4. newer than the rollback floor (what the edge last held)
5. not older than max_age_secs, when set
6. a new edge only: lists the edge's own client with the hash of its key
YesServe it at once. Source bootstrap, age from the export time. With snapshot.dir, keep it as the local file.
NoLog Ignoring bootstrap snapshot <source>: <reason>. With nothing loaded, try again after 5, 10, 20, 40 seconds, then every 60 seconds.
7
The stream comes back
The complete stream snapshot replaces whatever the edge serves, removes flags the backend no longer has, and the source becomes stream.

The bootstrap runs once after startup. A later outage does not read it again: the edge then serves its cache, and after a restart its local file.

Config source and config age

Every evaluation response carries two headers:

FluxGate-Config-Source: stream, local or bootstrap.
FluxGate-Config-Age: how old the served flags are, in whole seconds.

The age is 0 while the stream that delivered the flags is open. When the stream closes, the age counts from the last sync. For a file, it counts from the time the file was generated, so a bootstrap exported a week ago reports about 604800. The timeline below shows one outage with an edge restart in the middle.

Backend up
Source stream, age 0.
Backend down
The edge keeps serving. Source stays stream, the age grows.
Edge restarts during the outage
It loads its local file. Source local, the age counts from the file time.
Backend back
A full snapshot arrives. Source stream, age 0, queued events flush.
Optional limit
With ready_max_age_secs, an edge whose age passes the limit reports 503.

Every evaluation response carries the two values as headers, FluxGate-Config-Source and FluxGate-Config-Age. The same values are in /health/detail and /metrics. Alert on the age, not on the health status code.

On a child of a relay edge, both values describe the relay: the source is the relay's source, and the age is the relay's age plus the child's own time since its stream to the relay closed. A child is never shown as fresher than its relay. See Relay edges.

What keeps working during an outage

Edges serve the flags in their cache, whether those came from the stream, a local file or a bootstrap file.
Edges restart and keep serving when they have a local file or an accepted bootstrap file.
SDK keys that the edge verified earlier, or that are in an applied bootstrap file, keep working for client_stale_if_error_secs.
Relays keep serving their children from their caches.
Evaluation events and sticky assignments queue on the edge in bounded queues and flush when the backend is back.

A database failure behind a running backend counts as an outage too. The backend then answers client lookups with UNAVAILABLE, and the edge treats it like an unreachable backend.

What does not work during an outage

An outage keeps your apps running on the last known state. It cannot deliver new state.

Flag changes and kill switches do not reach edges. You cannot fire a kill switch while the backend is down. A change made just before the outage may not have reached every edge.
A client you disable, or an SDK key you rotate, keeps working on edges until the backend is back or the stale-credential window ends. In a security incident, block the client at the network layer, and revoke it when the backend returns.
A flag that is not in an edge's cache returns 502.
An unknown client or a wrong SDK key returns 502. A key the backend rejected is never accepted later.
A new edge with no local file is not ready without a bootstrap file that lists its own client.

When the backend returns, the stream sends a complete snapshot. The source returns to stream, the age drops to 0, the edge removes flags the backend no longer has, and the queued events flush.

License and downgrade

Item	After a downgrade to Business or Free, or in ReadOnly
Edges serving from the stream, a local file or a bootstrap file	Always, in every license state
Existing signing keys	Keep signing exports
Snapshot export	Keeps working while a key is active
Rotate a key	Allowed after a downgrade. Blocked in ReadOnly with `403 license_read_only`
Retire a key	Allowed in every state, because it only takes trust away
Create the first key	Free or Business: `403 license_required`. ReadOnly: `403 license_read_only`
Registered relays	Keep relaying
Register a relay	`403 license_required`, or `403 license_read_only` in ReadOnly
Unregister a relay, list keys, relays and the fleet	Allowed in every state

The pages in this chapter

Local snapshots: the Core step every edge should have.
Signed bootstrap snapshots: start an empty edge from a signed file.
Relay edges: one stream per region.
TLS and multi-region patterns: encrypt the edge link and choose a topology.
Monitoring and outages: headers, health, metrics, the fleet view, the runbook and the drill.

Start with the Core step on every edge: Local snapshots.