Local snapshots

Before you start

• An edge server that reaches the backend, as in Clients and edge server.
• A folder on persistent storage that only the edge process can read. In Docker or Kubernetes, that is a volume, not the container file system.
• Local snapshots are Core. They need no license.

Why a local file

Without a file, an edge that restarts while the backend is down starts with an empty cache. It is not ready, and every evaluation fails until the backend is back. That turns a backend outage plus a routine restart (a node drain, a crash, a deploy of the edge) into an outage of your apps.

With a local snapshot, the edge reads the flags it last had before it starts serving HTTP. It is ready at once with the config source local, and it switches to stream as soon as the backend is back.

Turn it on

Set a snapshot folder on every edge:

[snapshot]
dir = "/var/lib/fluxgate/edge"

Or in the environment:

export EDGE_SNAPSHOT__DIR=/var/lib/fluxgate/edge

The edge creates the folder when it is missing. An empty value turns the local snapshot off.

What the edge does with the file

Because the edge writes the file when its stream ends, a backend outage does not lose the last changes the edge received. An edge that is stopped or crashes within write_interval_secs of a change can lose that one change from the file.

What the file holds

The file holds at most cache.max_capacity flags (default 10000). When the cache is full, the edge logs a warn at most once per write_interval_secs, because the file may then miss flags. Raise cache.max_capacity if you see it.

With client_stale_if_error_secs = 0, the file holds no client credentials.

The file is signed with the edge's own secret

The edge signs the file with a key derived from its own client secret. This stops another edge, or another team, from loading a file that was not written for it.

The edge ignores a file it cannot use and logs one error line that names the file and the reason. It then starts as if no file existed. The causes are:

• The signature does not match, usually because the client secret was rotated.
• The file is damaged.
• The file was written for another client or another team.

Client keys during an outage

Each request authenticates its client: the edge checks the SDK key with the backend and caches the answer for cache.client_ttl_secs (default 300). During an outage the edge cannot ask, so it uses what it verified earlier.

• A key that the backend accepted before is accepted again for resilience.client_stale_if_error_secs after that check (default 86400). With a local file, this also holds after a restart, because the file stores the digests.
• Near the end of the window, each request asks the backend again first.
• Only transient errors use the last good answer: UNAVAILABLE, DEADLINE_EXCEEDED and the other retried codes. A backend whose database fails answers UNAVAILABLE, so that counts too.
• An unknown client or a wrong key gets 502. A key the backend rejected is never accepted this way, even when the backend becomes unreachable later.
• client_stale_if_error_secs = 0 turns this off: the edge then accepts only keys it can check live.

[resilience]
client_stale_if_error_secs = 86400

The consequence: a client that you disable, or a key that you rotate, during an outage keeps working until the backend is back or the window ends. In a security incident, block the client at the network layer as well.

Run it in containers

The file must survive a restart of the container. Mount a volume on the snapshot folder.

services:
  edge:
    image: forgeopslabs/flux-gate-edge:v2.0.1
    environment:
      EDGE_SNAPSHOT__DIR: /var/lib/fluxgate/edge
    volumes:
      - edge-snapshot:/var/lib/fluxgate/edge

volumes:
  edge-snapshot:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: fluxgate-edge
spec:
  serviceName: fluxgate-edge
  replicas: 2
  selector:
    matchLabels: { app: fluxgate-edge }
  template:
    metadata:
      labels: { app: fluxgate-edge }
    spec:
      containers:
        - name: edge
          image: forgeopslabs/flux-gate-edge:v2.0.1
          env:
            - name: EDGE_SNAPSHOT__DIR
              value: /var/lib/fluxgate/edge
          readinessProbe:
            httpGet: { path: /health/ready, port: 8081 }
          livenessProbe:
            httpGet: { path: /health/live, port: 8081 }
          volumeMounts:
            - name: snapshot
              mountPath: /var/lib/fluxgate/edge
  volumeClaimTemplates:
    - metadata:
        name: snapshot
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests:
            storage: 1Gi

Each edge needs its own folder or volume. The file name contains the client ID, so two edges with different clients can share a folder, but two edges with the same client would overwrite each other's file.

Use /health/ready as the readiness probe and /health/live as the liveness probe. An edge stays ready while it holds flags, so a backend outage does not restart or remove your edges.

Keep the file confidential

The file holds your flag configuration, including targeting rules, and those rules can contain user IDs and email addresses.

• On Unix the edge creates the file with mode 0600.
• Keep snapshot.dir on a volume that only the edge process can read.
• Leave the folder out of shared backups.

Check it

Run this once per environment:

curl -s http://localhost:8081/health/detail | jq '{status, streamConnected, configSource, configAgeSecs}'

{
  "status": "ready",
  "streamConnected": false,
  "configSource": "local",
  "configAgeSecs": 184
}

Good to know

• A local file covers a restart. It does not help a new edge with an empty disk. For that, use a signed bootstrap snapshot.
• A file written from an accepted bootstrap keeps the source bootstrap after a restart, and keeps the exported client hashes, so a restart during the same outage still authenticates them.
• When the stream writes the file, the edge drops exported client hashes from it. A restart in a later outage accepts only keys that the edge verified itself.
• A relay keeps its registration status in its file, but only when the file's node_id equals the relay's. Without snapshot.dir, a restarted relay is not registered until the backend answers, and its children wait.

Troubleshooting

The edge is not ready after a restart

The edge found no usable file and the backend is unreachable. Check the edge log for the single error line that names the file and the reason. A rotated client secret is the most common cause. Until the backend is back, a bootstrap file is the only way to start this edge.

The edge serves fewer flags after a restart

The cache was full when the file was written. Look for the warn about the cache, and raise cache.max_capacity.

Next

Start an edge that has no file at all: Signed bootstrap snapshots.