Relay edges

Before you start

• Your license has the edge_resilience entitlement (Enterprise). Only registering a relay checks it.
• You can sign in as admin System admin.
• One client per relay, created in the team the relay serves. In the story, the Checkout team has the client edge-relay-eu.
• A TLS certificate for the relay's gRPC listener, or a test setup where plaintext is acceptable.

What a relay is

A relay is an edge server that keeps one stream to the backend and also serves the edge gRPC API to other edges, its children. The children point their backend_grpc at the relay instead of the backend. They receive their flags, client lookups and the update stream from the relay, exactly as they would from the backend.

Use relays in a region or network where many edges would otherwise each open a stream to the backend: a second cloud region, an on-premises site, or a network where only one host may reach the backend.

Register the relay

The backend serves a relay's children only after an admin registers the relay.

(opens full size in a new tab) (opens full size in a new tab)

The REST API does the same:

Errors: 403 license_required without the entitlement, 404 for an unknown client, 400 for an invalid node ID, 409 relay_exists for a relay that is already registered. Registering and unregistering write the audit entries edge_relay_registered and edge_relay_removed.

Run the relay

A relay needs a stable node_id, because registration names the relay by it, and TLS on its gRPC listener.

backend_grpc = "https://backend.junipermarket.example:50051"
http_addr = "0.0.0.0:8081"
client_id = "<edge-relay-eu client ID>"
client_secret = "<edge-relay-eu API key>"
node_id = "relay-eu-1"

[relay]
enabled = true
grpc_addr = "0.0.0.0:50061"
tls_cert_file = "/etc/fluxgate/relay.crt"
tls_key_file = "/etc/fluxgate/relay.key"

[snapshot]
dir = "/var/lib/fluxgate/edge"

The relay stops at startup, with a message that names the setting, when it has no TLS files and allow_insecure is false, when its node ID is random, when serve_http is false without enabled, when max_children is 0 or above 50000, or when it cannot bind the gRPC address.

Set snapshot.dir on every relay. A relay that restarts while the backend is down uses the registration status saved in its local file, but only if the file's node_id equals its own. Without the folder, the relay is not registered until the backend answers, and its children wait.

Point the children at the relay

backend_grpc = "https://relay-eu.junipermarket.example:50061"
client_id = "<checkout-worker client ID>"
client_secret = "<checkout-worker API key>"
node_id = "worker-eu-1"

[grpc]
tls_ca_file = "/etc/fluxgate/ca.pem"

[snapshot]
dir = "/var/lib/fluxgate/edge"

Set [grpc] tls_ca_file when the relay certificate comes from a private CA. An empty value trusts the built-in web PKI roots, not the operating system store.

Make it highly available

A child that loses its relay reconnects through the balancer to the other one, and serves flags from its cache meanwhile. The backend sees two streams for the whole region.

What a child gets from its relay

• Flags. On connect, a snapshot of the relay's cached flags, then every update. When the relay itself resyncs or applies a bootstrap file, the children converge without reconnecting.
• Config source and age. The relay sends its own config source and age with the snapshot and with every heartbeat reply. The child reports the relay's source and adds its own lag to the relay's age. A child of a relay that serves a 10-minute-old local file reports local and about 600, although its own stream is open. The headers, /health/detail, /metrics, ready_max_age_secs and the child's local file all use these values.
• Client checks. The relay checks each child's credentials with the backend again every 60 seconds, bypassing its own cache. A child whose key was rotated, or whose client was disabled or deleted, has its stream closed with UNAUTHENTICATED within 60 seconds. A client that moved to another team gets FAILED_PRECONDITION. While the backend is unreachable, streams stay open.
• Stale keys behind a relay. A child accepts a stale relay answer only within its own client_stale_if_error_secs, counted from the time the backend verified the key. The relay sends that time with each answer, and the child keeps it across restarts. Once its stream to the backend is back, the relay checks a key it served stale with the backend again before it answers a child. Keep the child's window well above the relay's cache.client_ttl_secs.
• Ingest. Evaluation events, sticky assignments and metrics pass through the relay to the backend, unbuffered. When the backend is unreachable, the relay answers UNAVAILABLE, and the child keeps the items in its own bounded queues.
• Lookups during an outage. A child's client lookup costs one attempt on the relay. The relay marks a failed lookup as already retried, and the child does not retry it: it uses its last good lookup, or answers 502 at once.

What a relay refuses

In every case the child keeps its cache and, with snapshot.dir, its local file, and retries with backoff.

See the relay and its children

Every relay reports its open child streams to the backend every 60 seconds: client, node ID, address, version, config source and age of each. The Fleet tab shows each child as an edge whose Via column names the relay's node key.

(opens full size in a new tab) (opens full size in a new tab)

On the relay itself, GET /health/detail has a relay block, and /metrics has fluxgate_edge_relay_children:

{
  "relay": {
    "enabled": true,
    "registered": true,
    "children": 2,
    "listen": "0.0.0.0:50061",
    "servesHttp": true
  }
}

A report with more than 10000 children, or larger than 2 MiB, goes in parts 15 seconds apart. The backend records children only of the relay's team. The fleet list shows up to 5000 nodes: relays and directly connected edges first, then relay children.

Unregister a relay

Click Remove on the Relays tab, or send the DELETE request. Unregistering works after a downgrade and in ReadOnly, because it only takes access away.

A change in the registry reaches a running relay within about a minute. The relay then closes its child streams with FAILED_PRECONDITION and refuses new ones. The children fail over through their normal reconnect, for example to the second relay, and serve from their caches meanwhile. The relay keeps running and keeps serving its own HTTP traffic. The backend stops recording its children, so they age out of the 5-minute connected window of the fleet view.

Licensing

Each child counts as an edge node. A pure relay (serve_http = false) does not count once it is registered, because it serves no evaluations; until it is registered it counts. Registered relays keep relaying after a downgrade; only new registrations need the entitlement.

Troubleshooting

Children cannot connect and the relay logs nothing about them

The relay is not registered, or the registration has not reached it yet. Check the Relays tab and the relay's /health/detail: registered must be true. A registration reaches a running relay within about a minute.

A child gets "relay serves another team"

The child's client is in another team than the relay's client. Create the child's client in the relay's team, or run a relay per team.

The relay stops at startup

Read the message: it names the setting. Usually the relay has no TLS files (set tls_cert_file and tls_key_file, or allow_insecure = true for a test), or no stable node_id.

Next

Encrypt the edge link and choose a topology per region: TLS and multi-region patterns.