Skip to content

Production Hardening

beryl ships with all rate and connection limits disabled, because no default is right for every deployment. That is fine for development, but a production server with no abuse controls can be degraded by a single hostile (or buggy) client. beryl logs a warning at startup when every control is off; this guide explains what to turn on and why.

Even with no configuration, beryl enforces:

  • Frame size: inbound WebSocket frames over 1 MiB close the connection (with_max_inbound_frame_bytes to adjust).
  • Topic and event lengths: topics over 256 bytes and event names over 64 bytes are rejected before reaching a channel (with_max_topic_length, with_max_event_length).
  • Joined-topic cap: a socket may join at most 1000 topics (with_max_joined_topics_per_socket).
  • Protocol hygiene: reserved phx_* events, reserved beryl:* topics, and messages carrying a stale join_ref are dropped; heartbeat and leave frames count against the message rate limit when one is configured.
  • Heartbeat eviction: sockets that stop sending heartbeats are evicted and their connections closed (60 s window by default, with_heartbeat).
let config =
beryl.config(wire.phoenix_codec())
// Cap messages per socket. Size to your chattiest legitimate client:
// for most interactive apps 50-100 msg/s with 2x burst is generous.
|> beryl.with_message_rate(per_second: 100, burst: 200)
// Joins are much rarer than messages. 10/s per socket tolerates
// aggressive reconnect/rejoin loops while stopping join floods.
|> beryl.with_join_rate(per_second: 10, burst: 20)
// Concurrent connections per client IP. Size to your expected
// clients-behind-one-NAT worst case; see the caveat below.
|> beryl.with_max_connections_per_ip(max_connections: 100)
// Node-wide ceiling on concurrent connections across all IPs. Size to a
// single node's process/socket/coordinator budget; see below.
|> beryl.with_max_connections(max_connections: 10_000)

Optionally, with_channel_rate adds a per-socket-per-topic limit on top of the global per-socket message rate, useful when a single busy topic must not starve others.

with_max_connections_per_ip throttles a single abusive peer, while with_max_connections caps concurrent connections across the whole node. When both are set a connection must be under both ceilings to be admitted; otherwise the transport rejects it with 429 before allocating any long-lived channel or coordinator state. Freed capacity is reclaimed on normal close, transport failure, heartbeat eviction, crash, and setup failure.

The node-wide ceiling exists because a per-IP limit alone cannot stop many distinct source addresses — a botnet, or a single host rotating through an IPv6 range — from each opening a few connections and collectively exhausting the node's process, socket, and coordinator budget. The global ceiling bounds that total regardless of how the connections are spread across IPs.

Because it is enforced per BEAM node, a load-balanced cluster of N nodes has an effective ceiling of roughly max_connections × N. Size the per-node value against one node's capacity, and use your load balancer's own global connection/rate controls when you need a cluster-wide cap.

The connection limit uses the real TCP peer address and deliberately ignores forwarded headers like X-Forwarded-For, which clients can forge. Two consequences:

  • Behind a reverse proxy or load balancer, every connection appears to come from the proxy's IP, so a per-IP cap would throttle all clients together. Enforce per-client limits at the proxy layer instead.
  • Users behind carrier-grade NAT or a corporate gateway share one IP. Set the cap high enough for your worst legitimate case, or leave it off and rely on rate limits.

Per-socket limits are keyed by connection, so a client that hits a limit can reconnect for a fresh allowance. They bound the damage of any single connection; they are not a substitute for infrastructure-level controls (load balancer connection/request limits, WAF rules) against determined attackers rotating connections.

  • with_allowed_origins on the Mist transport rejects browser connections from unexpected origins before the WebSocket handshake.
  • with_on_connect authenticates the connection once, before upgrade — reject unauthenticated clients with a 403 rather than at join time.
  • Authorize each topic in your channel's join callback; clients cannot send events to topics they have not joined.

beryl's distributed PubSub and presence replication run over Erlang distribution. Every node in your Erlang cluster is fully trusted: a process on any peer node can subscribe to any topic, receive all broadcasts, and deliver messages that beryl's coordinator will process as legitimate internal traffic. Channel authorization callbacks and WebSocket-layer controls do not apply to messages delivered over distribution — they protect only inbound WebSocket clients.

SourceTrust levelGated by
WebSocket clientsUntrustedwith_on_connect authentication, channel join authorization, and handle_in event handling
Erlang distribution peersFully trustedErlang cookie + network controls

A hostile distribution peer can broadcast arbitrary messages into any channel and read all presence state. Secure the cluster boundary before deploying multi-node beryl.

Set a long, randomly generated cookie in vm.args or the RELEASE_COOKIE environment variable. The cookie is the only authentication mechanism for distribution peers; a weak or default cookie (CHANGE_ME, the hostname, etc.) allows any process that can reach your distribution port to join the cluster.

Generate a strong cookie with, for example:

Terminal window
openssl rand -base64 48

When nodes communicate across networks you do not fully control — cloud subnets, VPNs, or any link that may traverse untrusted infrastructure — enable TLS distribution. See the Erlang TLS Distribution guide for setup instructions.

The EPMD port (default 4369) and the Erlang distribution listen range must not be reachable from untrusted networks. Use firewall rules or security groups to allow only cluster-internal traffic. You can fix the distribution port to a known value for simpler rules:

Terminal window
# vm.args
-kernel inet_dist_listen_min 9100
-kernel inet_dist_listen_max 9100

Or set ERL_DIST_PORT when using Elixir/Mix releases. Restrict both 4369 and your chosen distribution port at the network layer.

Do not share clusters with untrusted tenants

Section titled “Do not share clusters with untrusted tenants”

Adding a node to an existing cluster grants it full visibility into all pg groups (PubSub topics) and all presence state on every node. Never connect beryl to a cluster that contains nodes owned or operated by parties outside your trust boundary.

See SECURITY.md for the full trust-boundary and distribution-hardening reference.

  • The coordinator is a single actor per channels system. The transport sheds oversized frames and (when configured) rate-limited traffic before it, but extreme fan-out workloads may want multiple channels systems sharded by topic space.
  • Enable and export Beryl's bounded telemetry taxonomy, and poll local coordinator snapshots, as described in the Observability guide. Monitor the coordinator mailbox, connection count, BEAM ports/processes, and edge saturation together. The benchmark server's /stats endpoint demonstrates the JSON shape; it is not a production exporter, and telemetry handlers remain application-owned and synchronous.
  • beryl always runs inside your own supervision tree via supervisor.start. The subtree has a one-for-one parent wrapping a rest-for-one channel supervisor, and restarts are bounded at 3 per 5 seconds before the beryl supervisor shuts down and defers to its parent. See the Supervision guide for the full tree.

Before setting production limits, establish a controlled capacity baseline on representative topology. The repository's load-testing guide covers Mist/Ewe parity, multi-source execution, OS and BEAM limits, NAT/proxy constraints, repeatability, and promotion of measured baselines to thresholds.