Monitor beryl

beryl provides these sources of monitoring data:

• Optional :telemetry events report rates, outcomes, and operation durations.
• beryl/snapshot.get reports local runtime state at one time.
• Queue snapshots report capacity and age without waiting for an actor turn.

beryl does not include a Prometheus or OpenTelemetry exporter. Your application must aggregate, label, and export the data.

Configure logging

Your application owns OTP Logger configuration. beryl emits structured Palabres reports through OTP Logger, but does not install handlers or change the primary level, filters, metadata, or formatter during startup or restart. Existing application logging setup stays in effect.

beryl.with_logging sets a local minimum level for one socket system's runtime and transport diagnostics. The default is InfoLevel. Two systems can use different thresholds without changing each other. Independently started presence, PubSub, and other support processes use the host's policy.

The host level and handler filters can further restrict output. For example, DebugLevel in beryl does not enable debug output if OTP Logger's primary level is info or notice. A release can set the host level in sys.config:

[{kernel, [{logger_level, info}]}].

Use debug there and beryl.DebugLevel for debug diagnostics. Keep existing application handlers and filters unless you intend to change them.

Earlier beryl versions called palabres.configure during runtime startup. Applications that relied on that setup must now configure their desired output at application startup. palabres.configure is optional and changes node-wide logger policy, including primary filters and default-handler formatting. If you use it, review its treatment of OTP crash and supervisor reports and preserve your intended policy. beryl does not require it.

See debug logging for the per-system configuration.

Enable telemetry events

Telemetry is disabled by default:

let config =
  beryl.config(wire.phoenix_codec())
  |> beryl.with_telemetry

Attach handlers before traffic begins. Erlang :telemetry invokes handlers synchronously in the process emitting the event. A slow handler therefore adds latency to a WebSocket connection process, a socket's runtime actor, or the runtime's router. Update a fixed-size counter or histogram, or enqueue a small message and return. Do not make network calls, format logs, or convert large sets of labels in the handler.

These stable event names use a fixed set of measurement and metadata keys:

Event	Measurements (numeric values)	Metadata (labels)
[:beryl, :transport, :upgrade, :stop]	count, duration	transport, outcome
[:beryl, :transport, :frame, :stop]	count, duration, bytes	transport, frame_type, outcome
[:beryl, :socket, :connected]	count	none
[:beryl, :socket, :disconnected]	count, duration, joined_channels	reason
[:beryl, :channel, :join, :stop]	count, duration	outcome
[:beryl, :channel, :message, :stop]	count, duration	kind, outcome, callback_result
[:beryl, :broadcast, :stop]	count, duration, recipients	origin
[:beryl, :queue, :occupancy]	items, bytes, max_items, max_bytes, high_items, high_bytes, rejected, cancelled, oldest_age_ms	boundary, outcome

Queue age uses monotonic milliseconds, unlike the duration fields below. Queue outcomes are changed and rejected. Admission failures also use admission_rejected in frame, join, message, callback, and disconnect outcomes. Enable presence queue events with presence.with_telemetry(config). See queue snapshots and labels for the snapshot functions, byte-accounting exclusions, and concurrent event ordering.

The :channel event-name segment is retained for telemetry compatibility even though raw dispatch now owns routing. For an app update, callback_result describes the first response-like effect (reply, reply_error, or push), no_reply when none is returned, and stop for a socket stop.

duration uses the BEAM's native monotonic time unit. Convert it with erlang:convert_time_unit(Value, native, microsecond) (or another desired unit) before export. It is not milliseconds. Counts and byte/recipient fields are integers.

Metadata values are atoms from fixed lists. They omit topics, socket IDs, payloads, and arbitrary error text, so the built-in labels remain bounded:

Key	Values
transport	mist, ewe, custom
upgrade outcome	success, origin_rejected, version_rejected, auth_rejected, capacity_rejected, handshake_failed
frame outcome	routed, oversized, rate_limited, decode_failed
join outcome	accepted, handler_rejected, no_handler, invalid_topic, topic_limit, rate_limited, callback_error, socket_missing
message outcome	handled, unjoined, stale, invalid, rate_limited, callback_error, socket_missing
frame_type	text, binary
message kind	text, binary, info, presence, heartbeat
callback_result	not_applicable, no_reply, reply, reply_error, push, stop, failed
disconnect reason	normal, heartbeat_timeout, shutdown, callback_error, admission_rejected, socket_process_down, encoding_error
broadcast origin	local, remote

Map future unknown values to an "unknown" label. Do not crash the handler.

Third-party transports construct an opaque context with transport.telemetry(sockets, transport.Custom) and pass it to the shared server pipeline. All third-party implementations share the fixed custom label; the API does not accept transport-name strings. This keeps transport label cardinality bounded to three values and preserves the bundled mist and ewe labels.

Custom contexts use the socket system's telemetry setting. With beryl.with_telemetry, they emit the same upgrade and inbound-frame events as bundled transports. Without it, shared setup and routing still work, but transport telemetry operations avoid clock calls and event construction.

Socket lifetimes

The router emits both socket lifecycle events. After it emits connected, it emits one disconnected for an orderly close, socket actor failure, or forced shutdown drain. Failed or abandoned admission before connected produces neither event. Disabling telemetry suppresses both events.

For orderly teardown, duration covers admission through teardown completion, and joined_channels counts memberships at the start of teardown. For a socket actor failure (socket_process_down) or forced drain (shutdown), duration covers admission through the router's cleanup observation. In those two cases, joined_channels counts memberships in the router's index at that observation. In-flight index changes can lag the actor, and partial teardown can have removed memberships. These measurements keep the same numeric keys and native duration unit; they do not substitute zero for missing actor state.

Use snapshots for current-count gauges. Whole-node or router failure, or a failing telemetry handler, can prevent lifecycle event delivery, so subtracting disconnect counts from connect counts cannot provide a reliable current count.

Export metrics to Prometheus or Grafana

Keep the exporter in your application:

1. Attach one handler with telemetry:attach_many/4.
2. Map each event to an application-owned counter or histogram. Preserve only the bounded metadata above as labels.
3. Convert native durations and update the in-memory metrics store, or send a message to a supervised metrics actor with a fixed queue limit.
4. Expose the aggregator through your existing Prometheus HTTP endpoint, OpenTelemetry SDK, or hosted metrics client.
5. Configure Prometheus to scrape that endpoint and use Grafana to query the resulting series.

For example, an application FFI module can attach a single Erlang handler:

-module(my_app_beryl_metrics).
-export([attach/1, detach/1]).

attach(AggregatorPid) ->
    Id = {?MODULE, AggregatorPid},
    Events = [
        [beryl, transport, upgrade, stop],
        [beryl, transport, frame, stop],
        [beryl, socket, connected],
        [beryl, socket, disconnected],
        [beryl, channel, join, stop],
        [beryl, channel, message, stop],
        [beryl, broadcast, stop]
    ],
    telemetry:attach_many(
        Id,
        Events,
        fun(Event, Measurements, Metadata, Pid) ->
            %% Keep this non-blocking and bound the receiver's mailbox.
            Pid ! {beryl_metric, Event, Measurements, Metadata}
        end,
        AggregatorPid
    ),
    Id.

detach(Id) ->
    telemetry:detach(Id).

The supervised metrics actor can convert messages for your metrics library. Monitor its mailbox and limit queued work. Another process avoids adding work to the request, but an unlimited mailbox can still overload the system. Detach the handler during shutdown. beryl does not need an exporter dependency.

Candidate derived signals include upgrade rejection rate by outcome, frame decode and rate-limit rates, join/message callback failures, connection lifetime, broadcast recipient counts, and latency histograms. Consider alerts for sustained rates and tail latency rather than individual events, then tune those thresholds from observed workload behavior.

Read runtime counts

beryl/snapshot.get(channels) requests a point-in-time view from the socket runtime represented by channels:

import beryl/snapshot

case snapshot.get(channels) {
  Ok(current) -> {
    let sockets = snapshot.connected_sockets(current)
    let joined_pairs = snapshot.joined_socket_topic_pairs(current)
    let topics = snapshot.active_topics(current)
    // Publish these gauges through the application's metrics system.
  }
  Error(snapshot.RuntimeUnavailable) -> {
    // Supervisor restart or shutdown: report the scrape/poll as unavailable.
  }
  Error(snapshot.RequestTimedOut) -> {
    // The bounded request was not serviced in approximately one second.
  }
}

The snapshot describes one socket runtime on one BEAM node. It is not an event stream or a single consistent view of the whole cluster. Aggregate gauges across nodes in the monitoring system. joined_socket_topic_pairs counts memberships. One socket on two topics adds two. The runtime records counts when it handles the request, so concurrent connection changes can appear later.

Start with polling at once per second or less, then measure the overhead for your workload. Each poll sends a request through the runtime, and many synchronized scrapers can add load. Use one application poller per node. Cache the latest successful snapshot, add jitter, and expose the snapshot age. Do not convert a timeout to a zero-valued snapshot. A timeout can mean restart or overload, not an idle system.

For a runnable JSON endpoint combining beryl and BEAM runtime gauges, see the benchmark server's /stats reference and http.gleam. That endpoint polls on request and is a benchmark fixture, not a bundled exporter. For a deployment, one application-owned poller per node can reduce scraper-driven load; serve its cached snapshot through your metrics handler and validate that this pattern fits your workload.

Measure capacity

Correlate telemetry and snapshots with BEAM process/port counts, memory, run queue, host CPU, open file descriptors, TCP statistics, and proxy/NAT utilization. The repository's load-testing guide documents profiles, result metadata, repeatable baselines, and safe tuning.