How to Diagnose Events That Do Not Reach Subscribers on Another Node
Find out why an event published on one node of a cluster does not reach a subscriber connected to another node, when subscribers on the publisher's own node receive it.
A lost event raises no error. The publisher's PUBLISH succeeds, so the metrics are the only place the loss shows. This guide finds the hop where the event stops.
Prerequisites: Prometheus scrapes every node; see Monitoring with Prometheus & Grafana. The queries group by node, the target label that the bundled Prometheus configuration sets. If your scrape configuration sets no such label, group by instance. Replace <publisher-node> and <subscriber-node> with the nodes the two sessions are connected to. For how a node relays an event, see Registry Routing (RIB).
Steps
1. Rule out the subscription and the publication
Confirm that the subscriber holds a subscription that matches the topic. Call wamp.subscription.match with the topic URI, through a session on the subscriber's node and in the subscriber's realm. A null result means that no subscription matches.
Then check the options of the PUBLISH. exclude, eligible and their authid and authrole forms remove subscribers by design. An empty eligible list delivers the event to no subscriber.
2. Check the publisher node's view of subscribers
The publisher's node relays an event only to the nodes that it knows hold a matching subscription. Compare how many subscription cells each node holds for the others:
sum by (node) (bondy_registry_rib_stub_cells{type="subscription"})On a healthy cluster with steady subscriptions, the values are close and flat.
- The publisher's node holds far fewer cells than its peers. Its view is behind, so it does not relay to nodes it has not learned about. This is the normal state of a node for a while after it restarts or rejoins from a network partition. The events published on it in that time are not delivered later. Follow Diagnosing a Node That Is Slow to Catch Up, and keep publishers away from the node until its count levels with its peers.
- The counts are level. The publisher's node knows the subscriber's node. Go to step 3.
3. Check whether the subscriber's node shed the event
A relayed event arrives on the subscriber's node in an ordered lane with a bounded capacity. When the lane is full, the node drops the event and counts it:
sum by (node, reason, family) (rate(bondy_wamp_dropped_total[5m]))reason | Meaning |
|---|---|
shed with family="relay" | The node dropped messages that arrived from a peer node, because the flow worker's share of load_regulation.router.flow_pool.capacity was in use. The subscribers on this node are slower than the publishers that feed them. |
payload_size_exceeded | An EVENT was longer than the message length a RawSocket subscriber announced. Only that subscriber missed it. See RawSocket message length. |
A shed rate on the subscriber's node is the loss. Go to step 4 to find what makes the lane slow.
4. Find the slow hop on the subscriber's node
Three histograms cover the path of a relayed event on the receiving node:
histogram_quantile(0.95, sum by (le) (
rate(bondy_router_flow_queue_depth_bucket{node="<subscriber-node>", family="relay"}[5m])))
histogram_quantile(0.95, sum by (le) (
rate(bondy_router_flow_service_microseconds_bucket{node="<subscriber-node>", family="relay"}[5m])))
histogram_quantile(0.95, sum by (le, transport) (
rate(bondy_wamp_egress_queue_depth_bucket{node="<subscriber-node>"}[5m])))| What you see | What it means |
|---|---|
| Flow queue depth grows, service time is flat | More relayed traffic arrives than the flow workers can run in order. Give the node more capacity, or spread the publishers that target one subscriber session. |
| Flow service time grows | Each relayed message takes longer to authorize, match and fan out. Look at bondy_broker_publish_match_microseconds for a match that grew with the number of pattern subscriptions. |
Egress queue depth grows for one transport | The subscribers' connections do not drain. The clients, or the network to them, are slower than the events arrive. |
Depth multiplied by service time estimates the wait of each message in a queue.
5. Apply the fix
| Cause | Fix |
|---|---|
No matching subscription, or the PUBLISH options exclude the subscriber | Correct the client. |
| The publisher node's view is behind | The node heals when anti-entropy catches up; see Diagnosing a Node That Is Slow to Catch Up. Keep publishers off the node until then. A publisher cannot detect or retry this loss. |
| The subscriber's node sheds relayed events | Remove the slow hop found in step 4. Raising load_regulation.router.flow_pool.capacity delays the drops and holds more memory; it does not make a slow consumer faster. |
An EVENT exceeds a subscriber's message length | Have the subscriber announce a larger length in the RawSocket handshake, or publish smaller payloads. |
Result
bondy_wamp_dropped_total stops growing on the subscriber's node, and bondy_registry_rib_stub_cells{type="subscription"} is level across the nodes. A test event published on the publisher's node reaches the subscriber.
See also
- Prometheus Metrics Reference for every histogram on the delivery path.
- WAMP Compliance for the ordering Bondy guarantees between nodes.
- Diagnosing Calls That Find No Callee for the same routing view applied to calls.