What's New in 1.2
This page lists the changes between Bondy Connect 1.0.0 and 1.2. It covers releases 1.0.1, 1.0.2, 1.1.0, 1.2.0 and 1.2.1. Each section starts with the changes that need action from you.
Coming from a release candidate?
A deployment that runs a 1.0.0 release candidate has more to change: configuration keys, the WAMP and HTTP APIs, and the realm authentication rules. Follow Migrating from a Release Candidate. It includes every change on this page.
1.2.1
This release makes a cluster recover faster when one of its nodes restarts while the cluster carries traffic. It changes nothing in the routing of a healthy cluster, or on a single node.
- A node that restarts routes correctly sooner. A node rebuilds its routing table from its peers after a restart. Under load that step could take minutes, and for that time calls through the node to procedures on other nodes failed with
wamp.error.no_such_procedure. The node now rebuilds the table as its first work. - A restart costs the other nodes less. A node that restarted repeated the rebuild many times and replicated the result each time, which kept every node busy for minutes. The node now does that work once.
- A planned restart is shorter. A node that stops no longer waits for the whole grace period when its clients have already left.
- Several nodes can recover at once. Each node limits the recovery work it does for its peers at one time, so recovering nodes do not all load the same node.
- Two metrics report the real value. The count of timed-out calls and the load gauge of the anti-entropy scheduler were misleading.
This release needs no action from you. It adds two configuration options, and both have defaults. The fixes below give the detail.
Fixes
- A node that restarts while its peers are busy rebuilds its routing table without delay. Such a node could take minutes, and for that time it reported ready and answered calls to procedures registered on other nodes with
wamp.error.no_such_procedure. The bootstraps that rebuild the table had to fit underdb.oplog.aae.max_concurrency, and live sync sessions kept that limit full. Bootstraps now have their own limit, the new optiondb.oplog.aae.bootstrap.max_concurrency(default4). A node that recovers can therefore run up to that many sync sessions abovedb.oplog.aae.max_concurrency. The[bondy_oplog, sync_scheduler, bootstrap_capped]telemetry event no longer carries thenode_currentandnode_capmeasurements. - A node that restarts rebuilds its routing table once per shard. It rebuilt the table once for every batch of 64 entries it received, and on each pass it wrote again every routing entry it owns. Those writes replicated to the other nodes, and the node applied routing changes from its peers late for as long as the passes lasted. A bootstrap now announces each table once, when the whole transfer for a shard is written.
- Load yielding no longer stops bootstraps. With
db.oplog.aae.load_adaptiveon, a node whose load was at or above the threshold started no bootstrap, so a node that restarted while busy rebuilt no routing table for as long as the load lasted. The setting no longer defers bootstraps. The[bondy_oplog, sync_scheduler, bootstrap_load_deferred]telemetry event and thebootstrap_load_deferredvalue ofbondy_oplog_sync_scheduler_events_totalare gone. - A node limits the bootstraps it serves. A node served every bootstrap its peers asked for, so many nodes that recovered at once could all load the same node. A node now serves at most
db.oplog.aae.bootstrap.serve.max_concurrencyat a time (new option, default8). It refuses the next one, and the asking node asks again after its retry backoff. The new metricbondy_oplog_bootstrap_serve_refused_totalcounts the refusals. A node whose bootstrap fails now tells the serving node at once, so the serving node no longer keeps that bootstrap open for 60 seconds. - A graceful stop ends when the clients have left. A node waited for the whole of
shutdown.grace_period(default30s), also when it had no client connected. The node now ends the wait when the last client connection closes and, in a cluster, the nodes it is connected to hold the removal of its registrations and subscriptions.shutdown.grace_periodis the longest it waits. See Shutdown. bondy_rpc_promise_timeouts_totalcounts only the calls that end withwamp.error.timeout. It also counted every expiry that Bondy answers with a new attempt for a call that set_retries, and each wait of such a call that found no callee. On a workload that uses_retries, the counter was higher than the number of timeouts callers received.bondy_oplog_sync_scheduler_loadreports the load with load yielding off. The gauge read0whiledb.oplog.aae.load_adaptivewas off, which is the default. The scheduler now samples the load on every tick, so the gauge shows how close a node runs todb.oplog.aae.load_run_queue_thresholdbefore you turn the setting on.
Known limitation
Every shard that a Bondy node starts backs the authentication freshness fence, and the scheduler does not throttle, defer or cap the live syncs of such a shard. On a Bondy node db.oplog.aae.live_sync, db.oplog.aae.live_sync.max and db.oplog.aae.load_adaptive therefore have no effect. db.oplog.aae.max_concurrency does not limit how many live syncs run; it only sets the size of the batch each session pulls. The bootstrap limits and db.oplog.aae.max_pages_in_flight apply as described.
1.2.0
Action required
- A node that runs 1.2 cannot go back to an earlier release. 1.2 upgrades the storage engine, and from its first start a node writes stored data in a format that earlier releases cannot read. Before you upgrade, take a logical export. To go back, deploy a new cluster on the earlier release and import the export; it holds nothing written after you took it.
Fixes
- A RawSocket client gets no message longer than the length it announces. Bondy ignored this length, and a client that enforces it closed the connection. Bondy now drops an
EVENTfor that subscriber only, and counts the drop inbondy_wamp_dropped_total. A caller getsERRORwithwamp.error.payload_size_exceededwhen itsRESULTis too long for it, or itsINVOCATIONtoo long for the callee. A progressive call ends, and Bondy interrupts its callee. WebSocket and HTTP clients announce no length, so nothing changes for them. See RawSocket message length. - A URI that ends in a newline is refused. Bondy accepted
com.example.thingfollowed by a newline under bothwamp.uri.strictnesssettings, so a topic, procedure, realm or grant could carry one and differ from the same URI without it. Bondy now refuses it, as it refuses a URI with any other control character. URI validation is also several times faster. - A refused
bondy.*call names the realm it received. A call whose realm argument was not the session's realm gotwamp.error.not_authorized, and the description named the session's realm instead. A realm sent as an Erlang charlist ("com.example") rather than a binary therefore looked like a correct call. The description now shows the realm received as an Erlang term, so its type is visible, and the error'sdetailscarry it asrequested_realm_uri. - A missing or malformed bearer token gets
bondy.error.token_invalid. On the Admin API and on API Gatewayoauth2paths, a request without anAuthorization: Bearerheader, or with a bearer value that is not a JWT, got401with aninternal_errorbody. The body now names the refusal. See OAuth2 Authentication. - The
WWW-Authenticate: Bearerchallenge follows RFC 6750. This covers the Admin API, API Gatewayoauth2paths, the credential-verification endpoints and the MCP endpoint. A request with no bearer token getsBearer realm="<realm>", and a refused token addserror="invalid_token". Before, the API Gateway sent Bondy's own error code and extra parameters, and the MCP endpoint sent a bareBearer. A client that read Bondy's error code from the header reads it from the response body instead. TheCookie,BasicandApiKeychallenges do not change.
1.1.0
Action required
- A node refuses to start on a key that it does not read. This covers a renamed key, a removed key and a typo. The error names each such key and up to three similar keys that the release knows. The node also refuses to start on a value that it cannot parse. Earlier releases ignored such a key, so its setting stayed at the default without a message. Run
scripts/migrate_conf.escript check <path>/bondy.confbefore you upgrade. - The
db.*keys are grouped by the component they configure. Most of them have a new name. For example,db.aae.*isdb.oplog.aae.*anddb.wal.*isdb.oplog.wal.*. Values, units and defaults do not change.scripts/migrate_conf.escript migratewrites the converted file. Storage and anti-entropy keys lists every rename.
New features
- Pack collection settings.
db.oplog.pack.gc.interval,db.oplog.pack.gc.growth_factoranddb.oplog.pack.gc.min_recordstune the reclamation of a durable table's anti-entropy tree. See Pack Collection.
1.0.2
Fixes
- Anti-entropy tree pages are reclaimed as writes grow. Release 1.0.1 reclaimed them at most once an hour, and never in a node's first hour. A burst of writes, such as a restore, could fill the data volume first. Reclamation now also runs when the sealed records of a shard's tree reach twice the count that the last run left, counted from at least 10,000 records. See Pack Collection.
1.0.1
Action required
- The Admin HTTP API needs a bearer token. Every listener that carries
admin_apineeds an OAuth2 access token from the master realm on each request other thanOPTIONS. A request without one gets401. The internal socket needs no token. See Authenticating to the Admin API. - Update procedures refuse credentials.
bondy.user.update,bondy.oauth2.client.updateandbondy.oauth2.resource_owner.update, and theirPUTroutes, refuse apassword. The client procedure and route also refuse aclient_secret. Usebondy.user.change_password. cookieandoidcrpare separate methods.cookieadmits a credential that arrives in a cookie.oidcrpadmits a ticket from the OIDC login flow. A realm that uses OIDC needs both in itsauthmethods, each with a source that allows it.- Sources apply to OIDC users without a user record. Add sources assigned to
allforoidcrpandcookie, for the networks your users connect from. - The verify route applies the realm's methods and sources. See Verify route.
oauth2.refresh_token.limitis a whole number of tokens. A value with a unit suffix, or0, stops the node at boot.- Three OIDC keys are removed. Delete
security.oidc.providers.enabled,security.oidc.providers.<id>.login_pathandsecurity.oidc.providers.<id>.redirect_path. No code read them. - MCP matches sources against the forwarded client address when the request comes from a trusted proxy.
Update realm authentication settings describes each authentication change.
New features
- OIDC on the Admin API and API Gateway. The Admin API accepts the master realm's OIDC ticket cookie. An API Gateway path with the
oidcscheme authenticates with the realm's ticket cookie. - Dry-run imports.
bondy.export.importtakesdry_run, andbondy.export.statusadds an import report. See Backup and Restore and the import report. - The
bondy_auth_fence_closedalarm. Bondy raises it when the authentication freshness fence refuses every new authentication for too long. See Alarms. - Alpine images. Images with the
-alpinesuffix are published again forlinux/amd64andlinux/arm64. See Install using Docker.
Fixes
The fixes that change a result your clients can see:
bondy.user.enableandbondy.user.disableanswerwamp.error.no_such_principalfor an unknown user.- A reserved user or group name, such as
alloranonymous, getsbondy.error.invalid_value. bondy.user.remove_group,bondy.group.add_group,bondy.group.add_groupsandbondy.group.remove_groupanswer an empty result, notbondy.error.internal_error.bondy.user.remove_aliasremoves only an alias that the user holds.- An API Gateway path with the
basicorapi_keyscheme answers401, not500. - A bearer value that is not a JWT gets
401, not400. - With
security.rate_limiton, Bondy charges API Gateway, Admin API and OAuth2 token-endpoint requests before authentication, so a flood of refused requests gets429. See Rate Limiting.