How Nexus PDP syncs data
This page explains how Permit Nexus PDP, a self-hosted policy decision point (PDP), receives policy and data changes from Permit's control plane, what consistency that gives your application, and how to size the container. It is for operators and architects who run Nexus PDP. For what Nexus PDP is, see Permit Nexus PDP.
In this page, the container PDP is the Edge PDP image permitio/pdp-v2, and the control plane is the Permit cloud service that sends policy and data updates to PDPs.
How policy and data stay in sync
The NATS leaf node inside the Nexus PDP container keeps a persistent connection to Permit's control plane. The NATS leaf node copies four independent streams, called planes, to local disk:
| Plane | Carries |
|---|---|
| Change stream | Transactions of authorization data: users, tenants, resource instances, and relationship tuples |
| Policy files | The compiled Rego bundle that Open Policy Agent (OPA) evaluates |
| Policy schema | Role and permission definitions from your policy |
| Snapshot | A one-time bulk transfer of the whole data set, used only during a cold start |
Each change arrives as a transaction: an ordered set of operations with a timestamp and a transaction ID. The message contains the changed data, so Nexus PDP does not fetch anything from Permit after it receives a change.
Change retention for each Nexus PDP
Each Nexus PDP has its own durable position in the change stream. The control plane keeps a change until that Nexus PDP acknowledges that it applied the change. A Nexus PDP that restarts, disconnects, or falls behind resumes from its last acknowledged position and does not fetch its full data set again.
Gap detection and database rebuild
Two background tasks keep the local copy complete:
- The gap detector compares the Nexus PDP's position in the change stream with the oldest change the control plane still holds. If the Nexus PDP has fallen behind that oldest change, the stream cannot close the gap, and the gap detector requests a rebuild.
- The rebuild orchestrator builds a new copy of the database in a separate directory, then swaps the new copy in atomically.
While a rebuild runs, Nexus PDP answers requests from the existing database. The authorization port stays open, and Nexus PDP refuses no requests. The health endpoint reports that the Nexus PDP is serving stale data.
When the control plane is unreachable, Nexus PDP keeps answering from its local copy for as long as the outage lasts. A control-plane outage affects data freshness, not the availability of decisions.
Consistency
Nexus PDP has one consistency model, with no settings to change. Three mechanisms combine:
- A durable, ordered change stream. The control plane sends changes in order on a persisted stream and keeps each change for each Nexus PDP until that Nexus PDP acknowledges it. A Nexus PDP that falls behind the oldest retained change rebuilds its database from a snapshot.
- Order-independent application. Each fact carries a timestamp and a transaction ID. Nexus PDP applies a write only if it is strictly newer than the stored value. Because of this rule, the final state does not depend on the order in which Nexus PDP applies changes, so Nexus PDP can apply several transactions concurrently without an older write replacing a newer one.
- Atomic transactions. Nexus PDP applies all operations in a transaction together, so a decision never reads half of a multi-part change.
Together, these mechanisms mean that two Nexus PDPs in the same environment that applied the same set of changes hold the same state, whatever the order or timing of delivery. Applying a change a second time has no effect.
The consistency model is eventual consistency with a convergent merge. Nexus PDPs converge on the same state, no update is lost, and replayed updates are safe.
Nexus PDP does not provide read-your-own-writes. Nexus PDP only reads your data: a fact that you write through the Permit API becomes visible after the change reaches the Nexus PDP. If your application must read a write immediately, use the container PDP with the local facts uploader.
A Nexus PDP that restarts with a backlog of changes becomes ready and serves requests while it catches up. The /health/detail endpoint reports whether the Nexus PDP is serving stale data and for how long.
Propagation latency
Nexus PDP delivers a change to the PDP in fewer steps than the container PDP:
Container PDP (pdp-v2) | Nexus PDP (pdp-v3) | |
|---|---|---|
| Change notification | WebSocket notification | Push delivery on a durable subscription for each PDP |
| Data fetch | A second request to the Permit API | None. The message contains the change. |
| Round trips for each change | 2 or more | 1 |
| Changes made while the PDP is offline | Reconnect, then fetch the data again | Kept by the control plane; the PDP resumes at its last position |
| Requests to the Permit API to fetch data | One for each update | None |
Because Nexus PDP skips the fetch step, propagation depends only on the change stream, not on the PDP reaching the Permit API.
Permit has not published measured propagation latency for Nexus PDP. The table above compares the number of steps, not measured time. If you have a latency target, book a call with Permit before you design around a specific number.
Cold start and warm resume
Nexus PDP chooses a startup path from the contents of its persistent volume.
Warm resume: restart with an intact volume
The embedded database, the event store, and every stream position are already on disk. Nexus PDP opens the database and resumes applying changes from its last acknowledged position, without a snapshot transfer and without fetching data again.
Warm resume covers pod restarts, rolling updates, and short disconnections, because the control plane kept the disconnected Nexus PDP's changes.
Cold start: first boot or a gap the stream cannot close
Nexus PDP requests a snapshot of its environment in two cases: on first boot with an empty volume, and when the gap detector finds a gap that the change stream cannot close.
The snapshot contains pre-built database files. Nexus PDP receives the files in chunks, verifies a SHA-256 checksum over the assembled payload, and loads the files directly into the storage engine instead of writing records one by one. The snapshot also carries the Rego bundle and the policy schema, so a cold-started Nexus PDP gets policy and data together.
The snapshot records the change-stream position where Permit took it. Nexus PDP resumes the change stream from the next change, so no change is skipped, and a change applied twice has no effect.
If the checksum does not match, a chunk is missing, or the snapshot is empty, the cold start fails. Nexus PDP does not start serving decisions from a partial data set.
A cold start transfers and loads your environment's whole data set, so it takes longer as your data grows. Configure a startup probe with enough time for your data set, and let the readiness probe gate traffic. The health endpoint on port 7001 answers during the whole cold start, and the authorization port 7000 opens only when Nexus PDP is ready. For probe settings, see Nexus PDP health and readiness.
Request handling at high volume
Nexus PDP avoids per-request work that depends on other systems:
- No network hop in the decision path. Every hop uses loopback or local disk.
- No control-plane request to authenticate. Nexus PDP compares the request's bearer token with its own
PDP_API_KEYin constant time. - No dependency on Permit's availability for decisions. A control-plane outage affects data freshness, not throughput or availability.
- Sync on separate tasks. Change ingest runs on its own tasks, separate from request handling.
- The managed Cloud PDP's evaluation core. Nexus PDP is built from the same authorization libraries as the managed Cloud PDP.
Permit has not published measured throughput for Nexus PDP. The Cloud PDP benchmarks measure the managed Cloud PDP, a different deployment, and are not Nexus PDP figures.
Resource footprint
Nexus PDP stores authorization data in a different place than the container PDP, which changes how you size the container.
Container PDP (pdp-v2) | Nexus PDP (pdp-v3) | |
|---|---|---|
| Authorization data | In OPA's in-memory document | On disk, in an embedded database |
| Memory as data grows | Grows with your data set | Limited by a cache size you configure |
| Processes | Rust API server, Python OPAL client (Horizon), and OPA | Rust binary, NATS leaf node, and OPA |
| Python runtime | Required, for the Open Policy Administration Layer (OPAL) client | Not present |
| Persistent storage | Not required | Required |
On the container PDP, your authorization data is raw JSON in memory. The OPA policy performance documentation states that raw JSON in OPA uses about 20 times the memory of the same data in a compact on-disk form. On Nexus PDP, the data set lives on disk in a compact form, and memory depends on the storage engine's cache sizes, which you set.
The default storage-engine settings of Nexus PDP target large workloads: a 512 MiB block cache and 256 MiB write buffers. If you give the container a memory limit of a few hundred MiB, the container is killed for running out of memory (OOM) at startup. Give the Nexus PDP container 4 GiB of memory to start.
To run Nexus PDP with less memory, lower the cache and write-buffer sizes in Nexus PDP storage engine settings, then load test against your own data set before you choose a size.
Disk
Two Nexus PDP paths must be on persistent storage: the embedded database (EDGE_DB_PATH) and the event store (EDGE_DATA_DIR). Plan for about twice the size of your data set, plus headroom. During a rebuild, the old and new database copies exist on disk together until Nexus PDP swaps in the new copy.
Next steps
- Permit Nexus PDP: what Nexus PDP is and when to choose it
- Nexus PDP architecture: the processes and ports inside the container
- Nexus PDP feature parity: supported endpoints and capabilities
- Deploy Nexus PDP: storage, probes, logs, and the security model
- Nexus PDP configuration reference: environment variables