Parallel Grant Statements for Databricks Unity Catalog¶

Configure the Databricks Unity Catalog connector to run the GRANT and REVOKE statements of one resource in parallel, so a policy change that produces many statements is applied faster.

Overview¶

By default, the connector applies the GRANT and REVOKE statements of a resource one at a time. When parallel grant statements are enabled, it runs several of them at the same time, so policy changes are applied faster.

• Revokes still run before grants, so the final permissions always match your policies.
• A failed statement is retried as before and does not stop the others.

Use this when policy changes often produce many grant statements, for example a table shared with many users, groups or roles.

Configuration properties¶

Matching custom properties (Portal and PrivaceraCloud):

Limit the grant statements per queue¶

Each changelog queue (on-demand, offline and, with the dedicated changelog queue, the fast queue) runs its grant statements with its own helper threads, so a large offline sync does not take the helper threads of on-demand or fast-queue policy changes.

• Per-queue helper counts: helper threads start with the first grant statements and stop after 60 seconds idle, so each queue scales between 0 and its maximum. Set ..._THREAD_MAX_ONDEMAND, ..._THREAD_MAX_OFFLINE or ..._THREAD_MAX_FAST to give one queue a different number of helper threads. A queue left at 0 uses ..._THREAD_COUNT. The fast queue shares the offline queue's helper threads until its own count is above 0.
• Helpers-only mode: by default, the thread that processes a resource also runs that resource's statements, together with the helper threads, so a queue can run up to its resource.process threads + helper threads statements at once. With CONNECTOR_DATABRICKS_UNITY_CATALOG_PARALLEL_GRANT_UPDATES_HELPERS_ONLY: "true", that thread only hands its statements to the helper threads and waits, so each queue runs at most its helper thread count of statements at once, and one resource never takes every helper thread. Use it to cap the load on Databricks; policy changes that touch many resources can take longer.
• In helpers-only mode, a resource can wait for a free helper thread, and a stuck GRANT / REVOKE keeps its helper thread busy until it times out. The limit is CONNECTOR_DATABRICKS_UNITY_CATALOG_JDBC_GRANT_STATEMENT_TIMEOUT_IN_SECONDS (custom property ranger.policysync.connector.0.jdbc.grant.statement.timeout.in.seconds). When it is 0 (the default), the JDBC statement timeout applies instead: CONNECTOR_DATABRICKS_UNITY_CATALOG_JDBC_STATEMENT_TIMEOUT_IN_SECONDS, 1800 seconds (30 minutes) by default. 0 therefore does not mean no timeout. To free a stuck helper thread sooner, set a lower GRANT statement timeout, for example 300; see Fine Tuning JDBC Connection.
• The fast queue, while it shares the offline queue's helper threads, keeps running its own statements in helpers-only mode, so it never queues for an offline helper thread. A resource that the offline queue is processing stays locked while its statements wait for a helper thread, though, so a fast-queue change to that same resource waits too (up to 30 seconds, then it is retried). With helpers-only mode and the dedicated changelog queue, set a positive CONNECTOR_DATABRICKS_UNITY_CATALOG_JDBC_GRANT_STATEMENT_TIMEOUT_IN_SECONDS (for example 300) so a stuck statement frees its helper thread sooner.

For example, to give the fast queue 4 helper threads of its own and cap each queue at its helper thread count:

1
2
3
4

CONNECTOR_DATABRICKS_UNITY_CATALOG_PARALLEL_GRANT_UPDATES_THREAD_MAX_ONDEMAND: "8"
CONNECTOR_DATABRICKS_UNITY_CATALOG_PARALLEL_GRANT_UPDATES_THREAD_MAX_OFFLINE: "16"
CONNECTOR_DATABRICKS_UNITY_CATALOG_PARALLEL_GRANT_UPDATES_THREAD_MAX_FAST: "4"
CONNECTOR_DATABRICKS_UNITY_CATALOG_PARALLEL_GRANT_UPDATES_HELPERS_ONLY: "true"

On startup, the connector logs the helper threads of each queue, for example helpers (ondemand/offline/fast)=8/16/4, and parallel.grant.updates.helpers.only=true. offline's in the fast position means the fast queue shares the offline queue's helper threads.

How the startup values are calculated¶

On startup, the connector logs two numbers for each queue, for example:

1

grant_apply_ondemand: parallel grant statements enabled: helpers=8, max in flight per resource=9, helpers only=false

• helpers is the queue's maximum number of helper threads, as configured: ..._THREAD_MAX_<QUEUE>, or ..._THREAD_COUNT when that is 0.
• max in flight per resource is how many statements of one resource (for example, one table) can run at the same time. It is not the helper count:
  • With helpers-only mode off (the default), the thread that processes the resource also runs its statements, so the limit is the helper threads + 1, capped at ..._MAX_INFLIGHT_PER_RESOURCE.
  • With helpers-only mode on, that thread only waits, and one resource always leaves a helper thread for the other resources of its queue, so the limit is the helper threads − 1 (at least 1), capped the same way.
  • The fast queue, while it shares the offline queue's helper threads, always runs its own statements, so it uses the first formula with the offline helper count, also in helpers-only mode.

For example, with 8 / 16 / 4 helper threads and ..._MAX_INFLIGHT_PER_RESOURCE: "15":

The limit applies to one resource. A queue as a whole can run more statements at once: up to its resource.process workers + helper threads by default, or up to its helper threads in helpers-only mode.

Size the resource.process workers per queue¶

The resource.process worker threads apply a policy change to each resource, and each worker can run that resource's grant statements and hold a JDBC connection. Each changelog queue has its own pool of workers. By default every queue uses the same size: resource.process.thread.min / resource.process.thread.max (5 / 10), or 3 / 10 with the changelog thread pool V2 (CONNECTOR_CHANGELOG_USE_THREAD_POOL_V2).

How the two values work:

• MIN is the number of workers that run. A queue normally runs exactly its MIN workers, so a policy change on that queue processes up to MIN resources at the same time. Raise MIN for more parallel resources, or lower it to use fewer connections.
• MAX only adds workers for large backlogs. Workers above MIN start only while the queue's task queue (ranger.policysync.connector.0.resource.process.queue.max.length, default 10000 tasks) is full. Raising MAX alone does not make a queue faster.

You can size one queue on its own, for example a small dedicated fast queue next to a larger offline queue. A queue left empty (Privacera Manager) or at 0 (custom property) keeps the shared size. Privacera Manager has variables for the per-queue sizes only; set the shared size, if needed, as a custom property.

For example, in Privacera Manager:

1
2
3
4
5
6

CONNECTOR_DATABRICKS_UNITY_CATALOG_RESOURCE_PROCESS_THREAD_MIN_ONDEMAND: "5"
CONNECTOR_DATABRICKS_UNITY_CATALOG_RESOURCE_PROCESS_THREAD_MAX_ONDEMAND: "10"
CONNECTOR_DATABRICKS_UNITY_CATALOG_RESOURCE_PROCESS_THREAD_MIN_OFFLINE: "10"
CONNECTOR_DATABRICKS_UNITY_CATALOG_RESOURCE_PROCESS_THREAD_MAX_OFFLINE: "16"
CONNECTOR_DATABRICKS_UNITY_CATALOG_RESOURCE_PROCESS_THREAD_MIN_FAST: "2"
CONNECTOR_DATABRICKS_UNITY_CATALOG_RESOURCE_PROCESS_THREAD_MAX_FAST: "4"

or as custom properties:

1
2
3
4
5
6

ranger.policysync.connector.0.resource.process.thread.min.ondemand=5
ranger.policysync.connector.0.resource.process.thread.max.ondemand=10
ranger.policysync.connector.0.resource.process.thread.min.offline=10
ranger.policysync.connector.0.resource.process.thread.max.offline=16
ranger.policysync.connector.0.resource.process.thread.min.fast=2
ranger.policysync.connector.0.resource.process.thread.max.fast=4

Here the on-demand queue runs 5 workers (the default size), the offline queue 10 and the fast queue 2. Set only the queues you want to change: a pair left out keeps the shared size.

With these values, a policy change on the fast queue processes 2 resources at a time, so its statements run on at most 2 workers when each resource has only a few statements. Raising ..._THREAD_MAX_FAST alone, for example to 10, does not change that: workers above MIN start only when the queue has a backlog of more than 10,000 tasks. To process more resources at once, raise MIN.

On startup, the connector logs the workers of each queue, for example resource.process threads per changelog queue (min/max): ondemand=5/10, offline=10/16, fast=2/4. Include each queue's workers when you size the JDBC connection pool (see above): MIN for normal load, MAX for the worst case.

Setup¶

Configure the properties and restart the connector to apply the changes.

Monitoring¶

When the feature is enabled, the connector:

• Writes a grant_apply stats line to the connector log every minute in which grant statements ran, with the statement count and timings and the JDBC connection pool usage. A waiting count above 0 in the pool part means statements are waiting for a connection: increase the pool size.
• Logs a warning when a grant statement runs longer than the GRANT statement timeout, or longer than the JDBC statement timeout (30 minutes by default) when no GRANT statement timeout is set (see Fine Tuning JDBC Connection).
• Includes the time a statement waited before it started in the ACCESS audit time, shown as queueTime in the audit extra info.

To turn the feature off, set CONNECTOR_DATABRICKS_UNITY_CATALOG_ENABLE_PARALLEL_GRANT_UPDATES (or the custom property ranger.policysync.connector.0.enable.parallel.grant.updates) to false and restart the connector.