Send on-demand sync audits to S3, SQS, or SNS¶

Overview¶

When a PolicySync connector is configured with on-demand resource sync, it can publish one ON_DEMAND audit event per requested resource to the policysync_audit stream. Each event carries that resource's load outcome (found, added, updated, deleted, or not-found), its outcome codes, the policies applied during the sync, and the request context.

These events flow through the same Audit Server path as every other connector audit. Once you turn the feature on for the connector, the ON_DEMAND events reach whichever destinations you have already configured for the policysync_audit stream — Apache Solr, Amazon S3 (through Audit Fluentd), Amazon SQS, and Amazon SNS (through an SNS→SQS subscription).

Because the stream is indexed into Apache Solr, the events are also visible in Privacera Portal alongside your other audit records.

The feature is opt-in and controlled by a single connector setting. When it is off, on-demand syncs still run and apply grants exactly as before — only the per-resource ON_DEMAND audit is not produced.

There are two parts to this configuration:

1. Point the policysync_audit stream at S3, SQS, or SNS using the standard destination configuration (Choose your destinations).
2. Enable the ON_DEMAND audit on the connector so the events are produced (Setup).

Prerequisites¶

On-demand resource sync is configured and working on the connector, so the connector is polling the Ops Server for RESOURCE_SYNC tasks.

Choose your destinations¶

Choose where the audits go before you turn the feature on, so the events have somewhere to land as soon as the connector starts producing them.

ON_DEMAND events travel on the policysync_audit stream, so they follow the destinations you set for that stream. There is no separate on-demand destination list — configure the outputs once, and the on-demand events are included automatically.

You also need AUDITSERVER_POLICYSYNC_AUDIT_HANDLER_V2_ENABLED: true. See Connector Audit Output Destinations.

Setup¶

Follow these steps on the host where Privacera Manager is installed.

1. Sign in to that host.
2. Open the Manager configuration directory:

   Bash
   1	cd ~/privacera/privacera-manager/config

3. Copy the connector sample variables into custom-vars if you have not already. The -n flag avoids overwriting an existing file. Replace the file name with the one for your connector (for example, vars.connector.databricks.unity.catalog.yml).

   Bash
   1	cp -n sample-vars/vars.connector.databricks.unity.catalog.yml custom-vars/

4. Edit the connector custom variables file:

   Bash
   1	vi custom-vars/vars.connector.databricks.unity.catalog.yml

5. Enable the on-demand audit by adding (or updating) the following property. This renders the connector property policysync.on-demand.audit.enable=true.

   Property	Purpose	Privacera Manager default
   CONNECTOR_ON_DEMAND_AUDIT_ENABLE	Emit one ON_DEMAND audit event per requested resource, on the policysync_audit stream	false

   Leaving this false only suppresses the audit. Resource load, grant application, and the Ops Server task-status flow behave the same either way.

1

CONNECTOR_ON_DEMAND_AUDIT_ENABLE: "true"

1. Confirm on-demand processing is enabled for the connector (CONNECTOR_ON_DEMAND_PROCESSING_ENABLE: "true"); the audit is only produced when on-demand syncs actually run.
2. Apply the configuration with Privacera Manager, then restart the connector.

1
2
3

cd ~/privacera/privacera-manager
./privacera-manager.sh setup
./pm_with_helm.sh upgrade 

Event format¶

ON_DEMAND is one more eventType on the policysync_audit stream, so it uses the same wire contract as every other PolicySync connector event: one JSON object per event, UTF-8, camelCase property names, and UPPERCASE enum-like string values. Unknown root or extraInfo keys may appear in future releases — ignore or store them unless you explicitly handle them. See Policy Tracing and Explainability for the shared contract and the other event types.

The same payload shape is indexed into Apache Solr and forwarded to Amazon S3, Amazon SQS, and Amazon SNS.

Envelope fields¶

An ON_DEMAND event fixes several envelope fields to constant values, because it is always produced by the resource loader running from an on-demand task.

A few of these fields need more than a one-line description:

• eventType — a scheduled resource load reports LOADER instead, so filtering on ON_DEMAND selects only the syncs that were explicitly requested.
• status — the outcome for this one resource, so one request can produce both COMPLETED and FAILED events. The exception is a failed ACL operation, which counts for the whole task: every resource in the request that reached grant application is then FAILED, even if its own grants applied. See Outcome codes.
• msg — fixed text for the outcome, safe to show to a user. It never contains the target service's error text, which is in errorDetails.serviceError. The wording can change between releases, so code should read extraInfo.response[] instead.
• timeTakenMs — measured from the start of the resource load to the moment the event is produced.

Output: what the sync did¶

The extraInfo.output object describes what happened to this resource. The outcome codes are next to it, in extraInfo.response; see Outcome codes.

When a field is empty rather than absent:

• resources is {} when the resource was not found, when the load failed, or when the request could not be matched to a single resource.
• policies is [] when this resource did not change, because no grants were applied to it.
• aclsExecuted is absent when the resource did not change. It is present on successful events too, not only on failures.
• errorDetails is absent on a success, because there is nothing to report.

Resource counts¶

The value under output.resources.<type>. Each count is 0 or 1, because one event describes exactly one requested resource.

The policies that were applied¶

Each entry of output.policies:

ACL totals¶

The output.aclsExecuted object adds up the ACL operations of the policies applied in this on-demand sync. Read the counts here, whatever the outcome.

Absent when the resource did not change, because no grants ran for it.

Why the outcome failed¶

The output.errorDetails object explains a failure. It never appears on a successful outcome, and it does not appear on every failed one either — a resource that never reached grant application has no cause to report. When it is present, it carries only the parts that apply.

Which of the two appear depends on the outcome; see Outcome codes. topReasons is a summary, and policies[].failedReasons has the per-policy breakdown. ACL counts are in ACL totals, and the failure's codes in extraInfo.response[].

In serviceError, ARNs, email addresses, IPv4 addresses, request IDs, and quoted or dotted object names are replaced with ?. A name with none of these shapes, such as a single-word table name, is kept, so treat the text as diagnostic data.

Input: the original request¶

The extraInfo.input object echoes back the request this event answers, so you can match an event to its request without calling the Ops Server.

The source value is the OPS_BRIDGE_SOURCE name configured for the connector, so it names the path the request came in through — the REST API, MSK, or another configured source. See On-Demand Sync for PolicySync Connectors.

Each element of input.resources[]:

Inside input.resources[].values:

These two fields are per resource, not per request. If one request asks for several resources on behalf of different people, each event carries its own requestor.

Sample event¶

One resource (table1) added by on-demand task 6, with one policy applied:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

{
  "eventType": "ON_DEMAND",
  "evtTime": "2026-07-28T07:14:35Z",
  "timeTakenMs": 2140,
  "status": "COMPLETED",
  "msg": "Resource synced; access grants applied",
  "connectorType": "databricks_unity_catalog",
  "connector": "databricks-unity-catalog-instance1",
  "changeLogId": 18472,
  "changeLogType": "SERVICE_RESOURCES",
  "loaderType": "LOAD_SERVICE_RESOURCES",
  "triggerFrom": "EVENT",
  "ondemandTaskId": 6,
  "extraInfo": {
    "response": [
      {
        "code": 2001,
        "msg": "Resource synced; access grants applied",
        "origin": "DBX",
        "count": 3
      }
    ],
    "output": {
      "resources": {
        "table": {
          "found": 1,
          "added": 1,
          "updated": 0,
          "deleted": 0
        }
      },
      "policies": [
        {
          "id": "160",
          "version": "12",
          "label": "",
          "name": "policy1",
          "aclsExecuted": {
            "total": 3,
            "applied": 3,
            "failed": 0,
            "skipped": 0,
            "timeTakenMS": 717
          },
          "failedReasons": {},
          "skippedReasons": {},
          "ownerIgnore": {
            "count": 0
          }
        }
      ],
      "aclsExecuted": {
        "total": 3,
        "applied": 3,
        "failed": 0,
        "skipped": 0
      }
    },
    "input": {
      "createTime": 1785222862000,
      "syncType": "RESOURCE_SYNC",
      "source": "REST_API",
      "appType": "PS_CONNECTOR",
      "resources": [
        {
          "type": "table",
          "values": {
            "catalog": "catalog1",
            "schema": "schema1",
            "table": "table1",
            "requestor_id": "6bab473e-612f-450b-a99c-e6a1d5de2424",
            "requestor_create_time": "1785222861"
          }
        }
      ]
    }
  }
}

All three ACL operations applied, so response has one entry, 2001 SYNCED_GRANTS_APPLIED, with count 3.

A request for three resources produces three events like this one, all sharing ondemandTaskId, changeLogId, and timeTakenMs.

Outcome codes¶

Every requested resource gets one event. Its status and msg give the resource's outcome, and extraInfo.response[] lists its outcome codes, on success and on failure, in the same shape as on ACCESS and POLICY_RESPONSE events. Branch on response[].code, not on msg: the codes stay the same between releases, the wording may not. For the fields of an entry, see The response entry.

What response[] lists¶

• A resource that reached grant application (it changed, and the on-demand task ran ACL operations) lists the outcomes of the task's ACL operations, grouped, with a count each. For example, 2001 for an applied grant, 1004 for a grant skipped because the grantee owns the resource, or 5001 for a transient failure. The codes are in ON_DEMAND codes and template-based row filter codes; see also the limitation on which ACL operations are counted.
  • These entries cover the whole task, not this resource alone. Every changed resource in the request lists the same ones, so do not add up count across a request's events.
  • A resource whose outcome is RESOURCE_DELETED also lists 2004, count 1, for example [{2001, count 3}, {2004, count 1}]. If the load, grant application, or any ACL operation of the task failed, its outcome is that failure and it lists no 2004. To detect a deletion in every case, read output.resources.<type>.deleted.
• Any other resource lists its own outcome, one entry with count 1, from the table below.
• When the resource load fails, every resource in the request also lists the load failure's code: 4505 RESOURCE_LOAD_FAILED; 5001 SYSTEM_ERROR when it is transient; 1006 RESOURCE_NOT_FOUND or 4503 PRINCIPAL_NOT_FOUND when the backend reports a missing resource or principal.
• A FAILED event never lists only SUCCESS entries.

The resource-level codes:

Which resources a failure affects¶

Failed use-cases¶

A failed outcome always emits status = FAILED, a readable msg from the table above, its codes in extraInfo.response[], and — where there is a cause to report — the underlying error in errorDetails.serviceError.

Resource not found¶

output.resources and output.policies are empty, and there is no errorDetails: the resource never reached grant application. input still shows which resource was requested. response[] has one entry, 1006 RESOURCE_NOT_FOUND, category EXPECTED.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

{
  "eventType": "ON_DEMAND",
  "evtTime": "2026-07-28T07:15:34Z",
  "timeTakenMs": 403,
  "status": "FAILED",
  "msg": "Resource not found in source; nothing to apply",
  "connectorType": "databricks_unity_catalog",
  "connector": "databricks-unity-catalog-instance1",
  "changeLogId": 18479,
  "changeLogType": "SERVICE_RESOURCES",
  "loaderType": "LOAD_SERVICE_RESOURCES",
  "triggerFrom": "EVENT",
  "ondemandTaskId": 8,
  "extraInfo": {
    "response": [
      {
        "code": 1006,
        "msg": "Resource not found in source; nothing to apply",
        "origin": "DBX",
        "count": 1
      }
    ],
    "output": {
      "resources": {},
      "policies": []
    },
    "input": {
      "createTime": 1785222921000,
      "syncType": "RESOURCE_SYNC",
      "source": "REST_API",
      "appType": "PS_CONNECTOR",
      "resources": [
        {
          "type": "table",
          "values": {
            "catalog": "catalog1",
            "schema": "schema1",
            "table": "does_not_exist",
            "requestor_id": "7bfaed32-8c8d-40b8-b122-0580cae8cfcc",
            "requestor_create_time": "1785222921"
          }
        }
      ]
    }
  }
}

Resource load failed¶

The load did not complete, so every resource in the task is FAILED with the same code, and output.resources is empty on all of them. The load error itself is in errorDetails.serviceError, not in msg. This failure is not transient, so the response[] entry is 4505 RESOURCE_LOAD_FAILED; a transient load failure is 5001 SYSTEM_ERROR, which is worth retrying.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

{
  "eventType": "ON_DEMAND",
  "evtTime": "2026-07-28T07:16:10Z",
  "timeTakenMs": 512,
  "status": "FAILED",
  "msg": "Resource load failed",
  "connectorType": "databricks_unity_catalog",
  "connector": "databricks-unity-catalog-instance1",
  "changeLogId": 18485,
  "changeLogType": "SERVICE_RESOURCES",
  "loaderType": "LOAD_SERVICE_RESOURCES",
  "triggerFrom": "EVENT",
  "ondemandTaskId": 9,
  "extraInfo": {
    "response": [
      {
        "code": 4505,
        "msg": "Resource load failed",
        "origin": "DBX",
        "count": 1
      }
    ],
    "output": {
      "resources": {},
      "policies": [],
      "errorDetails": {
        "serviceError": "resource key persist failed"
      }
    },
    "input": {
      "createTime": 1785222970000,
      "syncType": "RESOURCE_SYNC",
      "source": "REST_API",
      "appType": "PS_CONNECTOR",
      "resources": [
        {
          "type": "table",
          "values": {
            "catalog": "catalog1",
            "schema": "schema1",
            "table": "table1",
            "requestor_id": "5f2a1c9e-3b7d-4e21-9a44-1c0b2d8e6f30",
            "requestor_create_time": "1785222969"
          }
        }
      ]
    }
  }
}

Grant application partially applied¶

The resource changed, so the event has its counts and policy list, as a COMPLETED event would.

• output.aclsExecuted shows how far the grants got: 1 of 3 ACL operations applied and 2 failed. That is a partial grant, so the outcome is GRANT_PARTIALLY_APPLIED, not GRANT_FAILED.
• The target's error text is in errorDetails.topReasons, and per policy in policies[].failedReasons.
• response[] lists both outcomes: one grant applied (2001), and two failed because the grantee does not exist (4503 PRINCIPAL_NOT_FOUND), which a retry does not fix.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96

{
  "eventType": "ON_DEMAND",
  "evtTime": "2026-07-28T07:16:44Z",
  "timeTakenMs": 1633,
  "status": "FAILED",
  "msg": "Grants partially applied; some grants failed",
  "connectorType": "databricks_unity_catalog",
  "connector": "databricks-unity-catalog-instance1",
  "changeLogId": 18490,
  "changeLogType": "SERVICE_RESOURCES",
  "loaderType": "LOAD_SERVICE_RESOURCES",
  "triggerFrom": "EVENT",
  "ondemandTaskId": 10,
  "extraInfo": {
    "response": [
      {
        "code": 4503,
        "msg": "Grantee principal not found in the target service",
        "origin": "DBX",
        "serviceCode": "PRINCIPAL_DOES_NOT_EXIST.PRINCIPAL_DOES_NOT_EXIST",
        "count": 2
      },
      {
        "code": 2001,
        "msg": "Resource synced; access grants applied",
        "origin": "DBX",
        "count": 1
      }
    ],
    "output": {
      "resources": {
        "table": {
          "found": 1,
          "added": 1,
          "updated": 0,
          "deleted": 0
        }
      },
      "policies": [
        {
          "id": "160",
          "version": "12",
          "label": "",
          "name": "policy1",
          "aclsExecuted": {
            "total": 3,
            "applied": 1,
            "failed": 2,
            "skipped": 0,
            "timeTakenMS": 604
          },
          "failedReasons": {
            "[PRINCIPAL_NOT_FOUND | retryable=false] PRINCIPAL_DOES_NOT_EXIST.PRINCIPAL_DOES_NOT_EXIST (State: 42704)": 2
          },
          "skippedReasons": {},
          "ownerIgnore": {
            "count": 0
          }
        }
      ],
      "aclsExecuted": {
        "total": 3,
        "applied": 1,
        "failed": 2,
        "skipped": 0
      },
      "errorDetails": {
        "topReasons": [
          {
            "reason": "[PRINCIPAL_NOT_FOUND | retryable=false] PRINCIPAL_DOES_NOT_EXIST.PRINCIPAL_DOES_NOT_EXIST (State: 42704)",
            "count": 2
          }
        ],
        "serviceError": "changelog status=FAILED"
      }
    },
    "input": {
      "createTime": 1785223004000,
      "syncType": "RESOURCE_SYNC",
      "source": "REST_API",
      "appType": "PS_CONNECTOR",
      "resources": [
        {
          "type": "table",
          "values": {
            "catalog": "catalog1",
            "schema": "schema1",
            "table": "table1",
            "requestor_id": "a71e0d64-9c52-4f18-8b30-2e6f5a1d7c93",
            "requestor_create_time": "1785223003"
          }
        }
      ]
    }
  }
}

Validate¶

In Privacera Portal¶

The audits are indexed into Apache Solr, which is what Privacera Portal reads, so you can confirm the feature is working from the Portal without querying anything directly.

1. Sign in to Privacera Portal.
2. From the application menu at the top, select Access Controls.
3. In the left navigation, select Audits.
4. Select the Policy Sync tab.
5. Adjust the date range at the top right if your sync is older than the default window, and use the search box to narrow the list.
6. The tab lists every PolicySync event, so narrow it to the rows where Event Type is ON_DEMAND.

Each row is one requested resource. Read the outcome from these columns:

To review one request end to end, note its Ondemand Task ID and look at every row sharing that value.

If a sync produced no ON_DEMAND rows at all, the audit is most likely not enabled on the connector — check CONNECTOR_ON_DEMAND_AUDIT_ENABLE.

In Prometheus¶

The connector publishes one counter for on-demand audit emission, so you can confirm events are being produced — and spot events that failed to reach the audit pipeline at all, which by definition never reach the Portal or Solr.

Useful queries:

1
2
3
4

sum by (result) (ps_core_ondemand_audit_events_total)
sum by (status) (ps_core_ondemand_audit_events_total{result="emitted"})
ps_core_ondemand_audit_events_total{result="failed"}
ps_core_ondemand_audit_events_total{path!="normal"}

Any non-zero result="failed" means an audit event was produced but never delivered — worth alerting on. A non-zero path!="normal" means something upstream failed while the audit was still emitted.

In Amazon SQS¶

Use the built-in test script inside the Audit Server diagnostics container to confirm ON_DEMAND records are arriving in your queue.

1. Open a shell inside the Audit Server diagnostics container.
2. Navigate to the SQS test directory:

   Bash
   1	cd /workdir/manual_test/read_sqs

3. Set the required environment variables. Replace <account-id> and <queue-name> with your AWS account ID and queue name.

   Bash
   1 2	export SQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/<account-id>/<queue-name> export DELETE_AFTER_PROCESSING="false"

4. Run the script:

   Bash
   1	./run_read_sqs.sh

The script prints the messages it reads. Look for records with "eventType":"ON_DEMAND" to confirm on-demand audits are flowing to the queue.

Guidelines¶

• The ON_DEMAND audit is per resource — an on-demand request for N resources produces N audit records. Keep this in mind when sizing a large batch, and watch the SQS/SNS 256 KB per-message limit for very large, policy-heavy syncs.
• Build consumers against extraInfo.response[], not msg. New codes may be added across releases. See Response codes.
• Enabling CONNECTOR_ON_DEMAND_AUDIT_ENABLE does not change the Ops Server / task-status flow; that continues to work whether the flag is on or off.
• If you filter an Amazon SNS subscription by event type, allow eventType = ON_DEMAND — otherwise the on-demand events are dropped before they reach the subscribed queue even though the connector emitted them.