Skip to content

Setup for Access Management for Databricks all-purpose compute clusters with Fine-Grained Access Control (FGAC)

Configure

Perform the following steps to configure Databricks FGAC:

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to navigate to the /config directory.

    Bash
    1
    cd ~/privacera/privacera-manager/config

    Note

    JWT Authentication is enabled by-default for Databricks FGAC Plugin and thus needs JWT to be configured. Follow the steps given here to configure JWT for authentication.

  3. Run the following command to copy the databricks FGAC plugin yml file from sample-vars, if it's not already present in custom-vars:

    Bash
    1
    cp -n sample-vars/vars.databricks.plugin.yml custom-vars/vars.databricks.plugin.yml

  4. Update the following properties in the vars.databricks.plugin.yml file:

    Bash
    1
2
3
4
    vi custom-vars/vars.databricks.plugin.yml

DATABRICKS_HOST_URL: "<PLEASE_UPDATE>"
DATABRICKS_TOKEN: "<PLEASE_UPDATE>"

  5. Once the properties are configured, run the Privacera Manager post-install action Refer this

Enable Databricks Application

  1. In PrivaceraCloud, go to Settings -> Applications.
  2. On the Applications screen, select Databricks.
  3. Select Platform Type
  4. Enter the application Name and Description. Click Save. The name can be any name of your choice, e.g., Databricks.
  5. Open the Databricks application.
  6. Enable the Access Management option with toggle button.
  7. Click on save button

Note

JWT Authentication is enabled by-default for Databricks FGAC Plugin and thus needs JWT to be configured. Follow the steps given here to configure JWT for authentication.

Download Script from PrivaceraCloud

  1. Open the Databricks application.
  2. Click on Access Management
  3. Click on Download Script
  4. Save the script locally as privacera_databricks.sh

Create Init script in Databricks Workspace File

  1. Login Databricks Web UI
  2. Click on the Workspace sidebar
  3. Click on the Workspace folder
  4. Click on Create -> Folder
  5. Name the folder privacera
  6. Click on Create
  7. Go inside the privacera folder
  8. Click on Create -> File
  9. Enter the new file name as privacera_databricks.sh
  10. Copy the content of the local privacera_databricks.sh and paste it into the workspace file privacera_databricks.sh

Configure the connector in Runtime and download it as a bundle. The bundle includes the init script that installs the plugin when the cluster starts. Upload that script to the workspace or to S3, then attach it to the cluster.

Note

Runtime support for this connector is available on Self-Managed and D2P planes only.

Configure the Databricks Spark connector

  1. In the portal, open the runtime plane and go to the Connector tab. If no runtime plane exists yet, see Getting Started with Runtime Plane.

  2. Add the Databricks Spark connector. Under Access Control Configs -> Connector Configs, set the following:

    Field Definition Example
    Access control mode Fine-Grained Access Control (FGAC). Fixed for this connector; OLAC is not supported here. FGAC
    Enable JWT authentication Enables JWT Authentication and identifies the querying user from a JWT. false

  3. JWT Authentication - only if Enable JWT authentication is on. For the provider fields, see Configuring JWT for Authentication.

  4. Logging - only if Enable debug logging is on. For the fields and how to collect the logs afterwards, see Databricks enable logs.

  5. Custom Properties - optional. Additional properties can be added here as key-value pairs.

    For example:

    Key Value
    spark.example.property.enabled true

  6. Advanced Configuration - optional. Settings this form does not carry are set on the Databricks cluster or under Custom Properties. See Advanced Configuration.

  7. Save the configuration, then download the connector bundle.

    The bundle contents depend on the plane, because certificates are shipped only where the plugin needs them:

    Plane Bundle contents
    Self-Managed trust3ai_databricks_spark_plugin_<connector>_<version>.sh and certs.zip
    D2P trust3ai_databricks_spark_plugin_<connector>_<version>.sh only

Upload the init script

Upload the downloaded trust3ai_databricks_spark_plugin_<connector>_<version>.sh to a location the cluster can read - a folder in the Databricks workspace, or an S3 path such as s3://<bucket>/privacera/trust3ai_databricks_spark_plugin_<connector>_<version>.sh.

Note

For an S3 path, the cluster's instance profile needs s3:GetObject permission on it.

When creating the cluster, set the init script Source to match - Workspace or S3 - and give the path used above.

Upload the certificates

Self-Managed planes only. On D2P the bundle carries no certs.zip, and this section does not apply.

  1. Upload certs.zip from the bundle to a location the cluster can read - a DBFS path such as /dbfs/privacera/certs.zip, or an S3 path such as s3://<bucket>/privacera/certs.zip.

  2. Set TRUST3AI_CERTS_SOURCE_PATH on the cluster to that location. The init script reads it to fetch the certificates.

    Source Example
    DBFS /dbfs/privacera/certs.zip, or /dbfs/privacera - a path that does not end in .zip is treated as the folder holding it
    S3 s3://<bucket>/privacera/certs.zip - this must be the file itself, not the folder

    Note

    For an S3 path, the cluster's instance profile needs s3:GetObject permission on it.

Set a plugin property

Additional properties can be added under Custom Properties as key-value pairs, using the full property name as it appears in privacera_spark.properties. Runtime appends them after the connector's own settings, and the plugin keeps the last line for a key, so a custom property overrides the shipped default.

  1. In the portal, open the connector and go to Custom Properties.
  2. Enter the property name and value, for example privacera.spark.deny.unsupported.sql.primitives.enabled and true.
  3. Save the connector, then download the bundle again.
  4. Upload the init script and restart the clusters that use it.

A custom property applies to every cluster that uses this connector's bundle, and only reaches the plugin through a freshly downloaded bundle.

Note

privacera.deployment.type cannot be set this way - the mode comes from the connector and the installer refuses it.

Create Databricks cluster policy

We recommend to use Databricks cluster policy to control cluster configuration. Here are the steps to create cluster policy:

  • Log in to Databricks Web UI
  • Click on the Compute icon on the sidebar
  • Click on the Policies tab
  • Click on the Create policy button
  • Provide a name to the policy i.e privacera-fgac-cluster-policy
  • Add the appropriate policy definition (JSON) based on your Databricks Runtime version.

  • Replace <instance-profile-arn> with your actual instance profile ARN.

    Cluster policy JSON for Databricks Runtime Version 17.3 LTS
    JSON
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
    {
  "spark_conf.spark.databricks.isv.product": {
    "type": "fixed",
    "value": "privacera"
  },
  "spark_conf.spark.driver.extraJavaOptions": {
    "type": "fixed",
    "value": "-javaagent:/databricks/jars/privacera-agent.jar"
  },
  "spark_conf.spark.databricks.repl.allowedLanguages": {
    "type": "fixed",
    "value": "sql,python,r"
  },
  "spark_conf.spark.databricks.delta.formatCheck.enabled": {
    "type": "fixed",
    "value": "false"
  },
  "aws_attributes.instance_profile_arn": {
    "type": "fixed",
    "value": "<instance-profile-arn>"
  },
  "spark_conf.spark.sql.extensions": {
    "type": "fixed",
    "value": "com.privacera.spark4x.agent.Spark4xSQLExtension"
  },
  "spark_conf.spark.databricks.pyspark.dbconnect.enableProcessIsolation": {
    "type": "fixed",
    "value": "true"
  },
  "spark_conf.spark.databricks.pyspark.enableProcessIsolation": {
    "type": "fixed",
    "value": "true"
  },
  "spark_conf.spark.databricks.pyspark.enablePy4JSecurity": {
    "type": "fixed",
    "value": "true"
  },
  "spark_conf.spark.hadoop.privacera.custom.current_user.udf.names": {
    "type": "fixed",
    "value": "current_user()"
  }
}
    Cluster policy JSON for Databricks Runtime Version 16.4 LTS and below
    JSON
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
    {
  "spark_conf.spark.databricks.isv.product": {
    "type": "fixed",
    "value": "privacera"
  },
  "spark_conf.spark.driver.extraJavaOptions": {
    "type": "fixed",
    "value": "-javaagent:/databricks/jars/privacera-agent.jar"
  },
  "spark_conf.spark.databricks.repl.allowedLanguages": {
    "type": "fixed",
    "value": "sql,python,r"
  },
  "spark_conf.spark.databricks.delta.formatCheck.enabled": {
    "type": "fixed",
    "value": "false"
  },
  "aws_attributes.instance_profile_arn": {
    "type": "fixed",
    "value": "<instance-profile-arn>"
  },
  "spark_conf.spark.sql.extensions": {
    "type": "fixed",
    "value": "com.privacera.spark.agent.SparkSQLExtension"
  },
  "spark_conf.spark.databricks.pyspark.dbconnect.enableProcessIsolation": {
    "type": "fixed",
    "value": "true"
  },
  "spark_conf.spark.databricks.pyspark.enableProcessIsolation": {
    "type": "fixed",
    "value": "true"
  },
  "spark_conf.spark.databricks.pyspark.enablePy4JSecurity": {
    "type": "fixed",
    "value": "true"
  },
  "spark_conf.spark.hadoop.privacera.custom.current_user.udf.names": {
    "type": "fixed",
    "value": "current_user()"
  }
}

  • Save the policy.

Create Databricks cluster

Here are the steps to create Databricks cluster with Privacera plugin (FGAC):

  1. Log in to Databricks Web UI
  2. Click on the Compute icon on the sidebar
  3. Click the Create Compute button.
  4. Fill in the cluster configuration details.
  5. Under the Cluster Policies dropdown, select the policy you created in the previous step (i.e., privacera-fgac-cluster-policy)
  6. Under the Advanced option :
    • Select Workspace as the source for init scripts.
    • Specify the Workspace file path:
      • Self Managed
        • /privacera/{DEPLOYMENT_ENV_NAME}/ranger_enable.sh
      • PrivaceraCloud and Data Plane
        • /privacera/{DEPLOYMENT_ENV_NAME}/privacera_databricks.sh
      • Runtime
        • The path where you uploaded trust3ai_databricks_spark_plugin_<connector>_<version>.sh. See Upload the init script; for an S3 path, select S3 as the source instead.
  7. Click the Add button.
  8. Click Create Compute to create the cluster.

Validation

To confirm the successful association of an access management policy with data in your Databricks installation, follow these steps:

  1. Prerequisites:

    • A running Databricks cluster secured using the steps mentioned above.
    • At least one resource policy associated with your data that grants a user access to the database.
    • This resource policy must not be for Databrick's default database. Configure the policy for any database other than the default.

    • Sample Resource Policy:

      • Policy Name: Employee Details
      • Hive Database: sample_db
      • Hive Table: employees
      • Permissions: CREATE, UPDATE, SELECT, DROP
      • Select User: emily

      Sample resource policy

  2. Steps to Validate Policy:

    • Login to Databricks as a user who is defined in the resource policy.
    • Create or open an existing notebook. Associate the Notebook with the running Databricks cluster.
    • Select the database to which you have associated the policy.
    • In the notebook, run following SQL commands to Create a sample table, Insert and Select the data:
      SQL
       1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
      USE sample_db;

CREATE TABLE IF NOT EXISTS employees (
Emp_Id INT,
First_name STRING,
Last_name STRING
);

INSERT INTO employees (Emp_Id, First_name, Last_name) VALUES (1, 'John', 'Doe'), 
(2, 'Jane', 'Smith'), (3, 'Alice', 'Johnson');

SELECT * FROM employees;
    • On Privacera portal, go to Access Management -> Audits
    • Check for the success or failure of the resource policy. A successful access is indicated as Allowed and failure is indicated as Denied.

    • Sample Audit Logs:

      Sample audit logs