Skip to content

Setup for Access Management for Apache Spark

Configure

This section outlines the steps to set up the Apache Spark OLAC or OLAC_FGAC with the Privacera Plugin. Ensure that all prerequisites are completed before beginning the setup process.

Perform following steps to configure Apache Spark OLAC connector:

  1. SSH into the instance where Privacera Manager is installed.

  2. Run the following command to navigate to the /config directory and copy yml files: 

    cd ~/privacera/privacera-manager/config
cp sample-vars/vars.spark-standalone.yml custom-vars/

  3. Modify the following properties:

    • In the vars.spark-standalone.yml file, update the following properties with the appropriate values: 

      #Spark Env name.
SPARK_ENV_TYPE: "<PLEASE_CHANGE>"

#Add the spark home path
#Eg: "/opt/spark"
SPARK_HOME: "<PLEASE_CHANGE>"

#Spark user home directory
#Eg: "/opt/privacera"
SPARK_USER_HOME: "<PLEASE_CHANGE>"

      Variable Definition Example
      SPARK_ENV_TYPE Set the environment type. privacera_spark_olac
      SPARK_HOME Home path of your Spark installation. /opt/spark
      SPARK_USER_HOME User home directory of your Spark installation. /opt/privacera

  4. Once the properties are configured, update your Privacera Manager platform instance by following the commands

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

  5. Once the post-install process is complete, you will see spark-standalone folder in the ~/privacera/privacera-manager/output directory, with the following folder structure: 

    output/
├── spark-standalone/
│   ├── spark_custom_conf/ 
│   │   ├── FGAC/ 
│   │   │   ├── README.md
│   │   │   ├── jwttoken.pub
│   │   │   ├── privacera_spark.properties
│   │   ├── OLAC/
│   │   │   ├── privacera_spark.properties
│   │   │   ├── README.md
│   │   ├── OLAC_FGAC/
│   │   │   ├── README.md
│   │   │   ├── jwttoken.pub
│   │   │   ├── privacera_spark.properties
│   │   ├── auditserver-secrets-keystore.jks
│   │   ├── global-truststore.p12
│   │   ├── ranger-plugin-keystore.p12
│   │   ├── ranger.jceks
│   ├── privacera_setup.sh
│   ├── spark_custom_conf.zip
│   ├── standalone_spark_FGAC.sh
│   ├── standalone_spark_OLAC.sh
│   ├── standalone_spark_OLAC_FGAC.sh

Setup Scripts and Configuration Files

Creating Script Files

  1. Create privacera-oss-plugin folder and navigate to it 

    mkdir ~/privacera-oss-plugin
cd ~/privacera-oss-plugin

  2. Create a penv.sh file and copy the following content to it 

    cd ~/privacera-oss-plugin
vi penv.sh

    penv.sh
    penv.sh
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
    #!/bin/bash

## Set up environment variables for Privacera Spark plugin deployment


# Privacera configuration
export PRIVACERA_BASE_DOWNLOAD_URL="<PRIVACERA_BASE_DOWNLOAD_URL>"
# Supported spark-plugin types are: OLAC, OLAC_FGAC
export PRIVACERA_SPARK_PLUGIN_TYPE="OLAC"

# Uncomment this to enable delta lake support
#export OSS_DELTA_LAKE_ENABLE="true"

# Uncomment this to enable iceberg support
#export OSS_ICEBERG_ENABLE="true"
# Set Iceberg warehouse path
#export ICEBERG_WAREHOUSE_PATH="s3a://your-bucket/iceberg-warehouse"
# Set Iceberg catalog type (default: hadoop)
#export ICEBERG_CATALOG_TYPE="hadoop"

# Uncomment this to enable s3 table support
#export OSS_S3_TABLE_ENABLE="true"
# Set S3 Table warehouse path
#export S3_TABLE_WAREHOUSE="arn:aws:s3tables:<region>:<account-id>:bucket/<bucket-name>"

# Uncomment below to enable Privacera Distributor Plugin
#export OSS_SPARK_PRIVACERA_DISTRIBUTOR_PLUGIN_ENABLE="true"

# Uncomment below to enable Privacera Boto3 Signer support in the Spark plugin image.
# When enabled, the Boto3 Signer wheel is downloaded and baked into the image so
# Python boto3 S3 calls are routed through Privacera. Disabled by default.
#export OSS_BOTO3_SIGNER_ENABLE="true"
# Deployment type used to download the Boto3 Signer package. Supported: PLATFORM (Self-Managed)
export PRIVACERA_DEPLOYMENT_TYPE="PLATFORM"
# Privacera release version used to resolve the Boto3 Signer package (e.g. 9.2.30.1)
#export PRIVACERA_RELEASE_VERSION="<PRIVACERA_RELEASE_VERSION>"

# Spark configuration
# Spark Docker image (e.g. spark:3.5.5-python3, apache/spark:4.0.2-python3)
export SPARK_BASE_IMAGE="<SPARK_BASE_IMAGE>"
# Spark version (e.g., 3.5.5 or 4.0.2)
export SPARK_VERSION="<SPARK_VERSION>"


# Docker image configuration
export SPARK_PLUGIN_IMAGE="<SPARK_PLUGIN_IMAGE>"
export SPARK_DOCKER_PULL_SECRET="<SPARK_DOCKER_PULL_SECRET>"
export TAG="<DOCKER_IMAGE_TAG>"


## K8S configuration
export SPARK_NAME_SPACE="<SPARK_NAME_SPACE>"
export PRIVACERA_SECRET_NAME="privacera-spark-secret"
export PRIVACERA_CONFIGMAP_NAME="privacera-spark-configmap"

export SPARK_PLUGIN_ROLE_BINDING="privacera-sa-spark-plugin-role-binding"
export SPARK_PLUGIN_SERVICE_ACCOUNT="privacera-sa-spark-plugin"
export SPARK_PLUGIN_ROLE="privacera-sa-spark-plugin-role"
export SPARK_PLUGIN_APP_NAME="privacera-spark-examples"


# Input and output directories for k8s templates
export INPUT_DIR="./templates"
export OUTPUT_DIR="./output"

    • To get the PRIVACERA_BASE_DOWNLOAD_URL run the following command where Privacera is installed. Update this download URL in penv.sh script file 

      grep -i 'PRIVACERA_BASE_DOWNLOAD_URL=' ~/privacera/privacera-manager/output/spark-standalone/standalone_spark_OLAC.sh

      Note

      To enable OLAC_FGAC update the PRIVACERA_SPARK_PLUGIN_TYPE in the penv.sh script as shown below 

      PRIVACERA_SPARK_PLUGIN_TYPE="OLAC_FGAC"

  3. Run the following command to create and edit build_privacera_plugin.sh file. Expand the following section and copy it's content to the build_privacera_plugin.sh file 

    vi build_privacera_plugin.sh

    build_privacera_plugin.sh
    build_privacera_plugin.sh
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
    #!/bin/bash

set -x

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
ENV_FILE="${SCRIPT_DIR}/penv.sh"

# if penv.sh exists, load it
if [ -f ${ENV_FILE} ]; then
  echo "Sourcing env file ${ENV_FILE}"
  source ${ENV_FILE}
else
  echo "${ENV_FILE} file not found"
  exit 1
fi

SPARK_HOME=${SCRIPT_DIR}/spark_home/opt/spark
PRIVACERA_HOME=${SCRIPT_DIR}/privacera_home
PRIVACERA_WORK_DIR=${PRIVACERA_HOME}/work

PRIVACERA_SPARK_PLUGIN_TYPE=${PRIVACERA_SPARK_PLUGIN_TYPE}
ENV_TYPE=PLATFORM
SPARK_DEFAULT_CONF=${SPARK_HOME}/conf/spark-defaults.conf
USER_NAME=spark
GROUP_NAME=spark

export PRIVACERA_HOME=${PRIVACERA_HOME}
export PRIVACERA_WORK_DIR=${PRIVACERA_WORK_DIR}
export PRIVACERA_DOWNLOAD=$PRIVACERA_HOME/downloads
export SPARK_DEFAULT_CONF=${SPARK_DEFAULT_CONF}
export PRIVACERA_SPARK_PLUGIN_TYPE=${PRIVACERA_SPARK_PLUGIN_TYPE}
export PRIVACERA_BASE_DOWNLOAD_URL=${PRIVACERA_BASE_DOWNLOAD_URL}
export ENV_TYPE=${ENV_TYPE}
export OSS_DELTA_LAKE_ENABLE=${OSS_DELTA_LAKE_ENABLE}
export OSS_ICEBERG_ENABLE=${OSS_ICEBERG_ENABLE}
export ICEBERG_WAREHOUSE_PATH=${ICEBERG_WAREHOUSE_PATH}
export ICEBERG_CATALOG_TYPE=${ICEBERG_CATALOG_TYPE}
export OSS_S3_TABLE_ENABLE=${OSS_S3_TABLE_ENABLE}
export S3_TABLE_WAREHOUSE=${S3_TABLE_WAREHOUSE}
export OSS_SPARK_PRIVACERA_DISTRIBUTOR_PLUGIN_ENABLE=${OSS_SPARK_PRIVACERA_DISTRIBUTOR_PLUGIN_ENABLE}
export OSS_BOTO3_SIGNER_ENABLE=${OSS_BOTO3_SIGNER_ENABLE}
export PRIVACERA_DEPLOYMENT_TYPE=${PRIVACERA_DEPLOYMENT_TYPE}
export PRIVACERA_RELEASE_VERSION=${PRIVACERA_RELEASE_VERSION}
export SPARK_VERSION=${SPARK_VERSION}
export PRIVACERA_SETUP_SCRIPT="privacera_setup.sh"
export SPARK_USER=${USER_NAME}
export SPARK_GROUP=${GROUP_NAME}
export SPARK_HOME=${SPARK_HOME}

rm -rf ${PRIVACERA_HOME}
rm -rf ${SPARK_HOME}
# clean up any boto3 signer wheel staged by a previous build
rm -rf ${SCRIPT_DIR}/spark_home/opt/privacera/boto3_signer

mkdir -p ${PRIVACERA_HOME}
mkdir -p ${SPARK_HOME}
mkdir -p ${SPARK_HOME}/jars
mkdir -p ${SPARK_HOME}/conf

touch ${SPARK_HOME}/conf/spark-defaults.conf

# run the actual script that downloads and installs the plugin
bash scripts/privacera_setup.sh

# download and stage the Privacera Boto3 Signer wheel when enabled in penv.sh (Self-Managed only)
if [ "${OSS_BOTO3_SIGNER_ENABLE}" = "true" ]; then
  bash scripts/privacera_boto3_signer_setup.sh
fi

cp ${PRIVACERA_HOME}/work/privacera_version.txt ${SPARK_HOME}/conf
sed -i "s|${SPARK_HOME}|/opt/spark|g" ${SPARK_HOME}/conf/spark-defaults.conf

tar cvfz privacera-spark-plugin.tar.gz -C ${SPARK_HOME}/../.. .

  4. Create scripts folder and navigate to it 

    mkdir ~/privacera-oss-plugin/scripts
cd ~/privacera-oss-plugin/scripts

  5. Create privacera_setup.sh script file and copy the following content to it 

    vi privacera_setup.sh

    privacera_setup.sh
    privacera_setup.sh
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
    #!/bin/bash

set -x

echo "SPARK_DEFAULT_CONF ${SPARK_DEFAULT_CONF}"
echo "SPARK_HOME ${SPARK_HOME}"
echo "PRIVACERA_SPARK_PLUGIN_TYPE ${PRIVACERA_SPARK_PLUGIN_TYPE}"
echo "PRIVACERA_BASE_DOWNLOAD_URL ${PRIVACERA_BASE_DOWNLOAD_URL}"
echo "SPARK_VERSION ${SPARK_VERSION}"

export ENV_TYPE=${ENV_TYPE:-"PLATFORM"}
export OSS_DELTA_LAKE_ENABLE=${OSS_DELTA_LAKE_ENABLE:-false}
export OSS_ICEBERG_ENABLE=${OSS_ICEBERG_ENABLE:-false}
export ICEBERG_WAREHOUSE_PATH=${ICEBERG_WAREHOUSE_PATH:-""}
export ICEBERG_CATALOG_TYPE=${ICEBERG_CATALOG_TYPE:-"hadoop"}
export OSS_S3_TABLE_ENABLE=${OSS_S3_TABLE_ENABLE:-false}
export S3_TABLE_WAREHOUSE=${S3_TABLE_WAREHOUSE:-""}
export SPARK_VERSION=${SPARK_VERSION}

# spark folders
export SPARK_PLUGIN_TYPE=${PRIVACERA_SPARK_PLUGIN_TYPE}
SCRIPT_NAME=standalone_spark_setup.sh

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"

chmod +x ${SCRIPT_DIR}/${SCRIPT_NAME}
. ${SCRIPT_DIR}/${SCRIPT_NAME}

  6. Create standalone_spark_setup.sh script file and copy the following content to it 

    vi standalone_spark_setup.sh

    standalone_spark_setup.sh
    standalone_spark_setup.sh
      1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
    #!/bin/bash

set -x

#no need to change
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
PRIVACERA_WORK_DIR=${PRIVACERA_WORK_DIR:-${SCRIPT_DIR}/work}
PRIVACERA_OUT_FILE=${PRIVACERA_OUT_FILE:-${PRIVACERA_WORK_DIR}/privacera.out}
PKG_NAME=${PKG_NAME:-privacera-spark-plugin.tar.gz}
PRIVACERA_CONF_FOLDER=${PRIVACERA_CONF_FOLDER:-${SPARK_HOME}/conf}
PRIVACERA_CUSTOM_CONF_ZIP=${PRIVACERA_CUSTOM_CONF_ZIP:-spark_custom_conf.zip}
SPARK_DEFAULT_CONF=${SPARK_HOME}/conf/spark-defaults.conf
SPARK_PLUGIN_TYPE=${SPARK_PLUGIN_TYPE:-"OLAC"}
OSS_DELTA_LAKE_ENABLE=${OSS_DELTA_LAKE_ENABLE:-false}
OSS_ICEBERG_ENABLE=${OSS_ICEBERG_ENABLE:-false}
ICEBERG_WAREHOUSE_PATH=${ICEBERG_WAREHOUSE_PATH:-""}
ICEBERG_CATALOG_TYPE=${ICEBERG_CATALOG_TYPE:-"hadoop"}
OSS_S3_TABLE_ENABLE=${OSS_S3_TABLE_ENABLE:-false}
S3_TABLE_WAREHOUSE=${S3_TABLE_WAREHOUSE:-""}
ENV_TYPE=${ENV_TYPE:-"PLATFORM"}
SPARK_VERSION=${SPARK_VERSION}
OSS_SPARK_PRIVACERA_DISTRIBUTOR_PLUGIN_ENABLE=${OSS_SPARK_PRIVACERA_DISTRIBUTOR_PLUGIN_ENABLE:-false}

if [[ $ENV_TYPE != "PCLOUD" ]]; then
  # for platform we have to append as we are using same path for both
  export PRIVACERA_BASE_DOWNLOAD_URL=${PRIVACERA_BASE_DOWNLOAD_URL}/spark-plugin
fi

rm -rf ${PRIVACERA_OUT_FILE}
function log(){
  msg=$1
  currentTime=`date`
  echo "${currentTime} : ${msg} " >> ${PRIVACERA_OUT_FILE}
}

function createFolders() {
  #delete existing jars and work folder so update would not create issue as jar name is different in every release
  rm -rf ${SPARK_HOME}/jars/ranger-*
  rm -rf ${SPARK_HOME}/jars/privacera*
  rm -rf ${PRIVACERA_WORK_DIR}
  mkdir -p ${PRIVACERA_WORK_DIR}
}

function validation(){
  if [[ $SPARK_HOME == "" ]];then
    log " SPARK_HOME can't be empty "
    exit 1;
  fi
  log "SPARK_HOME = ${SPARK_HOME}"
}

download_pkg(){
  log "Downloading pkg from ${PRIVACERA_BASE_DOWNLOAD_URL}/${PKG_NAME}"
  cd ${PRIVACERA_WORK_DIR}
  if [[ ${PRIVACERA_BASE_DOWNLOAD_URL} == https* ]]; then
    log "Downloading pkg using wget ${PRIVACERA_BASE_DOWNLOAD_URL}/${PKG_NAME}"
    wget -nv ${PRIVACERA_BASE_DOWNLOAD_URL}/${PKG_NAME} -O ${PKG_NAME}
  else
    log "download pkg path is not yet supported ${PRIVACERA_BASE_DOWNLOAD_URL}/${PKG_NAME}"
  fi
}

function install() {
  log "Extracting ${PRIVACERA_WORK_DIR}/${PKG_NAME} into ${PRIVACERA_WORK_DIR}"
  tar xzf ${PRIVACERA_WORK_DIR}/${PKG_NAME} --directory ${PRIVACERA_WORK_DIR}

  #putting privacera jar into spark home
  cp -r ${PRIVACERA_WORK_DIR}/spark-plugin/* $SPARK_HOME/jars/

  if [ ! -f $SPARK_DEFAULT_CONF ]; then
   cp $SPARK_DEFAULT_CONF.template $SPARK_DEFAULT_CONF
  fi
}

function update_jars() {
  # find java major version
  JAVA_MAJOR_VERSION=$(java -version 2>&1 | sed -E -n 's/.* version "([^.-]*).*"/\1/p' | cut -d' ' -f1)
  echo "Java major version: ${JAVA_MAJOR_VERSION}"
  if [ ${JAVA_MAJOR_VERSION} -ge 15 ]; then
    cp -r ${PRIVACERA_WORK_DIR}/jdk15-jars/* $SPARK_HOME/jars/ranger-spark-plugin-impl/
  fi
}

function configure() {
    log "Configure started"

    #  NOTE: Equivalent of this will be done in the entrypoint script
    PRIVACERA_CONF_WORK_DIR="${PRIVACERA_WORK_DIR}/spark-conf"

    cp ${PRIVACERA_CONF_WORK_DIR}/resource_type_plugin_map.json ${PRIVACERA_CONF_FOLDER}/

    #setting spark agent
    if [ `grep -rin "spark.driver.extraJavaOptions" $SPARK_DEFAULT_CONF |  wc -l` -gt 0 ];then
      sed -i '' -e 's|spark.driver.extraJavaOptions .*|spark.driver.extraJavaOptions -javaagent:'${SPARK_HOME}'/jars/privacera-agent.jar -Dlog4j2.configurationFile=file:///privacera-conf/log4j2.properties|g' $SPARK_DEFAULT_CONF
    else
      echo "spark.driver.extraJavaOptions -javaagent:${SPARK_HOME}/jars/privacera-agent.jar -Dlog4j2.configurationFile=file:///privacera-conf/log4j2.properties" >> $SPARK_DEFAULT_CONF
    fi

    if [[ "${SPARK_PLUGIN_TYPE}" == *"FGAC"* ]]; then
      log "Spark python/sql plugin "
      if [ `grep -rin "spark.sql.extensions" $SPARK_DEFAULT_CONF |  wc -l` -gt 0 ];then
        sed -i '' -e 's|spark.sql.extensions.*|spark.sql.extensions com.privacera.spark.agent.SparkSQLExtension|g' $SPARK_DEFAULT_CONF
      else
        echo "#Setting Privacera spark-plugin properties" >> $SPARK_DEFAULT_CONF
        echo "spark.sql.extensions com.privacera.spark.agent.SparkSQLExtension" >> $SPARK_DEFAULT_CONF
      fi
    elif [[ "${SPARK_PLUGIN_TYPE}" == *"OLAC"* ]]; then
      echo "Setting agent for executor"
        #setting spark agent
        if [ `grep -rin "# spark.executor.extraJavaOptions" $SPARK_DEFAULT_CONF |  wc -l` -gt 0 ];then
          sed -i '' -e 's|# spark.executor.extraJavaOptions .*|spark.executor.extraJavaOptions -javaagent:'${SPARK_HOME}'/jars/privacera-agent.jar -Dlog4j2.configurationFile=file:///privacera-conf/log4j2.properties|g' $SPARK_DEFAULT_CONF
        elif [ `grep -rin "spark.executor.extraJavaOptions" $SPARK_DEFAULT_CONF |  wc -l` -gt 0 ];then
          sed -i '' -e 's|spark.executor.extraJavaOptions .*|spark.executor.extraJavaOptions -javaagent:'${SPARK_HOME}'/jars/privacera-agent.jar|g -Dlog4j2.configurationFile=file:///privacera-conf/log4j2.properties' $SPARK_DEFAULT_CONF
        else
          echo "spark.executor.extraJavaOptions -javaagent:${SPARK_HOME}/jars/privacera-agent.jar -Dlog4j2.configurationFile=file:///privacera-conf/log4j2.properties " >> $SPARK_DEFAULT_CONF
        fi
    fi

    ## add or append 'spark.sql.hive.metastore.sharedPrefixes'
    if [ `grep -rin "^spark.sql.hive.metastore.sharedPrefixes" $SPARK_DEFAULT_CONF |  wc -l` -gt 0 ];then
      sharedPrefixes=`grep -ri "^spark.sql.hive.metastore.sharedPrefixes" $SPARK_DEFAULT_CONF`
      sharedPrefixes="${sharedPrefixes#spark.sql.hive.metastore.sharedPrefixes}"
      # trim leading whitespaces ###
      sharedPrefixes="${sharedPrefixes##*( )}"
      # trim trailing whitespaces  ##
      sharedPrefixes="${sharedPrefixes%%*( )}"
      sed -i -e "s|^spark.sql.hive.metastore.sharedPrefixes .*|#spark.sql.hive.metastore.sharedPrefixes ${sharedPrefixes}|g" $SPARK_DEFAULT_CONF
      echo "spark.sql.hive.metastore.sharedPrefixes ${sharedPrefixes},com.privacera,com.amazonaws" >> $SPARK_DEFAULT_CONF
    else
      echo "spark.sql.hive.metastore.sharedPrefixes com.privacera,com.amazonaws" >> $SPARK_DEFAULT_CONF
    fi

    log "Configure completed"
}

function setup_deltalake() {
    # configure deltalake support
    log "setup_deltalake started"

    if [ "${OSS_DELTA_LAKE_ENABLE}" == true ]; then

        if [ "${SPARK_VERSION}" = "3.4.0" ] || [ "${SPARK_VERSION}" = "3.4.1" ]; then
            wget https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.4.0/delta-core_2.12-2.4.0.jar -P "${SPARK_HOME}/jars/"
            wget https://repo1.maven.org/maven2/io/delta/delta-storage/2.4.0/delta-storage-2.4.0.jar -P "${SPARK_HOME}/jars/"

        elif [ "${SPARK_VERSION}" = "3.5.0" ]; then
            wget https://repo1.maven.org/maven2/io/delta/delta-spark_2.12/3.0.0/delta-spark_2.12-3.0.0.jar -P "${SPARK_HOME}/jars/"
            wget https://repo1.maven.org/maven2/io/delta/delta-storage/3.0.0/delta-storage-3.0.0.jar -P "${SPARK_HOME}/jars/"

        elif [ "${SPARK_VERSION}" = "3.5.1" ]; then
            wget https://repo1.maven.org/maven2/io/delta/delta-spark_2.12/3.1.0/delta-spark_2.12-3.1.0.jar -P "${SPARK_HOME}/jars/"
            wget https://repo1.maven.org/maven2/io/delta/delta-storage/3.1.0/delta-storage-3.1.0.jar -P "${SPARK_HOME}/jars/"

        elif [ "${SPARK_VERSION}" = "3.5.3" ]; then
            wget https://repo1.maven.org/maven2/io/delta/delta-spark_2.12/3.2.1/delta-spark_2.12-3.2.1.jar -P "${SPARK_HOME}/jars/"
            wget https://repo1.maven.org/maven2/io/delta/delta-storage/3.2.1/delta-storage-3.2.1.jar -P "${SPARK_HOME}/jars/"

        elif [ "${SPARK_VERSION}" = "3.5.4" ]; then
            wget https://repo1.maven.org/maven2/io/delta/delta-spark_2.12/3.3.0/delta-spark_2.12-3.3.0.jar -P "${SPARK_HOME}/jars/"
            wget https://repo1.maven.org/maven2/io/delta/delta-storage/3.3.0/delta-storage-3.3.0.jar -P "${SPARK_HOME}/jars/"

        elif [ "${SPARK_VERSION}" = "3.5.5" ]; then
            wget https://repo1.maven.org/maven2/io/delta/delta-spark_2.12/3.3.1/delta-spark_2.12-3.3.1.jar -P "${SPARK_HOME}/jars/"
            wget https://repo1.maven.org/maven2/io/delta/delta-storage/3.3.1/delta-storage-3.3.1.jar -P "${SPARK_HOME}/jars/"

        elif [ "${SPARK_VERSION}" = "4.0.0" ] || [ "${SPARK_VERSION}" = "4.0.2" ]; then
            # Spark 4.0.x — Delta 4.0.0: delta-spark_2.13-4.0.0.jar + delta-storage-4.0.0.jar
            wget https://repo1.maven.org/maven2/io/delta/delta-spark_2.13/4.0.0/delta-spark_2.13-4.0.0.jar -P "${SPARK_HOME}/jars/"
            wget https://repo1.maven.org/maven2/io/delta/delta-storage/4.0.0/delta-storage-4.0.0.jar -P "${SPARK_HOME}/jars/"

        elif [ "${SPARK_VERSION}" = "4.1.1" ] || [ "${SPARK_VERSION}" = "4.1.2" ]; then
            # Spark 4.1.x — Delta 4.1.0: delta-spark_2.13-4.1.0.jar + delta-storage-4.1.0.jar
            wget https://repo1.maven.org/maven2/io/delta/delta-spark_2.13/4.1.0/delta-spark_2.13-4.1.0.jar -P "${SPARK_HOME}/jars/"
            wget https://repo1.maven.org/maven2/io/delta/delta-storage/4.1.0/delta-storage-4.1.0.jar -P "${SPARK_HOME}/jars/"

        elif [ "${SPARK_VERSION}" = "4.2.0" ] ; then
            # Spark 4.2.x — Delta 4.4.0: delta-spark_4.2_2.13-4.4.0.jar + delta-storage-4.4.0.jar
            wget https://repo1.maven.org/maven2/io/delta/delta-spark_4.2_2.13/4.4.0/delta-spark_4.2_2.13-4.4.0.jar -P "${SPARK_HOME}/jars/"
            wget https://repo1.maven.org/maven2/io/delta/delta-storage/4.4.0/delta-storage-4.4.0.jar -P "${SPARK_HOME}/jars/"

        else
            cp -r "${PRIVACERA_WORK_DIR}/oss-delta-jars/"* "${SPARK_HOME}/jars/"
        fi

        # update spark-defaults.conf
        if [ $(grep -rin "spark.sql.catalog.spark_catalog" "$SPARK_DEFAULT_CONF" | wc -l) -gt 0 ]; then
            sed -i '' -e 's|spark.sql.catalog.spark_catalog.*|spark.sql.catalog.spark_catalog org.apache.spark.sql.delta.catalog.DeltaCatalog|g' "$SPARK_DEFAULT_CONF"
        else
            echo "spark.sql.catalog.spark_catalog org.apache.spark.sql.delta.catalog.DeltaCatalog" >> "$SPARK_DEFAULT_CONF"
        fi

        if [[ "${SPARK_PLUGIN_TYPE}" == *"FGAC"* ]]; then
            if [ $(grep -rin "spark.sql.extensions" "$SPARK_DEFAULT_CONF" | wc -l) -gt 0 ]; then
                sed -i "s|spark.sql.extensions.*|spark.sql.extensions com.privacera.spark.agent.SparkSQLExtension,io.delta.sql.DeltaSparkSessionExtension|g" "$SPARK_DEFAULT_CONF"
            else
                echo "spark.sql.extensions com.privacera.spark.agent.SparkSQLExtension,io.delta.sql.DeltaSparkSessionExtension" >> "$SPARK_DEFAULT_CONF"
            fi
        else
            if [ $(grep -rin "spark.sql.extensions" "$SPARK_DEFAULT_CONF" | wc -l) -gt 0 ]; then
                sed -i "s|spark.sql.extensions.*|spark.sql.extensions io.delta.sql.DeltaSparkSessionExtension|g" "$SPARK_DEFAULT_CONF"
            else
                echo "spark.sql.extensions io.delta.sql.DeltaSparkSessionExtension" >> "$SPARK_DEFAULT_CONF"
            fi
        fi

    fi  
}

function setup_iceberg() {
    log "setup_iceberg started"

    # Run only if Iceberg enabled
    [ "${OSS_ICEBERG_ENABLE}" != true ] && return

    SPARK_MAJOR_VERSION=$(echo "${SPARK_VERSION}" | cut -d'.' -f1,2)

    # Download Iceberg runtime jar based on Spark version
    if [ "${SPARK_MAJOR_VERSION}" = "3.4" ]; then
        wget -q https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.4_2.12/1.10.1/iceberg-spark-runtime-3.4_2.12-1.10.1.jar -P "${SPARK_HOME}/jars/"
    elif [ "${SPARK_MAJOR_VERSION}" = "3.5" ]; then
        wget -q https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.10.1/iceberg-spark-runtime-3.5_2.12-1.10.1.jar -P "${SPARK_HOME}/jars/"
    elif [ "${SPARK_MAJOR_VERSION}" = "4.0" ]; then
        wget -q https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-4.0_2.13/1.10.1/iceberg-spark-runtime-4.0_2.13-1.10.1.jar -P "${SPARK_HOME}/jars/"
    elif [ "${SPARK_MAJOR_VERSION}" = "4.1" ]; then
        wget -q https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-4.1_2.13/1.10.1/iceberg-spark-runtime-4.1_2.13-1.10.1.jar -P "${SPARK_HOME}/jars/"
    else
        log "Unsupported Spark version ${SPARK_VERSION} for Iceberg"
        return
    fi

    # Configure Iceberg catalog
    if grep -q "^spark.sql.catalog.iceberg_catalog " "$SPARK_DEFAULT_CONF"; then
        sed -i '' -e 's|spark.sql.catalog.iceberg_catalog.*|spark.sql.catalog.iceberg_catalog org.apache.iceberg.spark.SparkCatalog|g' "$SPARK_DEFAULT_CONF"
    else
        echo "spark.sql.catalog.iceberg_catalog org.apache.iceberg.spark.SparkCatalog" >> "$SPARK_DEFAULT_CONF"
    fi

    if grep -q "^spark.sql.catalog.iceberg_catalog.type" "$SPARK_DEFAULT_CONF"; then
        sed -i '' -e "s|^spark.sql.catalog.iceberg_catalog.type.*|spark.sql.catalog.iceberg_catalog.type ${ICEBERG_CATALOG_TYPE}|g" "$SPARK_DEFAULT_CONF"
    else
        echo "spark.sql.catalog.iceberg_catalog.type ${ICEBERG_CATALOG_TYPE}" >> "$SPARK_DEFAULT_CONF"
    fi

    # Configure warehouse if provided
    if [ -n "${ICEBERG_WAREHOUSE_PATH}" ]; then
        if grep -q "^spark.sql.catalog.iceberg_catalog.warehouse" "$SPARK_DEFAULT_CONF"; then
            sed -i '' -e "s|^spark.sql.catalog.iceberg_catalog.warehouse.*|spark.sql.catalog.iceberg_catalog.warehouse ${ICEBERG_WAREHOUSE_PATH}|g" "$SPARK_DEFAULT_CONF"
        else
            echo "spark.sql.catalog.iceberg_catalog.warehouse ${ICEBERG_WAREHOUSE_PATH}" >> "$SPARK_DEFAULT_CONF"
        fi
    fi

    # Ensure Iceberg extension is present
    if grep -q "^spark.sql.extensions" "$SPARK_DEFAULT_CONF"; then
        if ! grep -q "IcebergSparkSessionExtensions" "$SPARK_DEFAULT_CONF"; then
            sed -i '' -e 's|spark.sql.extensions.*|&,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions|' "$SPARK_DEFAULT_CONF"
        fi
    else
        echo "spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" >> "$SPARK_DEFAULT_CONF"
    fi
}

function setup_s3tables() {
    log "setup_s3tables started"

    # Run only if S3Tables enabled
    [ "${OSS_S3_TABLE_ENABLE}" != true ] && return

    SPARK_MAJOR_VERSION=$(echo "${SPARK_VERSION}" | cut -d'.' -f1,2)

    # Determine Scala version
    if [ "${SPARK_MAJOR_VERSION}" = "4.0" ] || [ "${SPARK_MAJOR_VERSION}" = "4.1" ]; then
        SCALA_VERSION="2.13"
    else
        SCALA_VERSION="2.12"
    fi

    # Download Iceberg runtime (required for S3Tables because S3Tables uses Iceberg as the underlying table format)
    if ! ls ${SPARK_HOME}/jars/iceberg-spark-runtime-${SPARK_MAJOR_VERSION}_${SCALA_VERSION}-*.jar 1> /dev/null 2>&1; then
        wget -q https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-${SPARK_MAJOR_VERSION}_${SCALA_VERSION}/1.10.1/iceberg-spark-runtime-${SPARK_MAJOR_VERSION}_${SCALA_VERSION}-1.10.1.jar \
            -P "${SPARK_HOME}/jars/"
    fi

    # Download S3Tables catalog runtime
    wget -q https://repo1.maven.org/maven2/software/amazon/s3tables/s3-tables-catalog-for-iceberg-runtime/0.1.8/s3-tables-catalog-for-iceberg-runtime-0.1.8.jar \
        -P "${SPARK_HOME}/jars/"

    # AWS SDK bundle (skip for Spark 4.0 / 4.1 because Dockerfile already downloads it)
    if [ "${SPARK_MAJOR_VERSION}" != "4.0" ] && [ "${SPARK_MAJOR_VERSION}" != "4.1" ]; then
        wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.25.29/bundle-2.25.29.jar \
            -P "${SPARK_HOME}/jars/"
    fi

    # Configure S3Tables catalog
    if grep -q "^spark.sql.catalog.s3tables " "$SPARK_DEFAULT_CONF"; then
        sed -i '' -e 's|spark.sql.catalog.s3tables.*|spark.sql.catalog.s3tables org.apache.iceberg.spark.SparkCatalog|g' "$SPARK_DEFAULT_CONF"
    else
        echo "spark.sql.catalog.s3tables org.apache.iceberg.spark.SparkCatalog" >> "$SPARK_DEFAULT_CONF"
    fi

    # Configure catalog implementation
    if grep -q "^spark.sql.catalog.s3tables.catalog-impl" "$SPARK_DEFAULT_CONF"; then
        sed -i '' -e 's|^spark.sql.catalog.s3tables.catalog-impl.*|spark.sql.catalog.s3tables.catalog-impl software.amazon.s3tables.iceberg.S3TablesCatalog|g' "$SPARK_DEFAULT_CONF"
    else
        echo "spark.sql.catalog.s3tables.catalog-impl software.amazon.s3tables.iceberg.S3TablesCatalog" >> "$SPARK_DEFAULT_CONF"
    fi

    # Configure warehouse ARN
    if [ -n "${S3_TABLE_WAREHOUSE}" ]; then
        if grep -q "^spark.sql.catalog.s3tables.warehouse" "$SPARK_DEFAULT_CONF"; then
            sed -i '' -e "s|^spark.sql.catalog.s3tables.warehouse.*|spark.sql.catalog.s3tables.warehouse ${S3_TABLE_WAREHOUSE}|g" "$SPARK_DEFAULT_CONF"
        else
            echo "spark.sql.catalog.s3tables.warehouse ${S3_TABLE_WAREHOUSE}" >> "$SPARK_DEFAULT_CONF"
        fi
    fi

    # Ensure Iceberg extension is present
    if grep -q "^spark.sql.extensions" "$SPARK_DEFAULT_CONF"; then
        if ! grep -q "IcebergSparkSessionExtensions" "$SPARK_DEFAULT_CONF"; then
            sed -i '' -e 's|spark.sql.extensions.*|&,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions|' "$SPARK_DEFAULT_CONF"
        fi
    else
        echo "spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" >> "$SPARK_DEFAULT_CONF"
    fi
}

function setup_privacera_distributor_plugin() {
    log "setup_privacera_distributor_plugin started"
    if [[ "${SPARK_PLUGIN_TYPE}" == "OLAC" && "${OSS_SPARK_PRIVACERA_DISTRIBUTOR_PLUGIN_ENABLE}" == true ]]; then
        echo "Setting Privacera Distributor Plugin"
        if grep -q "^spark.plugins" "$SPARK_DEFAULT_CONF"; then
            if ! grep -q "PrivaceraSparkPlugin" "$SPARK_DEFAULT_CONF"; then
                sed -i '' -e 's|spark.plugins.*|&,com.privacera.spark.plugin.PrivaceraSparkPlugin|' "$SPARK_DEFAULT_CONF"
            fi
        else
            echo "spark.plugins com.privacera.spark.plugin.PrivaceraSparkPlugin" >> "$SPARK_DEFAULT_CONF"
        fi
    fi
}

function  verify() {
    log "Privacera Spark setup completed"
}

createFolders
validation
download_pkg
install
update_jars
configure
setup_deltalake
setup_iceberg
setup_s3tables
setup_privacera_distributor_plugin
verify

  7. Make the privacera_setup.sh and standalone_spark_setup.sh script files executable 

    chmod +x privacera_setup.sh standalone_spark_setup.sh

Create Configuration Files

  • Create config folder and navigate to it 
    mkdir ~/privacera-oss-plugin/config
cd ~/privacera-oss-plugin/config

  • Copy the privacera_spark.properties and global-truststore.p12 files to the config folder 

    cp ~/privacera/privacera-manager/output/spark-standalone/spark_custom_conf/OLAC/privacera_spark.properties ~/privacera-oss-plugin/config/
cp ~/privacera/privacera-manager/output/spark-standalone/spark_custom_conf/global-truststore.p12 ~/privacera-oss-plugin/config/

  • Navigate to config folder and update the privacera_spark.properties file as follow: 

    cd ~/privacera-oss-plugin/config/
vi privacera_spark.properties

privacera.signer.truststore=/opt/privacera/global-truststore.p12

  • Copy the privacera_spark.properties and required configuration files to the config folder 

    cp ~/privacera/privacera-manager/output/spark-standalone/spark_custom_conf/OLAC_FGAC/privacera_spark.properties ~/privacera-oss-plugin/config/
cp ~/privacera/privacera-manager/output/spark-standalone/spark_custom_conf/auditserver-secrets-keystore.jks ~/privacera-oss-plugin/config/
cp ~/privacera/privacera-manager/output/spark-standalone/spark_custom_conf/ranger-plugin-keystore.p12 ~/privacera-oss-plugin/config/
cp ~/privacera/privacera-manager/output/spark-standalone/spark_custom_conf/ranger.jceks ~/privacera-oss-plugin/config/
cp ~/privacera/privacera-manager/output/spark-standalone/spark_custom_conf/global-truststore.p12 ~/privacera-oss-plugin/config/

  • Navigate to config folder and update the privacera_spark.properties file as follow: 

    cd ~/privacera-oss-plugin/config/
vi privacera_spark.properties

privacera.signer.truststore=/opt/privacera/global-truststore.p12
xasecure.audit.keystore.path=/opt/privacera/auditserver-secrets-keystore.jks
xasecure.policymgr.clientssl.keystore=/opt/privacera/ranger-plugin-keystore.p12
xasecure.policymgr.clientssl.keystore.credential.file=jceks://file//opt/privacera/ranger.jceks
xasecure.policymgr.clientssl.truststore=/opt/privacera/global-truststore.p12
xasecure.policymgr.clientssl.truststore.credential.file=jceks://file//opt/privacera/ranger.jceks

  • Remove the following property from the privacera_spark.properties file: 

    vi privacera_spark.properties
privacera.clusterName=<CLUSTER_NAME>

  • Create log4j2.properties file and copy the following content to it 

    vi log4j2.properties

    log4j2.properties
    log4j2-properties
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
    # Root logger configuration
rootLogger.level = INFO
rootLogger.appenderRefs = stdout, file
rootLogger.appenderRef.stdout.ref = console
rootLogger.appenderRef.file.ref = fileLogger

# Console appender
appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} [%t] %p %c :%L %m%n%ex

## Rolling file config
appender.rolling.type = RollingFile
appender.rolling.name = fileLogger
appender.rolling.fileName = /tmp/${sys:user.name}/privacera.log
appender.rolling.filePattern = /tmp/${sys:user.name}/privacera-%d{yyyy-MM-dd}-%i.log
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} [%t] %p %c :%L %m%n%ex
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size = 100MB

# fileLogger (RollingFileAppender) rotation policy
appender.rolling.policies.time.modulate = true
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.delete.type = Delete
appender.rolling.strategy.delete.basePath = /tmp/${sys:user.name}
appender.rolling.strategy.delete.ifFileName.type = IfFileName
appender.rolling.strategy.delete.ifFileName.glob = privacera-*.log
appender.rolling.strategy.delete.maxDepth = 10
appender.rolling.strategy.delete.ifLastModified.type = IfLastModified

# Delete all files older than 30 days
appender.rolling.strategy.delete.ifLastModified.age = 30d

# Privacera
logger.privacera.name = com.privacera
logger.privacera.level = INFO
logger.privacera.additivity = false
logger.privacera.appenderRefs = rolling
logger.privacera.appenderRef.rolling.ref = fileLogger

# Ranger
logger.ranger.name = org.apache.ranger
logger.ranger.level = INFO
logger.ranger.additivity = false
logger.ranger.appenderRefs = rolling
logger.ranger.appenderRef.rolling.ref = fileLogger

# AWS SDK V1
logger.amazon.name = com.amazon
logger.amazon.level = INFO
logger.amazon.additivity = false
logger.amazon.appenderRefs = rolling
logger.amazon.appenderRef.rolling.ref = fileLogger

logger.amazonaws.name = com.amazonaws
logger.amazonaws.level = INFO
logger.amazonaws.additivity = false
logger.amazonaws.appenderRefs = rolling
logger.amazonaws.appenderRef.rolling.ref = fileLogger

# AWS SDK V2
logger.software-amazon.name = software.amazon.awssdk
logger.software-amazon.level = INFO
logger.software-amazon.additivity = false
logger.software-amazon.appenderRefs = rolling
logger.software-amazon.appenderRef.rolling.ref = fileLogger

# Apache http wire
logger.apache.name = org.apache.http.wire
logger.apache.level = INFO
logger.apache.additivity = false
logger.apache.appenderRefs = rolling
logger.apache.appenderRef.rolling.ref = fileLogger

# Apache http client
logger.httpclient.name = org.apache.http.client
logger.httpclient.level = INFO
logger.httpclient.additivity = false
logger.httpclient.appenderRefs = rolling
logger.httpclient.appenderRef.rolling.ref = fileLogger

# Spark 4.x JLine
logger.jline.name = org.jline
logger.jline.level = ERROR
logger.jline.additivity = false
logger.jline.appenderRefs = rolling
logger.jline.appenderRef.rolling.ref = fileLogger

Generate Privacera Deployment File

Note

  • Execute the build_privacera_plugin.sh script to generate the privacera deployment tarball file: 
    cd ~/privacera-oss-plugin
chmod +x build_privacera_plugin.sh
./build_privacera_plugin.sh

Building the Docker Image

Create Dockerfile

  • Create Dockerfile script file and copy the following content to it

    Note

    • The Docker file Dockerfile is provided here is as an example file that uses the open-source Apache Spark images. Instead, you can use your custom spark docker file and integrate the below steps to deploy Privacera Plugin into it.
    • To achieve that, add the below commands in your existing Dockerfile 
      ARG PRIVACERA_SPARK_PLUGIN_TAR_GZ=./privacera-spark-plugin.tar.gz

# Add the privacera-spark-plugin tar.gz to the image at the root
ADD ${PRIVACERA_SPARK_PLUGIN_TAR_GZ} 

RUN mkdir -p ${PRIVACERA_HOME}
RUN chown -R ${USER_NAME}:${GROUP_NAME} ${PRIVACERA_HOME}

# Symlinks for common files
RUN ln -sf /privacera-secret/privacera_spark.properties /opt/spark/conf/privacera_spark.properties && \
    ln -sf /privacera-secret/global-truststore.p12 /opt/privacera/global-truststore.p12 && \
    ln -sf /privacera-conf/log4j2.properties /opt/spark/conf/log4j2.properties

# Symlinks for OLAC_FGAC
RUN if [ "$PRIVACERA_SPARK_PLUGIN_TYPE" = "OLAC_FGAC" ]; then \
    ln -sf /privacera-secret/auditserver-secrets-keystore.jks /opt/privacera/auditserver-secrets-keystore.jks && \
    ln -sf /privacera-secret/ranger-plugin-keystore.p12 /opt/privacera/ranger-plugin-keystore.p12 && \
    ln -sf /privacera-secret/ranger.jceks /opt/privacera/ranger.jceks ; \
    fi

    cd ~/privacera-oss-plugin
vi Dockerfile
    Dockerfile
    Dockerfile
      1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
    ARG SPARK_BASE_IMAGE
FROM ${SPARK_BASE_IMAGE}

ARG SPARK_VERSION
ARG PRIVACERA_SPARK_PLUGIN_TAR_GZ=./privacera-spark-plugin.tar.gz
ARG PRIVACERA_SPARK_PLUGIN_TYPE
ARG SPARK_HOME=/opt/spark
ARG USER_NAME=spark
ARG GROUP_NAME=spark
ARG PRIVACERA_HOME=/opt/privacera

ENV PRIVACERA_HOME=${PRIVACERA_HOME}
ENV PRIVACERA_SPARK_PLUGIN_TYPE=${PRIVACERA_SPARK_PLUGIN_TYPE}
ENV OSS_DELTA_LAKE_ENABLE=${OSS_DELTA_LAKE_ENABLE}
ENV OSS_ICEBERG_ENABLE=${OSS_ICEBERG_ENABLE}
ENV OSS_S3_TABLE_ENABLE=${OSS_S3_TABLE_ENABLE}
ENV OSS_SPARK_PRIVACERA_DISTRIBUTOR_PLUGIN_ENABLE=${OSS_SPARK_PRIVACERA_DISTRIBUTOR_PLUGIN_ENABLE}
ENV SPARK_VERSION=${SPARK_VERSION}
ENV SPARK_USER=${USER_NAME}
ENV SPARK_GROUP=${GROUP_NAME}
ENV SPARK_HOME=${SPARK_HOME}

USER root

# Download AWS JARs for supported Spark versions
RUN if [ "${SPARK_VERSION}" = "3.5.3" ] || \
       [ "${SPARK_VERSION}" = "3.5.4" ] || \
       [ "${SPARK_VERSION}" = "3.5.5" ]; then \
        wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar \
            -P ${SPARK_HOME}/jars/; \
        wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.781/aws-java-sdk-bundle-1.12.781.jar \
            -P ${SPARK_HOME}/jars/; \
    elif [ "${SPARK_VERSION}" = "4.0.0" ] || [ "${SPARK_VERSION}" = "4.0.2" ]; then \
        # Hadoop AWS 3.4.1 + AWS SDK v1 bundle (S3A compatibility) + v2 bundle (Privacera signer); align versions with build_image.sh for Hadoop/v2
        wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.4.1/hadoop-aws-3.4.1.jar \
            -P ${SPARK_HOME}/jars/; \
        # AWS SDK v1 bundle
        wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.797/aws-java-sdk-bundle-1.12.797.jar \
            -P ${SPARK_HOME}/jars/; \
        # AWS SDK V2 Bundle
        wget https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.25.53/bundle-2.25.53.jar \
            -P ${SPARK_HOME}/jars/; \
    elif [ "${SPARK_VERSION}" = "4.1.1" ] || [ "${SPARK_VERSION}" = "4.1.2" ]; then \
        # Hadoop AWS 3.4.2 + AWS SDK v1 bundle + v2 bundle (same pattern as Spark 4.0.x)
        wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.4.2/hadoop-aws-3.4.2.jar \
            -P ${SPARK_HOME}/jars/; \
        # AWS SDK v1 bundle
        wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.797/aws-java-sdk-bundle-1.12.797.jar \
            -P ${SPARK_HOME}/jars/; \
        # AWS SDK V2 Bundle
        wget https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.29.52/bundle-2.29.52.jar \
            -P ${SPARK_HOME}/jars/; \
    elif [ "${SPARK_VERSION}" = "4.2.0" ]; then \
        # Hadoop AWS 3.5.0 + AWS SDK v1 bundle + v2 bundle
        wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.5.0/hadoop-aws-3.5.0.jar \
            -P ${SPARK_HOME}/jars/; \
        # AWS SDK v1 bundle
        wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.797/aws-java-sdk-bundle-1.12.797.jar \
            -P ${SPARK_HOME}/jars/; \
        # AWS SDK V2 Bundle
        wget https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.35.4/bundle-2.35.4.jar \
            -P ${SPARK_HOME}/jars/; \
        # S3 Analytics Accelerator - hadoop-aws 3.5.0 references this class at S3A stream-factory
        wget https://repo1.maven.org/maven2/software/amazon/s3/analyticsaccelerator/analyticsaccelerator-s3/1.3.1/analyticsaccelerator-s3-1.3.1.jar \
            -P ${SPARK_HOME}/jars/; \
    else \
        echo "No hadoop-aws and aws-java-sdk-bundle jars downloaded"; \
    fi

# Add the privacera-spark-plugin tar.gz to the image at the root
ADD ${PRIVACERA_SPARK_PLUGIN_TAR_GZ} /

# Set up Privacera directories and permissions
RUN mkdir -p ${PRIVACERA_HOME}
RUN chown -R ${USER_NAME}:${GROUP_NAME} ${PRIVACERA_HOME}

# Symlinks for common files
RUN ln -sf /privacera-secret/privacera_spark.properties /opt/spark/conf/privacera_spark.properties && \
    ln -sf /privacera-secret/global-truststore.p12 /opt/privacera/global-truststore.p12 && \
    ln -sf /privacera-conf/log4j2.properties /opt/spark/conf/log4j2.properties

# Symlinks for OLAC_FGAC
RUN if [ "$PRIVACERA_SPARK_PLUGIN_TYPE" = "OLAC_FGAC" ]; then \
      ln -sf /privacera-secret/auditserver-secrets-keystore.jks /opt/privacera/auditserver-secrets-keystore.jks && \
      ln -sf /privacera-secret/ranger-plugin-keystore.p12 /opt/privacera/ranger-plugin-keystore.p12 && \
      ln -sf /privacera-secret/ranger.jceks /opt/privacera/ranger.jceks ; \
     fi

# Spark 4.x (4.0.0, 4.0.2, 4.1.1, 4.1.2, 4.2.0): append S3A signer settings to spark-defaults.conf
RUN if [ "${SPARK_VERSION}" = "4.0.0" ] || [ "${SPARK_VERSION}" = "4.0.2" ] || [ "${SPARK_VERSION}" = "4.1.1" ] || [ "${SPARK_VERSION}" = "4.1.2" ] || [ "${SPARK_VERSION}" = "4.2.0" ]; then \
       # Enable Privacera AWS SDK v2 signer for S3A
       echo "spark.hadoop.fs.s3a.custom.signers PrivaceraAwsSdkV2Signer:com.privacera.spark.agent.signer.PrivaceraAwsSdkV2Signer" >> /opt/spark/conf/spark-defaults.conf; \
       echo "spark.hadoop.fs.s3a.s3.signing-algorithm PrivaceraAwsSdkV2Signer" >> /opt/spark/conf/spark-defaults.conf; \
       echo "spark.hadoop.fs.s3a.access.key P_ACCESS_KEY" >> /opt/spark/conf/spark-defaults.conf; \
       echo "spark.hadoop.fs.s3a.secret.key P_SECRET_KEY" >> /opt/spark/conf/spark-defaults.conf; \
       echo "spark.hadoop.fs.s3a.session.token P_SESSION_TOKEN" >> /opt/spark/conf/spark-defaults.conf; \
    fi

# To Fix NoSuchFileException in Apache Spark Connect for '.ammonite/rt-17.0.17.jar' and JLine warnings (Spark 4.x shell)
RUN mkdir -p /nonexistent
RUN chown -R ${USER_NAME}:${GROUP_NAME} /nonexistent

# install python dependencies - require for pyspark with Apache Spark Connect
RUN pip install --no-cache-dir grpcio grpcio-status pandas pyarrow zstandard

# Privacera Boto3 Signer support — runs only when the wheel was staged by
# build_privacera_plugin.sh (OSS_BOTO3_SIGNER_ENABLE="true" in penv.sh); no-op otherwise.
# Installs the signer into the system Python and installs python3-venv so virtual
# environments work inside the container without any runtime package installation.
# The wheel is kept at ${PRIVACERA_HOME}/boto3_signer/ so it can also be installed
# into plain virtual environments offline:
#   pip install ${PRIVACERA_HOME}/boto3_signer/privacera_boto3_signer-*.whl
RUN if ls ${PRIVACERA_HOME}/boto3_signer/privacera_boto3_signer-*.whl >/dev/null 2>&1; then \
        apt-get update && \
        apt-get install -y --no-install-recommends python3-venv && \
        rm -rf /var/lib/apt/lists/*; \
        # PIP_BREAK_SYSTEM_PACKAGES=1: no-op on older pip (Ubuntu 22.04 base), and
        # allows the system-Python install on PEP 668 externally-managed bases
        # (Ubuntu 24.04+). The --break-system-packages FLAG would error on old pip.
        PIP_BREAK_SYSTEM_PACKAGES=1 pip install --no-cache-dir ${PRIVACERA_HOME}/boto3_signer/privacera_boto3_signer-*.whl; \
    fi

USER ${USER_NAME}

Build the Docker Image

  1. Run the following command to create and edit build_spark_image.sh file. Expand the following section and copy it's content to the build_spark_image.sh file 

    cd ~/privacera-oss-plugin
vi build_spark_image.sh

    build_spark_image.sh
    build_spark_image.sh
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
    #!/bin/bash

set -x

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
ENV_FILE="${SCRIPT_DIR}/penv.sh"

# if penv.sh exists, load it
if [ -f ${ENV_FILE} ]; then
  echo "Sourcing env file ${ENV_FILE}"
  source ${ENV_FILE}
else
  echo "${ENV_FILE} file not found"
  exit 1
fi

# Fallback defaults if not set in penv.sh
SPARK_BASE_IMAGE="${SPARK_BASE_IMAGE:-spark:3.5.3-python3}"
SPARK_VERSION="${SPARK_VERSION:-3.5.3}"

# create plugin image
DOCKER_BUILDKIT=1 docker build \
    -t ${TAG} \
    --build-arg SPARK_BASE_IMAGE=${SPARK_BASE_IMAGE} \
    --build-arg SPARK_VERSION=${SPARK_VERSION} \
    --build-arg PRIVACERA_SPARK_PLUGIN_TYPE=${PRIVACERA_SPARK_PLUGIN_TYPE} \
    --progress plain \
    -f Dockerfile .

  2. Execute the build_spark_image.sh script to generate the privacera deployment files: 

    chmod +x build_spark_image.sh
./build_spark_image.sh

Push the Docker Image to the Remote HUB

  • Use your internal HUB to publish the image.

Kubernetes Deployment

Create Kubernetes YAML Template Files

  1. Create k8s/templates folder and navigate to it 

    mkdir -p ~/privacera-oss-plugin/k8s/templates
cd ~/privacera-oss-plugin/k8s/templates

  2. Create namespace.yml script file and copy the following content to it 

    vi namespace.yml

    namespace.yml
    namespace.yaml
    1
2
3
4
    apiVersion: v1
kind: Namespace
metadata:
  name: SPARK_NAME_SPACE

  3. Create role-binding.yml script file and copy the following content to it 

    vi role-binding.yml

    role-binding.yml
    role-binding.yaml
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
    apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: SPARK_PLUGIN_ROLE_BINDING
  namespace: SPARK_NAME_SPACE
subjects:
- kind: ServiceAccount
  name: SPARK_PLUGIN_SERVICE_ACCOUNT
  namespace: SPARK_NAME_SPACE
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: SPARK_PLUGIN_ROLE

  4. Create role.yml script file and copy the following content to it 

    vi role.yml

    role.yml
    role.yaml
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
    apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: SPARK_PLUGIN_ROLE
  namespace: SPARK_NAME_SPACE
rules:
- apiGroups: ["", "extensions", "apps"]
  resources: ["*"]
  verbs: ["*"]
- apiGroups: ["batch"]
  resources:
  - jobs
  - cronjobs
  verbs: ["*"]

  5. Create service-account.yml script file and copy the following content to it 

    vi service-account.yml

    service-account.yml
    service-account.yaml
    1
2
3
4
5
    apiVersion: v1
kind: ServiceAccount
metadata:
  name: SPARK_PLUGIN_SERVICE_ACCOUNT
  namespace:  SPARK_NAME_SPACE

  6. Create privacera-spark-examples.yml script file and copy the following content to it 

    vi privacera-spark-examples.yml

    privacera-spark-examples.yml
    privacera-spark-examples.yaml
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
    apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: SPARK_PLUGIN_APP_NAME
  name: SPARK_PLUGIN_APP_NAME
spec:
  replicas: 1
  selector:
    matchLabels:
      app: SPARK_PLUGIN_APP_NAME
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: SPARK_PLUGIN_APP_NAME
    spec:
      serviceAccountName: SPARK_PLUGIN_SERVICE_ACCOUNT
      securityContext:
        fsGroup: 200
      initContainers:
      imagePullSecrets:
        - name: SPARK_DOCKER_PULL_SECRET
      containers:
        - image: SPARK_PLUGIN_IMAGE
          name: SPARK_PLUGIN_APP_NAME-exec
          imagePullPolicy: Always
          command: [ "/bin/bash", "-ce", "tail -f /dev/null" ]
          env:
            - name: SPARK_PLUGIN_POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: SPARK_PLUGIN_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          ports:
            - name: spark-ui
              containerPort: 4040
            - name: sparkdriver
              containerPort: 7077
            - name: blockmanager
              containerPort: 7078
          resources:
            limits:
              memory: "2Gi"
              cpu: "0.5"
            requests:
              memory: "1Gi"
              cpu: "0.2"
          volumeMounts:
            - name: privacera-spark-secret-volume
              mountPath: /privacera-secret
            - name: privacera-spark-conf-volume
              mountPath: /privacera-conf
      restartPolicy: Always
      volumes:
        - name: privacera-spark-secret-volume
          secret:
            secretName: PRIVACERA_SECRET_NAME
        - name: privacera-spark-conf-volume
          configMap:
            name: PRIVACERA_CONFIGMAP_NAME
status: {}

Create Scripts to Generate Kubernetes Deployment Files from Templates

  1. Create replace.sh script file and copy the following content to it 

    cd ~/privacera-oss-plugin/k8s  
vi replace.sh

    replace.sh
    replace.sh
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
    #!/bin/bash

set -x

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

ENV_FILE="${SCRIPT_DIR}/../penv.sh"

# if penv.sh exists, load it
if [ -f ${ENV_FILE} ]; then
  echo "Sourcing env file ${ENV_FILE}"
  source ${ENV_FILE}
else
  echo "${ENV_FILE} file not found"
  exit 1
fi

SPARK_NAME_SPACE=${SPARK_NAME_SPACE:-privacera-spark-plugin-test}
SPARK_PLUGIN_ROLE_BINDING=${SPARK_PLUGIN_ROLE_BINDING:-privacera-sa-spark-plugin-role-binding}
SPARK_PLUGIN_SERVICE_ACCOUNT=${SPARK_PLUGIN_SERVICE_ACCOUNT:-privacera-sa-spark-plugin}
SPARK_PLUGIN_ROLE=${SPARK_PLUGIN_ROLE:-privacera-sa-spark-plugin-role}
SPARK_PLUGIN_APP_NAME=${SPARK_PLUGIN_APP_NAME:-privacera-spark-examples}
SPARK_PLUGIN_IMAGE=${SPARK_PLUGIN_IMAGE:-hub.docker.us/spark-plugin:latest}
SPARK_DOCKER_PULL_SECRET=${SPARK_DOCKER_PULL_SECRET:-docker-hub}
INPUT_DIR=${INPUT_DIR:-./templates}
OUTPUT_DIR=${OUTPUT_DIR:-./output}

mkdir -p ${OUTPUT_DIR}

files=(namespace.yml role-binding.yml service-account.yml privacera-spark-examples.yml role.yml)
for i in ${!files[@]}
do
    in_file="${INPUT_DIR}/${files[i]}"
    out_file="${OUTPUT_DIR}/${files[i]}"
    if [ -f "${in_file}" ]; then
      echo "replace variable in file ${in_file} > out_file=${out_file}"
      cp "${in_file}" "${out_file}"
      sed -i "s|SPARK_NAME_SPACE|${SPARK_NAME_SPACE}|g" "${out_file}"
      sed -i "s|SPARK_PLUGIN_ROLE_BINDING|${SPARK_PLUGIN_ROLE_BINDING}|g" "${out_file}"
      sed -i "s|SPARK_PLUGIN_SERVICE_ACCOUNT|${SPARK_PLUGIN_SERVICE_ACCOUNT}|g" "${out_file}"
      sed -i "s|SPARK_PLUGIN_ROLE|$SPARK_PLUGIN_ROLE|g" "${out_file}"
      sed -i "s|SPARK_PLUGIN_APP_NAME|$SPARK_PLUGIN_APP_NAME|g" "${out_file}"
      sed -i "s|SPARK_PLUGIN_IMAGE|$SPARK_PLUGIN_IMAGE|g" "${out_file}"
      sed -i "s|SPARK_DOCKER_PULL_SECRET|$SPARK_DOCKER_PULL_SECRET|g" "${out_file}"
      sed -i "s|PRIVACERA_SECRET_NAME|${PRIVACERA_SECRET_NAME}|g" "${out_file}"
      sed -i "s|PRIVACERA_CONFIGMAP_NAME|${PRIVACERA_CONFIGMAP_NAME}|g" "${out_file}"
    fi
done

  2. Create apply.sh script file and copy the following content to it 

    vi apply.sh

    apply.sh
    apply.sh
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
    #!/bin/bash

set -x

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

ENV_FILE="${SCRIPT_DIR}/../penv.sh"

# if penv.sh exists, load it
if [ -f ${ENV_FILE} ]; then
  echo "Sourcing env file ${ENV_FILE}"
  source ${ENV_FILE}
else
  echo "${ENV_FILE} file not found"
  exit 1
fi

kubectl apply -f ${OUTPUT_DIR}/namespace.yml
kubectl apply -f ${OUTPUT_DIR}/service-account.yml
kubectl apply -f ${OUTPUT_DIR}/role.yml
kubectl apply -f ${OUTPUT_DIR}/role-binding.yml

# Delete and recreate Kubernetes secret
kubectl delete secret ${PRIVACERA_SECRET_NAME} -n ${SPARK_NAME_SPACE}

if [[ "${PRIVACERA_SPARK_PLUGIN_TYPE}" == "OLAC" ]]; then
  kubectl create secret generic "${PRIVACERA_SECRET_NAME}" \
    --from-file="${SCRIPT_DIR}/../config/privacera_spark.properties" \
    --from-file="${SCRIPT_DIR}/../config/global-truststore.p12" \
    -n "${SPARK_NAME_SPACE}"

elif [[ "${PRIVACERA_SPARK_PLUGIN_TYPE}" == "OLAC_FGAC" ]]; then
  kubectl create secret generic "${PRIVACERA_SECRET_NAME}" \
    --from-file="${SCRIPT_DIR}/../config/privacera_spark.properties" \
    --from-file="${SCRIPT_DIR}/../config/global-truststore.p12" \
    --from-file="${SCRIPT_DIR}/../config/auditserver-secrets-keystore.jks" \
    --from-file="${SCRIPT_DIR}/../config/ranger-plugin-keystore.p12" \
    --from-file="${SCRIPT_DIR}/../config/ranger.jceks" \
    -n "${SPARK_NAME_SPACE}"
fi

# Delete and recreate Kubernetes configmap
kubectl delete configmap ${PRIVACERA_CONFIGMAP_NAME} -n ${SPARK_NAME_SPACE}
kubectl create configmap ${PRIVACERA_CONFIGMAP_NAME} \
  --from-file=${SCRIPT_DIR}/../config/log4j2.properties \
  -n ${SPARK_NAME_SPACE}

kubectl apply -f ${OUTPUT_DIR}/privacera-spark-examples.yml -n ${SPARK_NAME_SPACE}

Apply the Kubernetes Deployment Files

  1. Make the apply.sh and replace.sh script files executable 

    chmod +x apply.sh replace.sh

  2. Execute the replace.sh script to replace the variables in the k8s/templates folder 

    ./replace.sh

  3. Execute the apply.sh script to create necessary Kubernetes secrets and apply Spark deployment configs

    ./apply.sh

Validate the Deployment

  1. Check the status of the pods 

    export SPARK_NAME_SPACE=<SPARK_NAME_SPACE>
kubectl get pods -n ${SPARK_NAME_SPACE}

  2. Access the pod 

    kubectl exec -it <SPARK_PLUGIN_POD_NAME> -n ${SPARK_NAME_SPACE} -- /bin/bash

Configure the connector in Runtime and download it as a bundle. spark_env.sh unpacks the bundle and runs the generated script, which stages what the image build and the Kubernetes deployment need.

Configure the Apache Spark OLAC connector

Note

Runtime support for this connector is available on Self-Managed and D2P planes only.

Perform the following steps to configure the Apache Spark OLAC connector:

  1. In the portal, open the runtime plane and go to the Connector tab. If no runtime plane exists yet, see Getting Started with Runtime Plane.

  2. Add the Apache Spark connector. Under Access Control Configs -> Connector Configs, set the following:

    Field Definition Example
    Access control mode Object Level Access Control (OLAC). Fixed for this connector; OLAC_FGAC is not supported here. OLAC
    Spark home Spark installation directory. Plugin jars and configuration are written to jars/ and conf/ under this path. /opt/spark
    DataServer The DataServer this plugin signs its requests against, chosen from the DataServers on this plane. Required. dataserver-test-instance

  3. Advanced Options - optional. Each page covers what the option does and its default:

  4. Logging - only if Enable debug logging is on. For the fields and where the log is written, see Steps to Retrieve Logs for Troubleshooting.

  5. Custom Properties - optional. Additional properties can be added here as key-value pairs.

    For example:

    Key Value
    spark.example.property.enabled true

  6. Advanced Configuration - optional. Settings this form does not carry are set under Custom Properties. See Advanced Configuration.

  7. Save the configuration, then download the connector bundle.

    It holds trust3ai_apache_spark_plugin_<connector>_<version>.sh - the plugin download URL and every value the connector resolved - together with certs.zip.

    Note

    The bundle and the setup script inside it are both named after the connector and its version, so the names differ with each download.

    The next section copies the bundle onto the workstation that builds the Spark image. Nothing else needs to be downloaded from the portal.

Note

This connector is configuration only. Nothing is deployed to the runtime plane. The values reach Spark through the bundle, applied on the workstation that builds the image.

Setup Scripts and Configuration Files

Creating Script Files

  1. Create privacera-oss-plugin folder and navigate to it 

    mkdir ~/privacera-oss-plugin
cd ~/privacera-oss-plugin

  2. Copy the connector bundle into this folder, still zipped - spark_env.sh unpacks it. 

    ls trust3ai_apache_spark_plugin*_bundle.zip

  3. Create a spark_env.sh file and copy the following content to it. 

    vi spark_env.sh

    spark_env.sh
    spark_env.sh
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
    #!/bin/bash
set -euo pipefail

PRIVACERA_WORK_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export PRIVACERA_WORK_DIR

# The connector bundle downloaded from the portal. Its name carries the connector and version,
# so set it to the file that was downloaded.
export BUNDLE_ZIP="${PRIVACERA_WORK_DIR}/trust3ai_apache_spark_plugin_<connector>_<version>_bundle.zip"

##########################################
##               Plugin                 ##
##########################################
# The Spark release being built for, e.g. 4.1.1, 4.2.0
export SPARK_VERSION="<spark-version>"

# For Development only
#export PRIVACERA_RELEASES_BASE_URL="<base-url>"

##########################################
##               Docker                 ##
##########################################
# Base image, must match SPARK_VERSION, e.g. apache/spark:4.1.1-python3, spark:3.5.5-python3
export SPARK_BASE_IMAGE="<spark-base-image>"
# Spark Plugin Docker image, e.g. <account>.dkr.ecr.<region>.amazonaws.com/<repo>
export SPARK_PLUGIN_IMAGE="<spark-plugin-image>"
# Tag
export TAG="<tag>"

##########################################
##             Kubernetes               ##
##########################################
export SPARK_NAME_SPACE="<namespace>"
export SPARK_DOCKER_PULL_SECRET="spark-plugin-docker-hub"
export PRIVACERA_SECRET_NAME="privacera-spark-secret"
export SPARK_PLUGIN_SERVICE_ACCOUNT="privacera-sa-spark-plugin"
export SPARK_PLUGIN_ROLE="privacera-sa-spark-plugin-role"
export SPARK_PLUGIN_ROLE_BINDING="privacera-sa-spark-plugin-role-binding"
export SPARK_PLUGIN_APP_NAME="privacera-spark-examples"

##########################################

function die() { echo "FATAL: $*" >&2; exit 1; }

# Only when run, not when sourced: build_spark_image.sh and the k8s scripts source this file for
# the constants above, and sourcing must not kick off an install.
if [ "${BASH_SOURCE[0]}" = "$0" ]; then
    [ -f "${BUNDLE_ZIP}" ] || die "no bundle at ${BUNDLE_ZIP} - set BUNDLE_ZIP to the downloaded file"

    echo "Unpacking $(basename "${BUNDLE_ZIP}")"
    unzip -q -o "${BUNDLE_ZIP}" -d "${PRIVACERA_WORK_DIR}" || die "could not unpack ${BUNDLE_ZIP}"
    cd "${PRIVACERA_WORK_DIR}"

    # Ask the bundle which script it carries rather than scanning the directory: a previous
    # bundle's script is still here after an upgrade, and picking by name would run the old one.
    INIT_SCRIPT=$(unzip -Z1 "${BUNDLE_ZIP}" '*.sh' 2>/dev/null | head -1 || true)
    [ -n "${INIT_SCRIPT}" ] || die "${BUNDLE_ZIP} contained no setup script"

    echo "Running ${INIT_SCRIPT} for Spark ${SPARK_VERSION}"
    chmod +x "./${INIT_SCRIPT}"
    "./${INIT_SCRIPT}"

    echo "Done. ${PRIVACERA_WORK_DIR}/spark is ready to COPY into the image, and"
    echo "     ${PRIVACERA_WORK_DIR}/secret into the Kubernetes Secret."
fi

    Set only these:

    • BUNDLE_ZIP - the bundle file name.
    • SPARK_VERSION and SPARK_BASE_IMAGE - the Spark release being built for. These two must match.
    • SPARK_PLUGIN_IMAGE, TAG and SPARK_NAME_SPACE - where the image is pushed.

  4. Make spark_env.sh executable 

    chmod +x spark_env.sh

Generate Privacera Deployment File

  • Run spark_env.sh. It unpacks the bundle, runs the setup script generated for the connector, and that in turn downloads the plugin package and installs it: 

    cd ~/privacera-oss-plugin
./spark_env.sh

  • When it finishes the folder holds everything the remaining sections need:

    Folder What it is
    spark/ an image of SPARK_HOME - jars, spark-defaults.conf, the generated log4j2.properties. Copied into the Docker image.
    secret/ privacera_spark.properties and global-truststore.p12. Becomes the Kubernetes Secret, never an image layer.
    docker/ Dockerfile and build_spark_image.sh, versioned with the plugin.
    k8s/ replace.sh, apply.sh and the deployment templates.
    install.log the full output of the run.

    Note

    Re-running spark_env.sh rebuilds spark/ and secret/ from scratch, so a changed connector setting needs only a fresh bundle and another run.

Building the Docker Image

The Dockerfile and build_spark_image.sh are already in docker/ from the installer, so there is nothing to create. Spark version was set when spark_env.sh ran and is baked into spark/.

Using a custom Docker image

To build on a custom image, edit docker/Dockerfile - for example, change the base image - apply the changes there, and then run the build.

  1. Build the image 

    cd ~/privacera-oss-plugin
./docker/build_spark_image.sh

    Note

    The build context is spark/ alone, so nothing else in the install folder - secret/ above all - is uploaded to the Docker daemon. build_spark_image.sh sets that itself, so run it from either place.

  2. Push the image to the internal HUB.

Kubernetes Deployment

The deployment templates, replace.sh and apply.sh were staged into k8s/ by the installer, so there is nothing to create.

  1. Render the templates using the values from spark_env.sh 

    cd ~/privacera-oss-plugin/k8s
./replace.sh

  2. Create the namespace, service account, role, Secret and deployment 

    ./apply.sh

    Note

    apply.sh builds the Secret from everything in secret/, so the truststore and the properties file naming its password reach the pod at runtime rather than being baked into the image.

Validate the Deployment

  1. Check the status of the pods 

    export SPARK_NAME_SPACE=<SPARK_NAME_SPACE>
kubectl get pods -n ${SPARK_NAME_SPACE}

  2. Access the pod 

    kubectl exec -it <SPARK_PLUGIN_POD_NAME> -n ${SPARK_NAME_SPACE} -- /bin/bash