Overwriting Java Options in Spark Job Code¶

This page covers a configuration issue that prevents Privacera from enforcing access control in Apache Spark jobs and deployments using Spark Operator.

Problem

Setting spark.executor.extraJavaOptions or spark.driver.extraJavaOptions directly in the Spark job code (for example, in Python or Scala) overrides the default Privacera agent configuration. As a result, the Privacera agent may fail to load correctly.

Default Configuration

By default, the following properties are defined in /opt/spark/conf/spark-defaults.conf:

spark.executor.extraJavaOptions -javaagent:/opt/spark/jars/privacera-agent.jar -Dlog4j.configurationFile=file:///privacera-conf/log4j2.properties

spark.driver.extraJavaOptions -javaagent:/opt/spark/jars/privacera-agent.jar -Dlog4j.configurationFile=file:///privacera-conf/log4j2.properties

Incorrect Configuration Example

For example, in Python:

1
2
3
4

spark = SparkSession.builder \
    .config("spark.executor.extraJavaOptions", "<PROPERTY>") \
    .config("spark.driver.extraJavaOptions", "<PROPERTY>") \
    .getOrCreate()

Impact

• The Privacera agent and required JARs are not loaded on the Spark driver or executors.
• The Spark job may still succeed; however, Privacera will not be able to enforce access control.

Solution

Instead of overwriting these properties, append the custom options to the existing Privacera configuration. For example, in Python:

1
2
3
4

spark = SparkSession.builder \
    .config("spark.executor.extraJavaOptions", "-javaagent:/opt/spark/jars/privacera-agent.jar -Dlog4j.configurationFile=file:///privacera-conf/log4j2.properties <your-additional-options>") \
    .config("spark.driver.extraJavaOptions", "-javaagent:/opt/spark/jars/privacera-agent.jar -Dlog4j.configurationFile=file:///privacera-conf/log4j2.properties <your-additional-options>") \
    .getOrCreate()