Steps to enable logs for Databricks cluster

Perform the following steps to enable debug logs for com.privacera package:

  1. Prerequisites:
    • A running Databricks cluster.
  2. Log in to Databricks Web UI
  3. Click on the Workspace icon on the sidebar
  4. Navigate to folder where you want to create the debug script.

  5. Click on Create icon on top right side, then click on File, set file name as debug.sh and add below content to the file:

    Enable debug logs in Databricks Runtime Version 10.4 LTS

    Note: ROOT_LOG_LEVEL, PRIVACERA_LOG_LEVEL, RANGER_LOG_LEVEL can be set to INFO/DEBUG/TRACE as per usage.

    debug.sh
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
    #!/bin/bash

PRIVACERA_OUT_FILE=/root/privacera/privacera.out
PRIVACERA_CLUSTER_LOGS_DIR=${PRIVACERA_CLUSTER_LOGS_DIR:-/dbfs/privacera/cluster-logs/${DB_CLUSTER_NAME}}
ROOT_LOG_LEVEL=INFO
PRIVACERA_LOG_LEVEL=DEBUG
RANGER_LOG_LEVEL=INFO

function log(){
  msg=$1
  currentTime=`date`
  echo "${currentTime} : ${msg} " >> ${PRIVACERA_OUT_FILE}
}

log "======================debug.sh execution started!!!======================"
log "enabling ${PRIVACERA_LOG_LEVEL} logs in driver...."
DRIVER_LOG4J_CONF_FILE=/databricks/spark/dbconf/log4j/driver/log4j.properties
echo "#Privacera HERE " >> ${DRIVER_LOG4J_CONF_FILE}
echo "log4j.rootLogger=${ROOT_LOG_LEVEL},publicFile" >> ${DRIVER_LOG4J_CONF_FILE}
echo "log4j.category.com.privacera=${PRIVACERA_LOG_LEVEL},publicFile" >> ${DRIVER_LOG4J_CONF_FILE}
echo "log4j.additivity.com.privacera=false" >> ${DRIVER_LOG4J_CONF_FILE}
echo "log4j.category.org.apache.ranger=${RANGER_LOG_LEVEL},publicFile" >> ${DRIVER_LOG4J_CONF_FILE}
echo "log4j.additivity.org.apache.ranger=false" >> ${DRIVER_LOG4J_CONF_FILE}
echo "log4j.appender.publicFile.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss,SSS} %p %c:%L [%t] - %m%n" >> ${DRIVER_LOG4J_CONF_FILE}
log "enabled ${PRIVACERA_LOG_LEVEL} logs in driver successfully!!"

log "enabling ${PRIVACERA_LOG_LEVEL} logs in executor...."
EXECUTOR_LOG4J_CONF_FILE=/databricks/spark/dbconf/log4j/executor/log4j.properties
echo "#Privacera HERE " >> ${EXECUTOR_LOG4J_CONF_FILE}
echo "log4j.rootLogger=${ROOT_LOG_LEVEL}, console" >> ${EXECUTOR_LOG4J_CONF_FILE}
echo "log4j.category.com.privacera=${PRIVACERA_LOG_LEVEL},console" >> ${EXECUTOR_LOG4J_CONF_FILE}
echo "log4j.additivity.com.privacera=false" >> ${EXECUTOR_LOG4J_CONF_FILE}
echo "log4j.category.org.apache.ranger=${RANGER_LOG_LEVEL},console" >> ${EXECUTOR_LOG4J_CONF_FILE}
echo "log4j.additivity.org.apache.ranger=false" >> ${EXECUTOR_LOG4J_CONF_FILE}
echo "log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss,SSS} %p %c:%L [%t] - %m%n" >> ${EXECUTOR_LOG4J_CONF_FILE}
echo "#Privacera HERE " >> ${EXECUTOR_LOG4J_CONF_FILE}
log "enabled ${PRIVACERA_LOG_LEVEL} logs in executor successfully!!"
log "======================debug.sh execution completed!!!======================"

log "Copying privacera.out to ${PRIVACERA_CLUSTER_LOGS_DIR}"
cp ${PRIVACERA_OUT_FILE} ${PRIVACERA_CLUSTER_LOGS_DIR}/privacera.out
    Enable debug logs in Databricks Runtime Version 11.3 LTS and above

    Note:
    - You can set ROOT_LOG_LEVEL, PRIVACERA_LOG_LEVEL, and RANGER_LOG_LEVEL to INFO, DEBUG, or TRACE based on your logging needs.
    - Modify LOG_FILE_SIZE and LOG_FILE_BACKUP values as per your log retention and rotation requirements.

    debug.sh
      1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
    #!/bin/bash

PRIVACERA_OUT_FILE=/root/privacera/privacera.out
PRIVACERA_CLUSTER_LOGS_DIR=${PRIVACERA_CLUSTER_LOGS_DIR:-/dbfs/privacera/cluster-logs/${DB_CLUSTER_NAME}}
ROOT_LOG_LEVEL="INFO"
PRIVACERA_LOG_LEVEL="DEBUG"
RANGER_LOG_LEVEL="INFO"
LOG_FILE_SIZE="250MB"
LOG_FILE_BACKUP="10"

function log(){
  msg=$1
  currentTime=`date`
  echo "${currentTime} : ${msg} " >> ${PRIVACERA_OUT_FILE}
}

log "======================debug.sh execution started!!!======================"
log "enabling logs in driver...."
LOG4J_2_DRIVER_FILE_PATH="/databricks/spark/dbconf/log4j/driver/log4j2.xml"
cat << EOF >  ${LOG4J_2_DRIVER_FILE_PATH}
<?xml version="1.0" encoding="UTF-8"?><Configuration status="INFO" packages="com.databricks.logging" shutdownHook="disable">
  <Appenders>
    <RollingFile name="publicFile.rolling" fileName="logs/log4j-active.log" filePattern="logs/log4j-%d{yyyy-MM-dd-HH}.log.gz" immediateFlush="true" bufferedIO="false" bufferSize="8192" createOnDemand="true">
      <Policies>
        <TimeBasedTriggeringPolicy interval="1" modulate="true"/>  <!-- Rolls over every hour -->
        <SizeBasedTriggeringPolicy size="${LOG_FILE_SIZE}"/>  <!-- Rolls over when file reaches this size -->
      </Policies>
      <DefaultRolloverStrategy max="${LOG_FILE_BACKUP}" fileIndex="min"/>  <!-- Keep only the number of configured last backup files -->
      <PatternLayout pattern="%d{yy/MM/dd HH:mm:ss,SSS} %p %c:%L [%t] - %m%n%ex"/>
    </RollingFile>
    <Rewrite name="publicFile.rolling.rewrite">
      <ServiceRewriteAppender/>
      <AppenderRef ref="publicFile.rolling"/>
    </Rewrite>
    <RollingFile name="privateFile.rolling" fileName="logs/active.log" filePattern="logs/%d{yyyy-MM-dd-HH}.log.gz" immediateFlush="true" bufferedIO="false" bufferSize="8192" createOnDemand="true">
      <Policies>
        <TimeBasedTriggeringPolicy/>
      </Policies>
      <PatternLayout pattern="%d{yy/MM/dd HH:mm:ss,SSS} %p %c{1}: %m%n%ex"/>
    </RollingFile>
    <Rewrite name="privateFile.rolling.rewrite">
      <ServiceRewriteAppender/>
      <AppenderRef ref="privateFile.rolling"/>
    </Rewrite>
    <RollingFile name="com.databricks.UsageLogging.appender" fileName="logs/usage.json" filePattern="logs/%d{yyyy-MM-dd-HH}.usage.json.gz" immediateFlush="true" bufferedIO="false" bufferSize="8192" createOnDemand="true">
      <Policies>
        <TimeBasedTriggeringPolicy/>
      </Policies>
      <PatternLayout pattern="%m%n%ex"/>
    </RollingFile>
    <RollingFile name="com.databricks.ProductLogging.appender" fileName="logs/product.json" filePattern="logs/%d{yyyy-MM-dd-HH}.product.json.gz" immediateFlush="true" bufferedIO="false" bufferSize="8192" createOnDemand="true">
      <Policies>
        <TimeBasedTriggeringPolicy/>
      </Policies>
      <PatternLayout pattern="%m%n%ex"/>
    </RollingFile>
    <RollingFile name="com.databricks.LineageLogging.appender" fileName="logs/lineage.json" filePattern="logs/%d{yyyy-MM-dd-HH}.lineage.json.gz" immediateFlush="true" bufferedIO="false" bufferSize="8192" createOnDemand="true">
      <Policies>
        <TimeBasedTriggeringPolicy/>
      </Policies>
      <PatternLayout pattern="%m%n%ex"/>
    </RollingFile>
    <RollingFile name="com.databricks.MetricsLogging.appender" fileName="logs/metrics.json" filePattern="logs/%d{yyyy-MM-dd-HH}.metrics.json.gz" immediateFlush="true" bufferedIO="false" bufferSize="8192" createOnDemand="true">
      <Policies>
        <TimeBasedTriggeringPolicy/>
      </Policies>
      <PatternLayout pattern="%m%n%ex"/>
    </RollingFile>
    <RollingFile name="dltExecution.rolling" fileName="logs/dlt-execution.log" filePattern="logs/dlt-execution-%d{yyyy-MM-dd-HH}.log.gz" immediateFlush="true" bufferedIO="false" bufferSize="8192" createOnDemand="true">
      <Policies>
        <TimeBasedTriggeringPolicy/>
      </Policies>
      <PatternLayout pattern="%d{yy/MM/dd HH:mm:ss,SSS} %p %c:%L [%t] - %m%n%ex"/>
    </RollingFile>
    <Rewrite name="dltExecution.rolling.rewrite">
      <ServiceRewriteAppender/>
      <AppenderRef ref="dltExecution.rolling"/>
    </Rewrite>
  </Appenders>

  <Loggers>
    <Root level="${ROOT_LOG_LEVEL}">
      <AppenderRef ref="publicFile.rolling.rewrite"/>
    </Root>
    <Logger name="privateLog" level="INFO" additivity="false">
      <AppenderRef ref="privateFile.rolling.rewrite"/>
    </Logger>
    <Logger name="com.databricks.UsageLogging" level="INFO" additivity="false">
      <AppenderRef ref="com.databricks.UsageLogging.appender"/>
    </Logger>
    <Logger name="com.databricks.ProductLogging" level="INFO" additivity="false">
      <AppenderRef ref="com.databricks.ProductLogging.appender"/>
    </Logger>
    <Logger name="com.databricks.LineageLogging" level="INFO" additivity="false">
      <AppenderRef ref="com.databricks.LineageLogging.appender"/>
    </Logger>
    <Logger name="com.databricks.MetricsLogging" level="INFO" additivity="false">
      <AppenderRef ref="com.databricks.MetricsLogging.appender"/>
    </Logger>
    <Logger name="com.databricks.pipelines" level="INFO" additivity="true">
      <AppenderRef ref="dltExecution.rolling.rewrite"/>
    </Logger>
    <Logger name="org.apache.spark.rdd.NewHadoopRDD" level="WARN"/>
    <Logger name="com.microsoft.azure.datalake.store" level="DEBUG"/>
    <Logger name="com.microsoft.azure.datalake.store.HttpTransport" level="DEBUG">
      <RegexFilter onMatch="DENY" onMismatch="NEUTRAL" regex=".*HTTPRequest,Succeeded.*"/>
    </Logger>
    <Logger name="com.microsoft.azure.datalake.store.HttpTransport.tokens" level="DEBUG"/>
    <!-- privacera -->
    <Logger name="com.privacera" level="${PRIVACERA_LOG_LEVEL}" additivity="false">
      <AppenderRef ref="publicFile.rolling.rewrite" />
    </Logger>
    <!-- ranger -->
    <Logger name="org.apache.ranger" level="${RANGER_LOG_LEVEL}" additivity="false">
      <AppenderRef ref="publicFile.rolling.rewrite" />
    </Logger>
  </Loggers>

</Configuration>
EOF
log "enabled logs in driver successfully!!"

log "enabling logs in executor...."
LOG4J_2_EXECUTOR_FILE_PATH="/databricks/spark/dbconf/log4j/executor/log4j2.xml"    
cat << EOF >  ${LOG4J_2_EXECUTOR_FILE_PATH}
<?xml version="1.0" encoding="UTF-8"?><Configuration status="INFO" packages="com.databricks.logging" shutdownHook="disable">
  <Appenders>
    <Console name="console" target="SYSTEM_ERR">
    <PatternLayout pattern="%d{yy/MM/dd HH:mm:ss,SSS} %p %c:%L [%t] - %m%n%ex"/>
    </Console>
    <Rewrite name="console.rewrite">
      <ServiceRewriteAppender/>
      <AppenderRef ref="console"/>
    </Rewrite>
    <RollingFile name="com.databricks.UsageLogging.appender" fileName="logs/usage.json" filePattern="logs/%d{yyyy-MM-dd-HH}.usage.json.gz" immediateFlush="true" bufferedIO="false" bufferSize="8192" createOnDemand="true">
      <Policies>
        <TimeBasedTriggeringPolicy/>
      </Policies>
      <PatternLayout pattern="%m%n%ex"/>
    </RollingFile>

    <RollingFile name="publicFile.rolling" fileName="logs/log4j-active.log" filePattern="logs/log4j-%d{yyyy-MM-dd-HH}.log.gz" immediateFlush="true" bufferedIO="false" bufferSize="8192" createOnDemand="true">
      <Policies>
        <TimeBasedTriggeringPolicy interval="1" modulate="true"/>  <!-- Rolls over every hour -->
        <SizeBasedTriggeringPolicy size="${LOG_FILE_SIZE}"/>  <!-- Rolls over when file reaches this size -->
      </Policies>
      <DefaultRolloverStrategy max="${LOG_FILE_BACKUP}" fileIndex="min"/>  <!-- Keep only the number of configured last backup files -->
      <PatternLayout pattern="%d{yy/MM/dd HH:mm:ss,SSS} %p %c:%L [%t] - %m%n%ex"/>
    </RollingFile>
    <Rewrite name="publicFile.rolling.rewrite">
      <ServiceRewriteAppender/>
      <AppenderRef ref="publicFile.rolling"/>
    </Rewrite>
  </Appenders>

  <Loggers>
    <Logger name="org.apache.spark.rdd.NewHadoopRDD" level="WARN"/>
    <Logger name="com.microsoft.azure.datalake.store" level="DEBUG"/>
    <Logger name="com.microsoft.azure.datalake.store.HttpTransport" level="DEBUG">
      <RegexFilter onMatch="DENY" onMismatch="NEUTRAL" regex=".*HTTPRequest,Succeeded.*"/>
    </Logger>
    <Logger name="com.microsoft.azure.datalake.store.HttpTransport.tokens" level="DEBUG"/>
    <Logger name="com.databricks.UsageLogging" level="INFO" additivity="false">
      <AppenderRef ref="com.databricks.UsageLogging.appender"/>
    </Logger>
    <Root level="${ROOT_LOG_LEVEL}">
      <AppenderRef ref="console.rewrite"/>
      <AppenderRef ref="publicFile.rolling.rewrite"/>
    </Root>

    <!-- privacera -->
    <Logger name="com.privacera" level="${PRIVACERA_LOG_LEVEL}" additivity="false">
      <AppenderRef ref="console.rewrite" />
      <AppenderRef ref="publicFile.rolling.rewrite"/>
    </Logger>
    <!-- ranger -->
    <Logger name="org.apache.ranger" level="${RANGER_LOG_LEVEL}" additivity="false">
      <AppenderRef ref="console.rewrite" />
      <AppenderRef ref="publicFile.rolling.rewrite"/>
    </Logger>
  </Loggers>

</Configuration>
EOF
log "enabled logs in executor successfully!!"
log "======================debug.sh execution completed!!!======================"

log "Copying privacera.out to ${PRIVACERA_CLUSTER_LOGS_DIR}"
cp ${PRIVACERA_OUT_FILE} ${PRIVACERA_CLUSTER_LOGS_DIR}/privacera.out

  6. Save debug.sh file.

  7. Now click on the Compute icon on the sidebar, then click on the cluster where logs need to be enabled and Edit this cluster.
  8. Scroll down to Advanced option and click on Init Scripts.
    • Select Workspace as the source for init scripts.
    • Specify the debug.sh script path that created in step 5, after plugin scripts (ranger_enable.sh for FGAC and ranger_enable_scala.sh for OLAC), which must be added as first script.
  9. Click the Add button.

  10. Now in Logging section, define path to store logs

    • Select DBFS as Destination for logs folder.
    • in Log Path add the path, e.g. dbfs:/logs/<cluster-name>/<folder>

  11. Click Confirm to save changes.

  12. Click on Start to restart the cluster.
  13. To copy logs on local machine, run below command:
    Bash
    1
    dbfs cp -r dbfs:/logs/<cluster-name>/<folder> <local-folder>
  14. Once the cluster is started, Check privacera.out logs for validation of debug.sh script execution. Refer here for more details.

Applies to the Databricks Spark connector configured in Runtime.

Debug logging is set on the connector; the bundle's init script writes the log4j2.xml.

  1. In the portal, open the connector and turn on Enable debug logging. Under Logging, set Privacera log level to DEBUG. The section carries these fields:

    Field Definition Default
    Privacera log level Level for the Privacera plugin classes. INFO, DEBUG or TRACE. INFO
    Ranger log level Level for the Ranger policy engine and audit client. INFO
    Root log level Level for everything else on the cluster. INFO
    Log file size Size at which a log file rolls over. 250MB
    Log files retained How many rolled files are kept. 10

  2. Save the connector and download the bundle again. The levels only reach the plugin through a freshly downloaded bundle.

  3. Upload the bundle's init script to the workspace or S3, replacing the previous copy. See Upload the init script.

  4. Restart the cluster.

  5. Collect the logs the same way as in the other tab - set the cluster's Logging destination, then copy the folder down with dbfs cp -r.