@Meepoljd Spark will read the metadata of the table and get the list of files to be read and if there is continuous changes ( delete/overwrite)  on the table and on the execution of read operation, the spark job can fail with FNF exception. The option here would be to minimise the change duration and run the job when there are no changes  OR catch the exception within the spark code and rebuild the dataframe.  ... View more

@LSIMS you mentioned it is occasional , does it mean that it is failing only on few nodes ? Can you check with Informatica team on how to pass the kerberos keytab cerds .. I found this Informatica Article on passing the keytab details for spark+kafka setup . https://docs.informatica.com/data-engineering/data-engineering-integration/10-2-2/big-data-management-administrator-guide/connections/configuring-hadoop-connection-properties/spark-advanced-properties.html   ... View more

@Mamun_Shaheed Did the response help resolve your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future. ... View more

Hi @jaris  How are you passing the kerberos ticket? Did you run a kinit , before the livy statement execution.  Obtain a kerberos ticket via kinit and then try to execute the below sample job  1. Copy the JAR to HDFS: # hdfs dfs -put /opt/cloudera/parcels/CDH/jars/spark-examples<VERSION>.jar /tmp 2. Make sure the JAR is present. # hdfs dfs -ls /tmp/ 3. CURL command to run the Spark job using Livy API. # curl -v -u: --negotiate -X POST --data '{"className": "org.apache.spark.examples.SparkPi", "jars": ["/tmp/spark-examples<VERSION>.jar"], "name": "livy-test", "file": "hdfs:///tmp/spark-examples<VERSION>.jar", "args": [10]}' -H "Content-Type: application/json" -H "X-Requested-By: User" http://<LIVY_NODE>:<PORT>/batches 4. Check for the running and completed Livy sessions. # curl http://<LIVY_NODE>:<PORT>/batches/ | python -m json.tool   NOTE:         * Change the JAR version ( <VERSION> ) according your CDP version.         * Replace the LIVY_NODE and PORT with the actual values.         * If you are running the cluster in secure mode, then make sure you have a valid Kerberos ticket and use the Kerberos authentication in curl command. ... View more

@MarinaM the template is sent from netflow server. we need to check when the template would be sent by netflow server and it can be cached on the collector end .(listenNetflow processor)  ... View more

@Brunno  In ExecuteSQL , Query result will be converted to Avro format. Therefore , use the Convert Avro* or related processors to have the avro data converted to correct formatted file and then update to the target DB   https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/2.0.0/org.apache.nifi.processors.standard.ExecuteSQL/index.html ... View more