@Taisia00, Did the response assist in resolving your query? If it did, kindly mark the relevant reply as the solution to aid others in finding the answer more easily in the future.  ... View more

@Meepoljd, Did the response assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future. ... View more

@LSIMS you mentioned it is occasional , does it mean that it is failing only on few nodes ? Can you check with Informatica team on how to pass the kerberos keytab cerds .. I found this Informatica Article on passing the keytab details for spark+kafka setup . https://docs.informatica.com/data-engineering/data-engineering-integration/10-2-2/big-data-management-administrator-guide/connections/configuring-hadoop-connection-properties/spark-advanced-properties.html   ... View more

@Mamun_Shaheed Did the response help resolve your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future. ... View more

Hi @haridjh  Thanks for reply. Procedure you described in your reply is utilizing HDFS as a store for JAR files used by Spark job. We don't have problem to utilize HDFS in Spark job, problem is when trying to access Ozone FS, e.g. (ofs) when job is submitted via LIVY. 1. Access files on Ozone in spark job e.g.:   df = spark.read.parquet("ofs://ozone-service/volume/bucket/parquet")   2. Python job submitted via Livy:   kinit user   curl --negotiate -k -v -u : -X POST \ -H "Content-Type: application/json" \ --data '{ "file": "ozone_access.py"}' \ https://livy:28998/batches      3. Job is failing with:   Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]     When we are trying to access Ozone normally via spark-shell or spark-submit, everything works fine, e.g.:   spark-shell \ --keytab ${KEY_TAB} \ --principal ${PRINCIPAL} \ --conf spark.yarn.access.hadoopFileSystems=o3fs://bucket1.vol1.om.host.example.com:9862     Setting keytab and principal is not possible when submitting job via Livy, because we are using proxy users with Livy.  Thanks.     ... View more

Hi @haridjh, The server is supposed to stream all the NetFlow data to NiFi. However, why does the time delay between sending the NetFlow and receiving it in NiFi matter? ... View more

I did as suggested by colleague @SAMSAL. I saved column by column, until I discovered that I had corrupted data. I had a data where the year was like "1111". As I had little experience, it was difficult to spot possible errors. Thank you everyone, especially my colleague @dianaDianaTorres ... View more

Thank you @haridjh ! It worked! I am even further confused because the underlying parquet file is already partitioned. But when inserting a repartition() the code works! ... View more