Hi @Meepoljd  Please let me know still you need any help on this issue. If any of the above solutions is helped then mark Accept as Solution. ... View more

Hi @zhuw.bigdata  To locate Spark logs, follow these steps: Access the Spark UI: Open the Spark UI in your web browser. Identify Nodes: Navigate to the Executors tab to view information about the driver and executor nodes involved in the Spark application. Determine Log Directory: Within the Spark UI, find the Hadoop settings section and locate the value of the yarn.nodemanager.log-dirs property. This specifies the base directory for Spark logs on the cluster. Access Log Location: Using a terminal or SSH, log in to the relevant node (driver or executor) where the logs you need are located. Navigate to Application Log Directory: Within the yarn.nodemanager.log-dirs directory, access the subdirectory for the specific application using the pattern application_${appid}, where ${appid} is the unique application ID of the Spark job. Find Container Logs: Within the application directory, locate the individual container log directories named container_{$contid}, where ${contid} is the container ID. Review Log Files: Each container directory contains the following log files generated by that container: stderr: Standard error output stdin: Standard input (if applicable) syslog: System-level logs ... View more

Hi @sonnh  Generally it is not advisable to read and write the same table at a time. It can result in anything between data corruption and complete data loss in case of failure. As a temporary solution, First create a temporary view by reading the table data and later you can use that data and finally save the data to destination table. Reference:  https://stackoverflow.com/questions/38746773/read-from-a-hive-table-and-write-back-to-it-using-spark-sql https://issues.apache.org/jira/browse/SPARK-27030 ... View more

Hi @Taries  As I mentioned previously, only the hbase.spark.query.timerange parameter can be used for filtering data during read. The hbase.spark.scan parameter wouldn't be set for this purpose. To filter the data after reading, you can apply a Spark WHERE or filter clause with your desired conditions.   ... View more

Hi @Taries  You need to use the following two parameters to apply filter. hbase.spark.query.timerange.start hbase.spark.query.timerange.end Reference: https://github.com/apache/hbase-connectors/blob/307607cf7287084b3ce49cdd96d094e2ede9363a/spark/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/datasources/HBaseSparkConf.scala#L65 ... View more

Hi @sind65  Could you please clarify few things before proceeding any solutions: 1. Are you using CDP or CDH? 2. How are you submitting the spark job?  3. Are you using kinit or you have specified principal/keytab? 4. Have you followed the following documentation to setup and run the sample example https://community.cloudera.com/t5/Community-Articles/Spark-Ozone-Integration-in-CDP/ta-p/323132 ... View more