Hello Dear all, I am encountering an issue with an Oozie workflow that involves moving data from an external table to HDFS. The Oozie job successfully completes the process of bringing data from the external table to HDFS. After that, Sqoop should transfer this data into an ORC table. The issue occurs at a specific point in the Oozie workflow, where it tries to read two files from an HDFS directory: one containing the YARN application_id and the other with the instructions for database insertion. The job fails, and the directory remains empty. In the logs, I get the following error: org.apache.hadoop.ipc.Client - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Context: This error began after an incident where directories in HDFS were deleted. This may have affected the failover process and high availability (HA) of HDFS. The error seems to be related to the standby NameNode, suggesting that the client (Sqoop/Oozie) is trying to read/write to the standby NameNode, where read/write operations are not allowed. My Hypothesis: This problem may be related to the HDFS failover. The incident with the deleted directories may have affected HA, preventing Sqoop or Oozie from correctly communicating with the active NameNode, resulting in the failure to register files and the subsequent error. If the standby NameNode is not properly synchronized, or if failover is not functioning correctly, this could explain why the job is unable to write files to the HDFS directory, directly affecting the progress of the Oozie/Sqoop job. Questions: Could this issue be related to HDFS failover and the HA configuration being affected by the deleted directories? How can I validate whether the problem is related to HDFS HA and failover? Is there a way to force Sqoop/Oozie to properly use the active NameNode instead of the standby? I have checked the HA configuration, and failover seems to be functioning as the standby takes over when the active NameNode is restarted. However, the error persists when trying to read or write to HDFS. I appreciate any help and suggestions. Best regards ... View more

In the development cluster, the NiFi nodes do not show data or connection to Cloudera Manager, but if I access them via WebUI, the nodes appear properly connected and functional. Attached are NiFi logs and screen prints as evidence Previously I was instructed to review the Ranger policies as described in the Cloudera documentation, but the problem persisted, attached are the current Ranger settings. I also found the following error curious: Cookie rejected [hadoop.auth=""u=nifi&p=nifi/v321d186.prevnet@DEVDATALAKE.CDPDTP&t=kerberos&e=1721689709429&s=qJPbU6hDNqLtkL1BokMp...", version:0, domain:v321d175.prevnet, path:/solr/ranger_audits_shard1_replica_n1, expiry:null] Domain attribute "v321d175.prevnet" violates the Netscape cookie specification However, in the prod cluster the same error appears in the logs and I don't have this problem ... View more

We are experiencing an issue with converting data from string to timestamp in Hive, where the timestamps undergo unexpected hour changes during conversion and processing. We would like to verify if this could be a bug in the tool or if there are additional configurations required. Process and Steps to Reproduce the Issue: Loading Data into HDFS: 1. We use the hdfs put command to load a positional file into HDFS and store it in a directory accessible via an external table ex_sa_beneficios_mantidos where the data is in string format. 2. We create an external table in Hive to access the data as strings. 3. Converting String to Timestamp and Writing to ORC Table: When converting the string to timestamp, we notice an alteration in the hour of the dt_afast_trabalho field. 4. Verifying the Conversion and Data Reading: When reading the data from the ORC table, the hours in the dt_afast_trabalho field were different from expected. 5. Counting Timestamps by Hour: We performed a count by hour of the timestamps to verify the changes. Spark Configuration in Zeppelin: When initializing Hive in Zeppelin, we used the following parameters: %livy.spark from pyspark_llap import HiveWarehouseSession hive = HiveWarehouseSession.session(spark).build() spark.conf.set("spark.datasource.hive.warehouse.read.mode", value="SECURE_ACCESS") spark.conf.set("spark.datasource.hive.warehouse.load.staging.dir", value="hdfs://pcdpclusterdatalakennha:8020/dados/datalake/zona_temporaria/dadosprovisorios/hwc") spark.conf.set("spark.sql.legacy.allowCreatingManagedTableUsingNoemptyLocation", value="true") spark.conf.set("spark.sql.session.timeZone", "America/Sao_Paulo") Expected Behavior: We expect the hour values of the dt_afast_trabalho field to remain consistent throughout the conversion and processing stages, respecting the America/Sao_Paulo timezone. Current Behavior: We observe that the hour values of the dt_afast_trabalho field are being altered during the process of converting from string to timestamp and subsequent processing in Hive and Spark. ... View more