@Anderosn  1. If the content of your flow file is too large to be inserted into a single CLOB column, you can split it into smaller chunks and insert each chunk into the database separately. 2. Instead of storing the content in a CLOB column, you can consider storing it in a BLOB (Binary Large Object) column in your database. BLOB columns can store binary data, including large files, without the size limitations of CLOB columns. 3. Store the content of the flow file in an external storage system (e.g., HDFS, Amazon S3) and then insert the reference (e.g., file path or URL) into the database. This approach can be useful if the database has limitations on the size of CLOB or BLOB columns 4. If ExecuteScript is not approved, consider using an external script or application to perform the insertion into the database. You can trigger the script or application from NiFi using ExecuteProcess or InvokeHTTP processors   Regards, Chethan YM   ... View more

This is how to resolve this problem: To resolve this issue, set this property to 0 and restart Impala: CM > Impala > Configuration > Impala Daemon command-line safety valve: -idle_client_poll_period_s=0 This is a startup flag, not a query option. Its default value is 30 seconds and that is why the session in the above excerpt was closed after 30 secs. By setting it to 0, Impala will not periodically check the client connection. The client connections will remain open until they are explicitly closed on the client applications' side. ... View more

Hi @MrBeasr  Review the oozie logs for this workflow if there is anything suspicious and you can paste here. oozie job -oozie http://<oozie-server-host>:11000 -log <workflow-id>   Regards, Chethan YM     ... View more

@shofialau Did the response assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.  ... View more

@RobertusAgung, Did the response assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.  ... View more

Hive typically relies on the schema definition provided during table creation, and it doesn't perform automatic type conversion while loading data. If there's a mismatch between the data type in the CSV file and the expected data type in the Hive table, it may result in null or incorrect values. Use the CAST function to explicitly convert the data types during the INSERT statement.   INSERT INTO TABLE target_table SELECT CAST(column1 AS INT), CAST(column2 AS STRING), ... FROM source_table;   Preprocess your CSV data before loading it into Hive. You can use tools like Apache NiFi or custom scripts to clean and validate the data before ingestion. Remember to thoroughly validate and clean your data before loading it into Hive to avoid unexpected issues. Also, the choice of method depends on your specific use case and the level of control you want over the data loading process. ... View more

@Kolli  Based on the logs and the Spark-submit command provided, it seems like there are discrepancies between the authentication mechanisms used in the driver and the executor environments, leading to authentication errors. Here are some potential issues and solutions: Mismatch in Authentication Mechanisms: The driver seems to authenticate using Kerberos (kerberos), while the executor uses simple authentication (SIMPLE). Ensure consistency in the authentication mechanisms across the driver and executor environments. Kerberos Configuration: Verify that the Kerberos configuration (krb5.conf) provided in the spark.driver.extraJavaOptions and spark.executor.extraJavaOptions is correct and accessible by both the driver and executor. Check if the Kerberos principal and keytab specified in the spark-submit command are accurate and valid. SPNEGO Configuration: Ensure that SPNEGO authentication is properly configured for the Spark Elasticsearch connector. Verify that the SPNEGO principal (elasticsearch/hadoop.hadoop.com@HADOOP.COM) specified in the spark-submit command matches the one configured in the environment. Permission Issues: Check the permissions of the keytab file (user.keytab) specified in the spark-submit command to ensure that it is accessible by both the driver and executor. Token Renewal: Review the token renewal mechanism to ensure that tokens are properly renewed and propagated to the executor. To address the issue, consider the following steps: Ensure that both the driver and executor environments are configured consistently for Kerberos authentication. Double-check all Kerberos-related configurations, including the Kerberos principal, keytab, and krb5.conf file paths. Verify that the SPNEGO authentication settings are correctly configured for the Spark Elasticsearch connector. Check for any permission issues with the keytab file or other Kerberos-related files. Review the token renewal mechanism to ensure proper token propagation.   Regards, Chethan YM ... View more

@liorh  The error message you're encountering, "[HY000] [Cloudera][ThriftExtension] (11) Error occurred while contacting server: EAGAIN (timed out)," typically indicates a timeout issue while attempting to establish a connection with the Impala server using the Thrift protocol. Below is just generic troubleshooting tips, You need to analyse and troubleshoot your environment. Causes of the Error: Network Latency: The error may occur due to network latency or connectivity issues between your application and the Impala server. This can lead to timeouts during the connection attempt. Server Load: If the Impala server is under heavy load or experiencing resource constraints, it may not be able to handle incoming connection requests promptly, resulting in timeouts. Thrift Protocol Issues: The error message mentions using a binary mechanism for authentication. If there are inconsistencies or misconfigurations in the Thrift protocol settings between your application and the Impala server, it could lead to connection failures. Dealing with the Error: Retry Mechanism: As you mentioned, implementing a retry mechanism in your software to make multiple attempts to run the action is a good approach. This can help mitigate transient network issues or server load spikes that may cause the initial connection attempt to fail. Preventing the Problem: Optimize Network Configuration: Review and optimize the network configuration between your application and the Impala server to minimize latency and improve reliability. This may include configuring network settings, optimizing routing, or using dedicated network connections. Server Performance Tuning: Monitor the performance of the Impala server and address any resource bottlenecks or performance issues that could lead to connection timeouts. This may involve optimizing server configuration, increasing resources, or tuning Impala parameters. Thrift Protocol Configuration: Ensure that the Thrift protocol settings, including authentication mechanisms, are correctly configured and consistent between your application and the Impala server.    Review the trace level driver logs, Impalad, statestore and catalog logs at the time of issue to see if we can get some details of the issue.   Regards, Chethan YM     ... View more

@krishna2023  It seems like you're encountering issues with loading data into partitions in Impala after executing the provided steps.  Create table as select * from db2.t1 where 1=2: This step creates an empty table db1.t1 based on the schema of db2.t1 without any data. Ensure that the table schema matches between db1.t1 and db2.t1. Alter table set location: After creating the empty table, you're altering its location to a new path. Make sure that the specified path exists and has the necessary permissions for Impala to read and write data. Add partition for every day: Adding partitions should involve specifying the loading date for each partition and its corresponding HDFS directory path. Double-check that the HDFS directory paths specified in each partition definition are correct and accessible by Impala. Refresh table: The REFRESH command updates the metadata of the table to reflect changes made in the underlying data directory. After adding partitions, running REFRESH is necessary to inform Impala about the new partitions. Make sure to execute this command after adding partitions. Compute stats: The COMPUTE STATS command gathers statistics about the table, which helps Impala optimize query execution. While this command is not directly related to loading data into partitions, it's good practice to run it after making significant changes to the table. To further troubleshoot the issue, consider the following additional steps: Check Impala logs for any error messages or warnings that might indicate issues with loading data or adding partitions. Verify that the data files corresponding to the partitions are present in the specified HDFS directory paths. Ensure that the partitioning column (loading_date) values in the data files match the partition definitions specified in the ALTER TABLE statements.   Regards, Chethan YM ... View more