@liorh  The error message you're encountering, "[HY000] [Cloudera][ThriftExtension] (11) Error occurred while contacting server: EAGAIN (timed out)," typically indicates a timeout issue while attempting to establish a connection with the Impala server using the Thrift protocol. Below is just generic troubleshooting tips, You need to analyse and troubleshoot your environment. Causes of the Error: Network Latency: The error may occur due to network latency or connectivity issues between your application and the Impala server. This can lead to timeouts during the connection attempt. Server Load: If the Impala server is under heavy load or experiencing resource constraints, it may not be able to handle incoming connection requests promptly, resulting in timeouts. Thrift Protocol Issues: The error message mentions using a binary mechanism for authentication. If there are inconsistencies or misconfigurations in the Thrift protocol settings between your application and the Impala server, it could lead to connection failures. Dealing with the Error: Retry Mechanism: As you mentioned, implementing a retry mechanism in your software to make multiple attempts to run the action is a good approach. This can help mitigate transient network issues or server load spikes that may cause the initial connection attempt to fail. Preventing the Problem: Optimize Network Configuration: Review and optimize the network configuration between your application and the Impala server to minimize latency and improve reliability. This may include configuring network settings, optimizing routing, or using dedicated network connections. Server Performance Tuning: Monitor the performance of the Impala server and address any resource bottlenecks or performance issues that could lead to connection timeouts. This may involve optimizing server configuration, increasing resources, or tuning Impala parameters. Thrift Protocol Configuration: Ensure that the Thrift protocol settings, including authentication mechanisms, are correctly configured and consistent between your application and the Impala server.    Review the trace level driver logs, Impalad, statestore and catalog logs at the time of issue to see if we can get some details of the issue.   Regards, Chethan YM     ... View more

@krishna2023  It seems like you're encountering issues with loading data into partitions in Impala after executing the provided steps.  Create table as select * from db2.t1 where 1=2: This step creates an empty table db1.t1 based on the schema of db2.t1 without any data. Ensure that the table schema matches between db1.t1 and db2.t1. Alter table set location: After creating the empty table, you're altering its location to a new path. Make sure that the specified path exists and has the necessary permissions for Impala to read and write data. Add partition for every day: Adding partitions should involve specifying the loading date for each partition and its corresponding HDFS directory path. Double-check that the HDFS directory paths specified in each partition definition are correct and accessible by Impala. Refresh table: The REFRESH command updates the metadata of the table to reflect changes made in the underlying data directory. After adding partitions, running REFRESH is necessary to inform Impala about the new partitions. Make sure to execute this command after adding partitions. Compute stats: The COMPUTE STATS command gathers statistics about the table, which helps Impala optimize query execution. While this command is not directly related to loading data into partitions, it's good practice to run it after making significant changes to the table. To further troubleshoot the issue, consider the following additional steps: Check Impala logs for any error messages or warnings that might indicate issues with loading data or adding partitions. Verify that the data files corresponding to the partitions are present in the specified HDFS directory paths. Ensure that the partitioning column (loading_date) values in the data files match the partition definitions specified in the ALTER TABLE statements.   Regards, Chethan YM ... View more

@vhp1360  Given the behavior you've observed with different batch sizes and column counts, it's possible that there is a memory or resource constraint causing the error when dealing with a large number of columns and rows. Here are some potential causes and troubleshooting steps to consider: Memory Constraints: Loading a dataset with 200 columns and 20 million rows can require a significant amount of memory, especially if each column contains large amounts of data. Ensure that the system running IBM DataStage has sufficient memory allocated to handle the processing requirements. Configuration Limits: Check if there are any configuration limits or restrictions in the IBM DataStage or Hive connector settings that might be causing the issue. For example, there could be a maximum allowed stack size or buffer size that is being exceeded when processing large datasets. Resource Utilization: Monitor the resource utilization (CPU, memory, disk I/O) on the system running IBM DataStage during the data loading process. High resource utilization or contention could indicate a bottleneck that is causing the error. Optimization Techniques: Consider optimizing the data loading process by adjusting parameters such as batch size, record count, or buffer size. Experiment with different configurations to find the optimal settings that can handle the larger dataset without encountering errors. Data Format Issues: Verify that the data format and schema of the dataset are consistent and compatible with the Hive table schema. Data inconsistencies or mismatches could potentially cause errors during the loading process.   Regards, Chethan YM ... View more

@muneeralnajdi  The issue you're encountering with the Hive external table, where it fails when using COUNT(*) or WHERE clauses, seems to be related to the custom input format not being properly utilized during query execution. This can lead to errors when Hive attempts to read the files using the default input format. Ensure Custom Input Format is Used: Verify that the custom input format (CustomAvroContainerInputFormat) is correctly configured and loaded in the Hive environment. Confirm that the JAR containing the custom input format class is added to the Hive session or cluster, and that there are no errors or warnings during the JAR loading process. Check Table Properties: Ensure that the custom input format class is correctly specified in the table properties (INPUTFORMAT), and that there are no typos or syntax errors in the table definition. Test with Basic Queries: Start with basic queries (SELECT *) to ensure that the custom input format is properly utilized and data can be read from the Avro files(I think it is working). If basic queries work fine but more complex queries fail, it may indicate issues with the input format's compatibility with certain Hive operations. Consider Alternative Approaches: If troubleshooting the custom input format does not resolve the issue, consider alternative approaches for filtering the files based on their format. For example, you could pre-process the data to separate Avro and JSON files into different directories or partitions, or use other techniques such as external scripts or custom SerDes to handle different file formats within the same directory.   Regards, Chethan YM       ... View more

@yagoaparecidoti  To estimate how much data the new DataNode 7 will receive after performing a rebalance in HDFS, we need to consider the current data distribution across the existing DataNodes and how the rebalancing algorithm will redistribute the data. Even Data Distribution: The rebalancing process aims to achieve an even distribution of data blocks across all DataNodes in the cluster. This means that HDFS will attempt to redistribute the existing data blocks among all DataNodes, including the new DataNode 7, to balance storage utilization. Redistribution Strategy: HDFS will analyze the current data distribution and determine an optimal redistribution strategy to achieve balance. This strategy may involve moving some data blocks from existing DataNodes to DataNode 7, but it's unlikely that all data from all existing DataNodes will be moved to the new DataNode. Optimization and Efficiency: HDFS aims to minimize data movement and optimize the rebalancing process to achieve a balanced state with minimal disruption. The rebalancing algorithm considers factors such as network bandwidth, disk I/O, and cluster performance to determine the most efficient redistribution strategy. Given these considerations, it's difficult to provide an exact estimate of how much data DataNode 7 will receive after the rebalance without knowing the specific details of the cluster configuration and the rebalancing algorithm used. However, DataNode 7 will likely receive a portion of the existing data blocks from the other DataNodes to help achieve a balanced distribution of data across the cluster. Regards, Chethan YM ... View more

@BrianChan  Cluster Average Utilization Calculation: The cluster average utilization during HDFS rebalancing is typically calculated based on the configured capacity of the cluster. The configured capacity represents the total storage capacity allocated to the HDFS cluster as defined in the cluster's configuration settings. Individual Utilization Calculation: Individual utilization during HDFS rebalancing is usually calculated based on the sum of DFS used and remaining space for each datanode. This calculation provides an accurate representation of how much storage is currently being utilized on each datanode and how much space is available for additional data storage. Difference in File Moving Size: The difference between the initially reported file moving size and the actual file moving size in the balancer log can occur due to various factors. These may include changes in data distribution across datanodes during the rebalancing process, optimizations performed by the balancer algorithm, or adjustments made based on real-time cluster conditions and performance considerations. Exceeding DataNode Balancing Bandwidth: While the datanode balancing bandwidth is configured to limit the amount of data transferred between datanodes per second during HDFS rebalancing, it's possible for the actual bandwidth consumption to exceed this limit under certain circumstances. Factors such as network congestion, variations in data transfer rates, or optimizations performed by the balancer algorithm can contribute to bandwidth consumption exceeding the configured limit.   Regards, Chethan YM ... View more

@Shivakuk When you replace a disk in an HDFS cluster, especially if it's a DataNode disk, the Hadoop system should handle data replication and rebalancing automatically. This means that once the new disk is added and the DataNode is back online, HDFS will redistribute the data across the cluster to maintain the configured replication factor. If data was wiped during or after the disk replacement process, it's critical to investigate why this occurred and take measures to prevent data loss in the future. Ensure that proper backup and recovery procedures are in place, and consider implementing data mirroring or replication to minimize the risk of data loss due to hardware failures. Regards, Chethan YM ... View more

Hi, The error message you've provided indicates a problem with the agent's ability to send heartbeats to the master. This can occur due to various reasons, such as network issues, firewall settings, or misconfigurations. Check the master server is available and reachable, check the network/firewall settings in the system, re-check the agent config files regarding the hostnames, port numbers etc... Regards, Chethan YM ... View more