About HanzalaShaikh

HanzalaShaikh · ‎07-12-2021

Hello Experts, We have identified that 2 records have been duplicated in our hive tables. We have taken the backup of tables in case if we need to rollback. But now when we run insert overwrite command (e.g. insert overwrite table demo select distinct * from demo;)on smallest table with raw volume of "570 GB" but we are getting the following error. INFO : 2021-07-11 15:33:47,756 Stage-0_0: 122/122 Finished Stage-1_0: 70(+380,-64)/978 INFO : state = STARTED INFO : state = FAILED ERROR : Status: Failed ERROR : FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask DEBUG : Shutting down query insert overwrite table raw_switch.partitiontable select distinct * from raw_switch.partitiontable INFO : Completed executing command(queryId=hive_19660743242525_d9c3a756-452f-472c-a92e-2b966c37d0ce); Time taken: 4078.407 seconds DEBUG : Shutting down query insert overwrite table raw_switch.partitiontable select distinct * from raw_switch.partitiontable Error: Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask (state=08S01,code=3) Please find below hive server2 logs:- 2021-07-11 15:33:49,834 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: [HiveServer2-Background-Pool: Thread-29919]: Call: delete took 30ms 2021-07-11 15:33:49,834 ERROR org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-29919]: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask The default parameters of hive are as follows:- hive.execution.engine=spark; spark.executor.memory=12g; spark.executor.cores=4; hive.optimize.sort.dynamic.partition=true; hive.exec.dynamic.partition.mode=strict; Kindly suggest how to resolve this issue. Do we need to change any of the above default parameters or some other parameters which we have missed. Hope, we are running the correct query of insert overwrite to remove duplicate records.

HanzalaShaikh · ‎12-14-2020

@Tim Armstrong Thanks for your quick reply. I did ran the compute stats command to know the query performance however I got the error below: compute stats <table name>; ERROR: AnalysisException: COMPUTE STATS not supported for view: I also got the details from the Query details after running the query. I had to cancel it as it takes a lot of resources and goes in executing state for longer time. Query Info Query ID: 130b304cc42b5010:19ef656c00000000 User: hadmin@COMPS-NVIRGINIA.LOCAL Database: prod_cdb Coordinator: usnprod4.n-virginia.dc Query Type: QUERY Query State: EXCEPTION Start Time: Dec 15, 2020 7:17:34 AM End Time: Dec 15, 2020 7:18:59 AM Duration: 1m, 24s Rows Produced: 0 Admission Result: Admitted immediately Admission Wait Time: 0ms Aggregate Peak Memory Usage: 13.2 GiB Bytes Streamed: 15.6 GiB Client Fetch Wait Time: 0ms Client Fetch Wait Time Percentage: 0 Connected User: hadmin@COMPS-NVIRGINIA.LOCAL Estimated per Node Peak Memory: 12.8 GiB File Formats: PARQUET/NONE,PARQUET/SNAPPY HDFS Average Scan Range: 90.5 KiB HDFS Bytes Read: 24.7 GiB HDFS Bytes Read From Cache: 0 B HDFS Bytes Read From Cache Percentage: 0 HDFS Local Bytes Read: 23.7 GiB HDFS Local Bytes Read Percentage: 96 HDFS Remote Bytes Read: 1.1 GiB HDFS Remote Bytes Read Percentage: 4 HDFS Scanner Average Read Throughput: 155.9 MiB/s HDFS Short Circuit Bytes Read: 23.7 GiB HDFS Short Circuit Bytes Read Percentage: 96 Impala Version: impalad version 2.11.0-cdh5.14.4 RELEASE Memory Accrual: 158,669,819,348 byte seconds Memory Spilled: 1.0 GiB Network Address: 10.206.100.226:42238 Node with Peak Memory Usage: usnprod3.n-virginia.dc:22000 Out of Memory: false Per Node Peak Memory Usage: 5.2 GiB Planning Wait Time: 6.69s Planning Wait Time Percentage: 8 Pool: root.default Query Status: Cancelled Session ID: 9b32167b6eef775e:293bgd8a4350f200 Session Type: BEESWAX Statistics Corrupt: false Statistics Missing: true Threads: CPU Time: 2.9m Threads: CPU Time Percentage: 2 Threads: Network Receive Wait Time: 11.4m Threads: Network Receive Wait Time Percentage: 6 Threads: Network Send Wait Time: 45.4m Threads: Network Send Wait Time Percentage: 24 Threads: Storage Wait Time: 2.2h Threads: Storage Wait Time Percentage: 69 Threads: Total Time: 3.2h Please suggest is it due to memory issue that the query is actually getting time out and there is no output from the impala shell, however it shows query executing in cloudera. Please note: The Default Query Memory Limit in Impala = 6 GB Max Memory = 270 GB A quick reply would be highly appreciated.

HanzalaShaikh · ‎12-14-2020

Hello, One of our end user tried to run an impala query, which is actually a view with a long query statement. We have noticed that the query time out's and we have found below logs in query details in cloudera. WARNING: The following tables are missing relevant table and/or column statistics. prod_cdb.mig_pdg_common, cdb.mig.accepted_common, prod_cdb.mig_post_correlation_common, prod_cdb.output_pre_correlation_common, prod_cdb.retained_for_correlation_common What could be the reason for this, does the view need to be validated or does impala does not support such view. what is the other way to know the root cause. Any suggestions, would be highly appreciated.

HanzalaShaikh · ‎11-07-2020

Hi Ateka, Can you share what error you are getting. while submitting the command to better understand the issue.

HanzalaShaikh · ‎11-01-2020

Hi, I am trying to see a summary of a query's progress that updates in real-time, Hence I ran 'set LIVE_PROGRESS=1;'. But I am getting the message stating as below: ERROR: User hadmin@HADOOP-GROUP.LOCAL is not authorized to access the runtime profile or execution summary. However, the user hadmin is the authorized user to view that particular database and table. Please suggest what could be the reason and how to fix this, Since I have taken the Kerberos ticket as well.

HanzalaShaikh · ‎10-24-2020

Hi Tushar Thanks a lot for your quick reply. The resolution you provided has worked. I am accepting it as a solution. Thanks once again.

HanzalaShaikh · ‎10-20-2020

Hello, We are using a beeline or impala-shell to extract data from hive tables as per requirement from the end-user. However, the request for data extraction is for high records for more than 1000 or sometimes more than 3000. It is very tedious to extract using the select query and dump to excel sheet. Is there any alternative way to take the output in a CSV file?. Like the output of select query moves to CSV file. Please suggest.

HanzalaShaikh · ‎10-16-2020

Hi, To know what should be the heap memory of the name node, please provide the storage capacity of each node and the replication factor. Based on this we can calculate the heap memory. the default block size is recommended as a default block size for the large and small clusters.

HanzalaShaikh · ‎10-14-2020

Hi, I want to enable an on-demand metadata feature for impala as it gives a lot of improvement. one of the features is that the cached metadata gets evicted automatically under memory pressure. I want to know if this feature can be enabled in Cloudera CDH 5.14.4 version. Has anyone implemented this on prod on CDH 5.14.4 version?

HanzalaShaikh · ‎10-14-2020

Hi Tim, Your suggestion was very helpful. I have a good understanding now. I am accepting as a solution. I just have one more thing to ask, to fix the issue of the query utilizing the resources it is better to increase the Impala Daemon Memory Limit (mem_limit). what do you suggest?

HanzalaShaikh · ‎10-03-2020

Hi Tim, Can you explain in more detail how can I do this? E.g. you could set up memory-based admission control with a min memory limit of 2GB and a max memory limit of 20GB to prevent anyone query from taking up all the memory on a node.

HanzalaShaikh · ‎10-03-2020

Hi Tim, Thanks for your reply I can only see two parameters for the memory limit. one is Single Pool Mem Limit (default_pool_mem_limit) = -1 B the second one is Impala Daemon memory limit (mem_limt) = 60 GB So how do I set now min memory limit and max memory limit.

HanzalaShaikh · ‎10-02-2020

After running Impala query select distinct(partition_date) parts from mddb_servt; I am getting below error: Execution time 10 seconds 2) java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000, errorMessage:ExecQueryFInstances rpc query_id=937d334667doe010:4967d4b900000111 failed: Failed to get minimum memory reservation of 68.00 MB on daemon ec2-3-128-13.us-east-2:22000 for query 937d334667doe010:4967d4b900000111 because it would exceed an applicable memory limit. Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error. Memory usage: Process: Limit=60.00 GB Total=50.68 GB Peak=50.70 GB Buffer Pool: Free Buffers: Total=0 Buffer Pool: Clean Pages: Total=2.31 GB Buffer Pool: Unused Reservation: Total=-2.29 GB Free Disk IO Buffers: Total=1.10 GB Peak=1.15 GB RequestPool=root.default: Total=48.28 GB Peak=48.37 GB Query(78befceb1eef47:d33db5f200030000): Reservation=47.49 GB ReservationLimit=48.00 GB OtherMemory=293.93 MB Total=47.78 GB Peak=47.81 GB Query(e12345ed0a094d14:f4616fb90030000): Reservation=238.00 MB ReservationLimit=48.00 GB OtherMemory=4.27 MB Total=242.27 MB Peak=303.22 MB Query(be7896564af6f2c:1e675bb00000000): Reservation=272.00 MB ReservationLimit=48.00 GB OtherMemory=4.23 MB Total=276.23 MB Peak=314.22 MB Query(914d001522ce0e10:264bd4b900000000): Reservation=0 ReservationLimit=48.00 GB OtherMemory=0 Total=0 Peak=0 RequestPool=root.anp: Total=0 Peak=536.50 MB Untracked Memory: Total=1.28 GB Please note: Impala Daemon Memory Limit mem_limit = 60 GiB Please let me know what could be the reason.

HanzalaShaikh · ‎08-15-2020

@GangWar thanks for your quick reply. As per your suggestion I have checked the logs on both the locations on namenode and datanodes. On one of the data nodes I have checked the logs in /var/run/cloudera-scm-agent/process/28-hdfs-DATANODE/logs, I have found below results in the log when searched by the keyword Error: ++ replace_pid -Xms521142272 -Xmx521142272 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/hdfs_hdfs-DATANODE-111b6db5e742dbffe061f0c1d6bc8878_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh ++ sed 's#{{PID}}#5409#g' However, I don't see any error message in /var/log/hadoop-hdfs, please also suggest which log file to check to debug. audit hadoop-cmf-hdfs-NAMENODE-namenode1.us-east1-b.c.coherent-elf-271314.internal.log.out hdfs-audit.log SecurityAuth-hdfs.audit stacks

HanzalaShaikh · ‎08-15-2020

I am stuck in the middle of Cluster setup after installing Cloudera manager. The first 3 steps have completed however when it comes to starting HDFS it has successfully formatted the name directories of the current NameNode but got stuck in Start HDFS, please find below the error. Cluster Setup First Run Command Status Running Aug 15, 9:16:45 AM There was an error when communicating with the server. See the log file for more information. Completed 3 of 8 step(s). Show All Steps Show Only Failed Steps Show Running Steps Ensuring that the expected software releases are installed on hosts. Aug 15, 9:16:45 AM 90ms Deploying Client Configuration Cluster 1 Aug 15, 9:16:45 AM 16.13s Start Cloudera Management Service, ZooKeeper Aug 15, 9:17:01 AM 27.87s Start HDFS 0/1 steps completed. Aug 15, 9:17:29 AM Execute 3 steps in sequence Waiting for command (Start (77)) to finish Aug 15, 9:17:29 AM Formatting the name directories of the current NameNode. If the name directories are not empty, this is expected to fail. NameNode (namenode1) Aug 15, 9:17:29 AM 14.86s Start HDFS There was an error when communicating with the server. See the log file for more information. I am unable to check the logs as the cluster is not fully setup, please suggest what could be the reason and how to fix these. I am installing with version 5.16.2

HanzalaShaikh · ‎08-11-2020

Thanks a lot, @GangWar . You are absolutely correct I was using OpenJDK instead of Oracle JDK. I thought there is a bug in this but I didn't have any clue how to fix this but after making changes as per your suggestion it worked. Thanks a lot. I am accepting it as a solution.

HanzalaShaikh · ‎08-06-2020

I can't open /run/cloudera-scm-agent/process/256-yarn-NODEMANAGER/container-executor.cfg: Permission denied. + perl -pi -e 's#{{CGROUP_GROUP_CPU}}##g' /run/cloudera-scm-agent/process/256-yarn-NODEMANAGER/yarn-site.xml I am getting this error after enabling Kerberos in CDH cluster, HDFS and yarn are not able to start. After checking the yarn Node Manager logs I see the below error. yarn nodemanager logs: : org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.authorize.AuthorizationException: : User: cloudera@CLUSTERIE.LOCAL is not allowed to impersonate yarn/ip-10-0-xxxxx@xyz.com Any suggestion why am I getting this error, when I disable the Kerberos everything works well. Please assist as the severity of this is very high.

HanzalaShaikh · ‎07-30-2020

Thanks a lot for your quick reply and for providing a detailed explanation, really appreciated. I was using a 5.14.1 version of Cloudera. Yes, there is a bug in the older version I realized soon after that, but your explanation has made me understand the reason behind the error.

HanzalaShaikh · ‎07-27-2020

I am setting up a Pre prod cluster from Path B. I ran the command "/usr/share/cmf/schema/scm_prepare_database.sh mysql -h mysqldatabase scm temp password" to check the database status. I am getting the error of log4j however the SCM database is configured correctly. Please find the below error and suggest any workaround for this error. [root@clouderamanage cloudera-scm-server]# /usr/share/cmf/schema/scm_prepare_database.sh mysql -h webserver scmmysqldatabase password JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/java/jdk1.7.0_67-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. log4j: ERROR Could not find value for key log4j.appender.A log4j: ERROR Could not instantiate appender named "A". [2020-07-27 10:20:19,893] INFO 0[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor.testDbConnection(DbCommandExecutor.java) - Successfully connected to database. All done, your SCM database is configured correctly!

HanzalaShaikh · ‎05-03-2020

Hi everyone, My name is Hanzala Shaikh. I have 5+ years of experience in core IT which includes projects on Big Data Hadoop platform specialization on Cloudera EDH in AWS cloud. Worked on Multi-Cloud Environment on AWS/GCP. My expertise is in Hadoop Administration and managing production-grade cluster on AWS Cloud. I have worked on multiple projects in the healthcare domain on a secured cluster. I have good exposure to Cloudera and Hortonworks distribution as I have worked on both platforms. Now, I am here to contribute my knowledge to the community in the best possible way as I can and also to gain some more knowledge and get my queries answered.

Online	Offline
Last Visited	‎07-19-2021 10:02 PM

Date Registered	‎04-25-2020 12:26 AM
Last Visited	‎07-19-2021 10:02 PM
Total Messages Posted	19

Error while running Insert overwrite query on hive...

Re: Impala query time out's

Impala query time out's

Re: How to check the size of Hive view table?

Not able to access the runtime profile or executi...

Re: Convert select query output to CSV from beelin...

Convert select query output to CSV from beeline or...

Re: heap size of namenode/ blocksize

Impala - On-demand metadata

Re: Impala query failed

Re: Impala query failed

Re: Impala query failed

Impala query failed

Re: Error while cluster setup in installation.

Error while cluster setup in installation.

Re: Error after enabling kerberos in CDH cluster

Error after enabling kerberos in CDH cluster

Re: log4j:ERROR Could not find value for key log4j...

log4j:ERROR Could not find value for key log4j.app...

Re: INTRODUCE YOURSELF: Please Reply here to tell ...