relationship between Hive query and missing blocks on cluster

[#1]

I decommissioned and deleted 2 out of 3 my HDFS data nodes. Although I expected blocks to have been replicated, it had not.

I started getting under replication error on my cluster. I have started HDFS balancer now but hive queries are terribly slow.

Is there some relation between two? Is it because it has to write to three nodes when files are underreplicated?

[#2]

@Geoffrey Shelton Okot: Thanks. Is there something else that I can do to resolve this issue in Map-red? Spark isn't working all that well for me.

[#3]

@Sim kaur

For sure all was working when you had 3 datanodes !!! The default replication factor is 3 so if you delete 2 out of 3 my HDFS data nodes, that literally means you have ONLY one copy of your file.

With 6 nodes you could have a setup like this

• 2 Master node
  
• 3 Datanodes(every datanode should have a node manager default)
  
• I Edge Node (Low-end node )
  

You should have a least 3 zookeeper servers running and a client on each node! When you are not running NameNode HA you will see NN,SNN running on the same node the SNN daemon is only an NN helper for merging the edits and fsimage, it offloads the task of merging from the NN but if you plan to have High Availability then you should configure a real NameNode HA the primary and standby NameNodes MUST run on 2 different node !!

There is no better document than the HWX Multi-home-cluster but from your setup you are running CDH, I don't thisnk there is a big difference.

Please set add the custom hdfs-site properties check the HDFS configuration parameters in CDH. .

dfs.client.use.datanode.hostname=true

In your previous post above can you explain me the

• 3
• 4

And please revert

[#4]

Thanks. Let me check with

1. dfs.client.use.datanode.hostname=true

and I will confirm.

Also, adding 3 zookeeper servers as well.

Thanks for clarifying that is required to take SNN on a different node when only Namenode is working in HA mode.

But what I am not able to understand is if the error:

Job Submission failed with exception 'java.lang.RuntimeException(Error caching map.xml: java.io.InterruptedIOException: Interruped while waiting for IO on channel java.nio.channels.SocketChannel[connected local=/172.31.4.192:42632 remote=/172.31.4.192:50010].

and

ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
INFO  : Completed executing command(queryId=hive_20180504194545_4cbaf9cb-9f83-410f-9804-1618f1ee10d1); Time taken: 0.156 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask (state=08S01,code=1)
Closing: 0: jdbc:hive2://ip-172-31-4-192.ap-south-1.compute.internal:10000/default
Intercepting System.exit(2)

is related to having under-replicated data. My data has been replicated across 3 nodes now and I am still running into this error and so was I before decommissioning my data nodes that have run out of space and commissioning the new ones. @Geoffrey Shelton Okot

3. Gateway server has following roles:

7 Role(s) HDFS Gateway HDFS HttpFS Hive Gateway YARN (MR2 Included) NodeManager Sentry Gateway Spark Gateway

9.28s ago

4. This server only has

• YARN (MR2 Included) NodeManager

Added it since our vCPU's are upto full usage sometimes.

I would also like to clarify if it is okay to have HDFS Data Node on a server without Node Manager on it

and

if it is okay to have node manager without having HDFS on same node.

[#5]

@Sim kaur

I would suggest you install at least the node managers on the servers where datanodes are running this way node managers can find the data locally. Datanodes are part of HDFS and node managers are part of Yarn.

Datanodes are used to store data on HDFS whereas Nodemanagers are used to start a container on Yarn. There is no strict rule that datanodes and node managers have to be on the same host. If you have nodemanagers on all nodes, in this case, the containers running on hosts where datanodes aren't installed will still run the application by copying data from datanodes. That could be the issue of timeouts you are experiencing.

[#6]

@Sim kaur

Any updates?

[#7]

@Geoffrey Shelton Okot : I have made the changes

i.e

1. dfs.client.use.datanode.hostname=true

2. Removed yarn from nodes that did not have HDFS

3. Although still have a node that only has HDFS (Is there a way that Hadoop can prioritize execution of jobs on worker nodes that have both of these services started on them?)

The error happens randomly throughout the day and in morning hours when most of our hive reports run. I am in IST time zone. Please give me until tomorrow morning IST to confirm if this has completely resolved the issue 🙂

Also, I would like to know what does exactly happen if zookeeper data is lost? I tried installing zookeeper server instance on other nodes but the new instances did not start and kept returning supervisord failed to start.

[#8]

@Sim kaur

Okay will wait for your feedback!!

How many zookeepers do you have running now? Anything less than 3 is NOT good! check zookeeper split brain documentation

For the failed installation run

ps aux |grep supervisor 

if there is an existing supervisor process PID kill it if one exists and then start the agent to make sure the agent is using the right supervisord.conf /var/run/cloudera-scm-agent/supervisor/supervisor.conf

[#9]

@Geoffrey Shelton Okot I am still getting the error

Also, from the metadata, I see that it is executing on the node that has both Nodemanager and HDFS.

Is there somewhere I can trace down reason for this error?

Note: /var/log/hive/hadoop-cmf-CD-HIVE-XCVXskZf-HIVESERVER2-ip-172-31-4-192.ap-south-1.compute.internal.lo... This path does not exist on any of my nodes.

Once I start getting this error, all my hive jobs fail until after a while and are usually fixed if I restart my cluster.

ERROR : Execution failed with exit status: 1
ERROR : Obtaining error information
ERROR : 
Task failed!
Task ID:
  Stage-20

Logs:

ERROR : <a href="http://PUBLICDOMAINNAME:8888/filebrowser/view=/var/log/hive/hadoop-cmf-CD-HIVE-XCVXskZf-HIVESERVER2-ip-172-31-4-192.ap-south-1.compute.internal.log.out">/var/log/hive/hadoop-cmf-CD-HIVE-XCVXskZf-HIVESERVER2-ip-172-31-4-192.ap-south-1.compute.internal.log.out</a>
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
WARN  : Shutting down task : Stage-1:MAPRED
WARN  : Shutting down task : Stage-7:MAPRED
WARN  : Shutting down task : Stage-11:MAPRED
INFO  : Completed executing command(queryId=hive_20180506020606_5277a480-1de4-4a48-9469-b6a2e7d0b200); Time taken: 0.163 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask (state=08S01,code=1)
Closing: 0: jdbc:hive2://ip-172-31-4-192.ap-south-1.compute.internal:10000/default
Intercepting System.exit(2)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2]

[#10]

@Sim kaur

Just answered your YARN memory setup on another thread, you were scheduling you hive queries through oozie 🙂 Please do use the excel file to help you setup correctly your Vcores and Memory.

Please accept and close the previous threads where part of your issues was resolved because the Memory issue is different from the zookeeper setup or Cluster setup question which I had already answered.

Try to keep your threads component specific its easier to resolve and open a new thread 🙂

[#11]

@Sim kaur

Have you tried switching to beeline? and executed your queries?

set hive.execution.engine=mr;

Take note Hive CLI should be deprecated as the Hive community has long recommended using the Beeline plus HiveServer2 configuration