Reply
Highlighted
Explorer
Posts: 8
Registered: ‎09-14-2015

Hive job failed: Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.

Hi Guys,

 

Enabled CDH5.4.8 cluster with Cloudera Security:

1. Kerberos

2. CM Services & CDH Services TLS/SSL Encryption

3. Sentry

4. CDH Data Encryption at REST and in Transit

 

Configured privileges to Hive Server and databases. While running job in Hive JDBC Map job successfully completed and Reduce job failed with the following error

 

Task with the most failures(4): 
-----
Task ID:
  task_1458300931532_0001_r_000000

URL:
  http://enggbds10.solixindia.com:8088/taskdetails.jsp?jobid=job_1458300931532_0001&tipid=task_1458300931532_0001_r_000000
-----
Diagnostic Messages for this Task:
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#10
	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1707)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
	at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:366)
	at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:288)
	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:354)
	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)

 

All the jobs failed at Reduce phase only and it is not shuffle the map out data to reducer. I have checked my network and DNS and ports, all are working good. 

 

--

Regards

Ram G

 

Announcements