Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

teragen failing with replication to 0 nodes

Highlighted

teragen failing with replication to 0 nodes

New Contributor

First time running teragen/sort/validate test went ok on 100 Gig test size.

After re-running test, teragen fails and log says following: I see onyl 26 maps are processed of the total of 92.

Any ideas...stack below.

2017-07-19 14:01:12,719 INFO [Thread-145] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STOPPED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /mr-history/tmp/user1/job_1499796473254_0009.summary_tmp could only be replicated to 0 nodes instead of minReplication (=1). There are 4 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1620) at

...

org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2211) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2207) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2205) org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /mr-history/tmp/user1/job_1499796473254_0009.summary_tmp could only be replicated to 0 nodes instead of minReplication (=1). There are 4 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1620)

Other info:

hdfs dfsadmin -report

Configured Capacity: 1074290753536 (1000.51 GB)

Present Capacity: 601786240099 (560.46 GB)

DFS Remaining: 172010277124 (160.20 GB)

DFS Used: 429775962975 (400.26 GB)

DFS Used%: 71.42%

Under replicated blocks: 307

Blocks with corrupt replicas: 0

Missing blocks: 0

Missing blocks (with replication factor 1): 0

Don't have an account?
Coming from Hortonworks? Activate your account here