Issues with ImportTsv - Job Fails with Exit Code 1

[#1]

I am currently facing an issue while using the ImportTsv command to load data into an HBase table. I am running the command and the job is failing with a non-zero exit code 1. 

./hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns="HBASE_ROW_KEY,cf:name,cf:age,cf:gender" -Dimporttsv.separator="\t" my_1 /hbase/test2.txt

ERROR LOG:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/super/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/super/hbase/lib/client-facing-thirdparty/slf4j-reload4j-1.7.33.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
2024-09-02 09:48:21,871 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-09-02 09:48:22,149 INFO [ReadOnlyZKClient-localhost:2181@0x68267da0] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.8.4-9316c2a7a97e1666d8f4593f34dd6fc36ecc436c, built on 2024-02-12 22:16 UTC
2024-09-02 09:48:22,149 INFO [ReadOnlyZKClient-localhost:2181@0x68267da0] zookeeper.ZooKeeper: Client environment:host.name=dc1-apache-hbase.____________
...
2024-09-02 09:48:32,995 INFO [main] mapreduce.Job: 8]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Hadoop version : 3.4.0

Hbase version : 2.4.18

Resource Manager Logs:

[2024-09-02 09:48:32.499]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

For more detailed output, check the application tracking page: http://______________________:8088/cluster/app/application_1725184331906_0005 Then click on links to logs of each attempt.
. Failing the application.
2024-09-02 09:48:32,505 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1725184331906_0005 State change from FINAL_SAVING to FAILED on event = APP_UPDATE_SAVED
2024-09-02 09:48:32,506 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue: Application removed - appId: application_1725184331906_0005 user: super leaf-queue of parent: root #applications: 0
2024-09-02 09:48:32,506 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1725184331906_0005,name=importtsv_my_1,user=super,queue=root.default,state=FAILED,trackingUrl=http://_________________________:8088/cluster/app/application_1725184331906_0005,appMasterHost=dc1-a...<memory:0\, vCores:0>,applicationType=MAPREDUCE,resourceSeconds=10734 MB-seconds\, 5 vcore-seconds,preemptedResourceSeconds=0 MB-seconds\, 0 vcore-seconds,applicationTags=,applicationNodeLabel=,diagnostics=Application application_1725184331906_0005 failed 2 times due to AM Container for appattempt_1725184331906_0005_000002 exited with exitCode: 1\nFailing this attempt.Diagnostics: [2024-09-02 09:48:32.496]Exception from container-launch.\nContainer id: container_1725184331906_0005_02_000001\nExit code: 1\n\n[2024-09-02 09:48:32.498]Container exited with a non-zero exit code 1. Error file: prelaunch.err.\nLast 4096 bytes of prelaunch.err :\nLast 4096 bytes of stderr :\nlog4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).\nlog4j:WARN Please initialize the log4j system properly.\nlog4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.\n\n\n[2024-09-02 09:48:32.499]Container exited with a non-zero exit code 1. Error file: prelaunch.err.\nLast 4096 bytes of prelaunch.err :\nLast 4096 bytes of stderr :\nlog4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).\nlog4j:WARN Please initialize the log4j system properly.\nlog4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.\n\n\nFor more detailed output\, check the application tracking page: http://_________________________________________________________--/cluster/app/application_172518433... Then click on links to logs of each attempt.\n. Failing the application.,totalAllocatedContainers=2

[#2]

@Amandi Can you try to run the command from a Region server host. The connection is going to a localhost on port 2181. So if Zookeeper is not running on the host from where you are running the command, it will fail.

[#3]

@rki_Thank you for the response. Could you please suggest a solution that doesn’t involve using the Region Server host?

Here are the details for port 2181 :
[@dc1-apache-hbase ~]$ ss -tuln | grep 2181
tcp LISTEN 0 50 *:2181 *:*
[@dc1-apache-hbase ~]$ lsof -i :2181
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 388228 _____ 558u IPv6 10088613 0t0 TCP *:eforward (LISTEN)
java 388228 _____ 562u IPv6 10092015 0t0 TCP localhost:eforward->localhost:56476 (ESTABLISHED)
java 388228 _____ 565u IPv6 10092023 0t0 TCP localhost:eforward->localhost:56492 (ESTABLISHED)
java 388372 _____ 575u IPv6 10092011 0t0 TCP localhost:56476->localhost:eforward (ESTABLISHED)
java 388566 _____ 575u IPv6 10088671 0t0 TCP localhost:56492->localhost:eforward (ESTABLISHED)

[#4]

can you confirm if Zookeeper service is running on this host? Apart from the log4J errors, Do you see any others in the Application log?

[#5]

The RegionServer host is localhost, and when I run the command jps -l | grep QuorumPeerMain, there is no output. 

Using the ssh command to connect to the RegionServer host and running the importtsv command still results in the same error as before.

jps:

359952 SecondaryNameNode
360176 ResourceManager
361026 DataNode
400017 HQuorumPeer
400353 HRegionServer
400159 HMaster
406172 Jps
401800 ThriftServer
359578 NameNode
360298 NodeManager

[#6]

If you are still not able to run the command successfully from the Region server host, then its most likely an issue with YARN. Can you check if there are any other WARNs or ERRORs in the app log apart from log4j warnings.

[#7]

rki here I am attaching the log of the resource manager:

For more detailed output, check the application tracking page: http://dc1-apache/cluster/app/application_1725184331906_0014 Then click on links to logs of each attempt.
. Failing the application.
2024-09-03 14:36:38,511 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application Attempt appattempt_1725184331906_0014_000002 is done. finalState=FAILED
2024-09-03 14:36:38,511 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1725184331906_0014 requests cleared
2024-09-03 14:36:38,511 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractLeafQueue: Application removed - appId: application_1725184331906_0014 user: super queue: root.default #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0
2024-09-03 14:36:38,512 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1725184331906_0014 State change from FINAL_SAVING to FAILED on event = APP_UPDATE_SAVED
2024-09-03 14:36:38,515 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue: Application removed - appId: application_1725184331906_0014 user: super leaf-queue of parent: root #applications: 0
2024-09-03 14:36:38,515 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1725184331906_0014,name=importtsv_my_1,user=super,queue=root.default,state=FAILED,trackingUrl=http://dc1/cluster/app/application_1725184331906_0014,appMasterHost=dc1-apache-hbase.mobitel.lk,subm..., vCores:0>,applicationType=MAPREDUCE,resourceSeconds=10919 MB-seconds\, 5 vcore-seconds,preemptedResourceSeconds=0 MB-seconds\, 0 vcore-seconds,applicationTags=,applicationNodeLabel=,diagnostics=Application application_1725184331906_0014 failed 2 times due to AM Container for appattempt_1725184331906_0014_000002 exited with  exitCode: 1\nFailing this attempt.Diagnostics: [2024-09-03 14:36:38.499]Exception from container-launch.\nContainer id: container_1725184331906_0014_02_000001\nExit code: 1\n\n[2024-09-03 14:36:38.503]Container exited with a non-zero exit code 1. Error file: prelaunch.err.\nLast 4096 bytes of prelaunch.err :\nLast 4096 bytes of stderr :\nlog4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).\nlog4j:WARN Please initialize the log4j system properly.\nlog4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.\n\n\n[2024-09-03 14:36:38.504]Container exited with a non-zero exit code 1. Error file: prelaunch.err.\nLast 4096 bytes of prelaunch.err :\nLast 4096 bytes of stderr :\nlog4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).\nlog4j:WARN Please initialize the log4j system properly.\nlog4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.\n\n\nFor more detailed output\, check the application tracking page: http://dc1/cluster/app/application_1725184331906_0014 Then click on links to logs of each attempt.\n. Failing the application.,totalAllocatedContainers=2

[#8]

See if you can raise a support ticket with Cloudera. The app log needs a detailed review to know what is causing the container to get fail.