Member since
09-01-2024
7
Posts
6
Kudos Received
0
Solutions
09-11-2024
08:02 PM
1 Kudo
hi @shubham_sharma , I ran a simple job as you requested, and it seems to have run without any issues. For your reference, I am attaching the output as well. Output: [super@dc1-apache-hbase mapreduce]$ /home/super/hadoop/bin/hadoop jar /home/super/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.0.jar pi 10 100 Number of Maps = 10 Samples per Map = 100 2024-09-12 08:25:59,664 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Wrote input for Map #0 Wrote input for Map #1 Wrote input for Map #2 Wrote input for Map #3 Wrote input for Map #4 Wrote input for Map #5 Wrote input for Map #6 Wrote input for Map #7 Wrote input for Map #8 Wrote input for Map #9 Starting Job 2024-09-12 08:26:01,082 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032 2024-09-12 08:26:01,580 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/super/.staging/job_1725184331906_0019 2024-09-12 08:26:01,730 INFO input.FileInputFormat: Total input files to process : 10 2024-09-12 08:26:01,775 INFO mapreduce.JobSubmitter: number of splits:10 2024-09-12 08:26:01,931 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1725184331906_0019 2024-09-12 08:26:01,931 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2024-09-12 08:26:02,186 INFO conf.Configuration: resource-types.xml not found 2024-09-12 08:26:02,187 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2024-09-12 08:26:02,284 INFO impl.YarnClientImpl: Submitted application application_1725184331906_0019 2024-09-12 08:26:02,317 INFO mapreduce.Job: The url to track the job: http://dc1-apache-hbase.mobitel.lk:8088/proxy/application_1725184331906_0019/ 2024-09-12 08:26:02,318 INFO mapreduce.Job: Running job: job_1725184331906_0019 2024-09-12 08:26:10,443 INFO mapreduce.Job: Job job_1725184331906_0019 running in uber mode : false 2024-09-12 08:26:10,445 INFO mapreduce.Job: map 0% reduce 0% 2024-09-12 08:26:20,637 INFO mapreduce.Job: map 60% reduce 0% 2024-09-12 08:26:27,696 INFO mapreduce.Job: map 100% reduce 0% 2024-09-12 08:26:29,713 INFO mapreduce.Job: map 100% reduce 100% 2024-09-12 08:26:31,745 INFO mapreduce.Job: Job job_1725184331906_0019 completed successfully 2024-09-12 08:26:31,912 INFO mapreduce.Job: Counters: 54 File System Counters FILE: Number of bytes read=67 FILE: Number of bytes written=3407061 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=2650 HDFS: Number of bytes written=215 HDFS: Number of read operations=45 HDFS: Number of large read operations=0 HDFS: Number of write operations=3 HDFS: Number of bytes read erasure-coded=0 Job Counters Launched map tasks=10 Launched reduce tasks=1 Data-local map tasks=10 Total time spent by all maps in occupied slots (ms)=68883 Total time spent by all reduces in occupied slots (ms)=6614 Total time spent by all map tasks (ms)=68883 Total time spent by all reduce tasks (ms)=6614 Total vcore-milliseconds taken by all map tasks=68883 Total vcore-milliseconds taken by all reduce tasks=6614 Total megabyte-milliseconds taken by all map tasks=70536192 Total megabyte-milliseconds taken by all reduce tasks=6772736 Map-Reduce Framework Map input records=10 Map output records=20 Map output bytes=180 Map output materialized bytes=250 Input split bytes=1470 Combine input records=0 Combine output records=0 Reduce input groups=2 Reduce shuffle bytes=250 Reduce input records=20 Reduce output records=0 Spilled Records=40 Shuffled Maps =10 Failed Shuffles=0 Merged Map outputs=10 GC time elapsed (ms)=1565 CPU time spent (ms)=6180 Physical memory (bytes) snapshot=3867213824 Virtual memory (bytes) snapshot=28280057856 Total committed heap usage (bytes)=3555196928 Peak Map Physical memory (bytes)=369606656 Peak Map Virtual memory (bytes)=2578915328 Peak Reduce Physical memory (bytes)=284368896 Peak Reduce Virtual memory (bytes)=2575523840 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=1180 File Output Format Counters Bytes Written=97 Job Finished in 31.068 seconds Estimated value of Pi is 3.14800000000000000000
... View more
09-11-2024
03:37 AM
1 Kudo
hello shubham, I am encountering errors while running the following command: "/home/super/hbase/bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns=HBASE_ROW_KEY,Name,Age,Gender my_1 /hbase/test2.txt" The error message is: 2024-09-11 15:53:44,808 INFO [main] impl.YarnClientImpl: Submitted application application_1725184331906_0018 2024-09-11 15:53:44,847 INFO [main] mapreduce.Job: The url to track the job: http://dc1-apache-hbase.mobitel.lk:8088/proxy/application_1725184331906_0018/ 2024-09-11 15:53:44,848 INFO [main] mapreduce.Job: Running job: job_1725184331906_0018 2024-09-11 15:53:52,941 INFO [main] mapreduce.Job: Job job_1725184331906_0018 running in uber mode : false 2024-09-11 15:53:52,943 INFO [main] mapreduce.Job: map 0% reduce 0% 2024-09-11 15:53:52,952 INFO [main] mapreduce.Job: 2]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. [2024-09-11 15:53:51.942]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. For more detailed output, check the application tracking page: http://___________________________________________/cluster/app/application_1725184331906_0018 Then click on links to logs of each attempt. . Failing the application. 2024-09-11 15:53:52,967 INFO [main] mapreduce.Job: Counters: 0 [super@dc1-apache-hbase mapreduce-job]$ What are the possible solutions, and how can I fix this?
... View more
09-08-2024
10:20 PM
Dear Cloudera Community, We are currently conducting warehouse testing using Apache HBase and need to load large files into HBase tables. Could you kindly suggest any tools or specifically designed for bulk loading large datasets into HBase? Thank You!
... View more
Labels:
- Labels:
-
Apache HBase
09-03-2024
02:10 AM
1 Kudo
rki here I am attaching the log of the resource manager: For more detailed output, check the application tracking page: http://dc1-apache/cluster/app/application_1725184331906_0014 Then click on links to logs of each attempt. . Failing the application. 2024-09-03 14:36:38,511 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application Attempt appattempt_1725184331906_0014_000002 is done. finalState=FAILED 2024-09-03 14:36:38,511 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1725184331906_0014 requests cleared 2024-09-03 14:36:38,511 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractLeafQueue: Application removed - appId: application_1725184331906_0014 user: super queue: root.default #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0 2024-09-03 14:36:38,512 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1725184331906_0014 State change from FINAL_SAVING to FAILED on event = APP_UPDATE_SAVED 2024-09-03 14:36:38,515 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue: Application removed - appId: application_1725184331906_0014 user: super leaf-queue of parent: root #applications: 0 2024-09-03 14:36:38,515 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1725184331906_0014,name=importtsv_my_1,user=super,queue=root.default,state=FAILED,trackingUrl=http://dc1/cluster/app/application_1725184331906_0014,appMasterHost=dc1-apache-hbase.mobitel.lk,submitTime=1725354391682,startTime=1725354391696,launchTime=1725354392197,finishTime=1725354398511,finalStatus=FAILED,memorySeconds=10919,vcoreSeconds=5,preemptedMemorySeconds=0,preemptedVcoreSeconds=0,preemptedAMContainers=0,preemptedNonAMContainers=0,preemptedResources=<memory:0\, vCores:0>,applicationType=MAPREDUCE,resourceSeconds=10919 MB-seconds\, 5 vcore-seconds,preemptedResourceSeconds=0 MB-seconds\, 0 vcore-seconds,applicationTags=,applicationNodeLabel=,diagnostics=Application application_1725184331906_0014 failed 2 times due to AM Container for appattempt_1725184331906_0014_000002 exited with exitCode: 1\nFailing this attempt.Diagnostics: [2024-09-03 14:36:38.499]Exception from container-launch.\nContainer id: container_1725184331906_0014_02_000001\nExit code: 1\n\n[2024-09-03 14:36:38.503]Container exited with a non-zero exit code 1. Error file: prelaunch.err.\nLast 4096 bytes of prelaunch.err :\nLast 4096 bytes of stderr :\nlog4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).\nlog4j:WARN Please initialize the log4j system properly.\nlog4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.\n\n\n[2024-09-03 14:36:38.504]Container exited with a non-zero exit code 1. Error file: prelaunch.err.\nLast 4096 bytes of prelaunch.err :\nLast 4096 bytes of stderr :\nlog4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).\nlog4j:WARN Please initialize the log4j system properly.\nlog4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.\n\n\nFor more detailed output\, check the application tracking page: http://dc1/cluster/app/application_1725184331906_0014 Then click on links to logs of each attempt.\n. Failing the application.,totalAllocatedContainers=2
... View more
09-02-2024
04:07 AM
1 Kudo
The RegionServer host is localhost, and when I run the command jps -l | grep QuorumPeerMain, there is no output. Using the ssh command to connect to the RegionServer host and running the importtsv command still results in the same error as before. jps: 359952 SecondaryNameNode 360176 ResourceManager 361026 DataNode 400017 HQuorumPeer 400353 HRegionServer 400159 HMaster 406172 Jps 401800 ThriftServer 359578 NameNode 360298 NodeManager
... View more
09-02-2024
12:11 AM
1 Kudo
@rki_Thank you for the response. Could you please suggest a solution that doesn’t involve using the Region Server host? Here are the details for port 2181 : [@dc1-apache-hbase ~]$ ss -tuln | grep 2181 tcp LISTEN 0 50 *:2181 *:* [@dc1-apache-hbase ~]$ lsof -i :2181 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 388228 _____ 558u IPv6 10088613 0t0 TCP *:eforward (LISTEN) java 388228 _____ 562u IPv6 10092015 0t0 TCP localhost:eforward->localhost:56476 (ESTABLISHED) java 388228 _____ 565u IPv6 10092023 0t0 TCP localhost:eforward->localhost:56492 (ESTABLISHED) java 388372 _____ 575u IPv6 10092011 0t0 TCP localhost:56476->localhost:eforward (ESTABLISHED) java 388566 _____ 575u IPv6 10088671 0t0 TCP localhost:56492->localhost:eforward (ESTABLISHED)
... View more
09-01-2024
09:32 PM
1 Kudo
I am currently facing an issue while using the ImportTsv command to load data into an HBase table. I am running the command and the job is failing with a non-zero exit code 1. ./hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns="HBASE_ROW_KEY,cf:name,cf:age,cf:gender" -Dimporttsv.separator="\t" my_1 /hbase/test2.txt ERROR LOG: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/super/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/super/hbase/lib/client-facing-thirdparty/slf4j-reload4j-1.7.33.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory] 2024-09-02 09:48:21,871 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2024-09-02 09:48:22,149 INFO [ReadOnlyZKClient-localhost:2181@0x68267da0] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.8.4-9316c2a7a97e1666d8f4593f34dd6fc36ecc436c, built on 2024-02-12 22:16 UTC 2024-09-02 09:48:22,149 INFO [ReadOnlyZKClient-localhost:2181@0x68267da0] zookeeper.ZooKeeper: Client environment:host.name=dc1-apache-hbase.____________ ... 2024-09-02 09:48:32,995 INFO [main] mapreduce.Job: 8]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Hadoop version : 3.4.0 Hbase version : 2.4.18 Resource Manager Logs: [2024-09-02 09:48:32.499]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. For more detailed output, check the application tracking page: http://______________________:8088/cluster/app/application_1725184331906_0005 Then click on links to logs of each attempt. . Failing the application. 2024-09-02 09:48:32,505 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1725184331906_0005 State change from FINAL_SAVING to FAILED on event = APP_UPDATE_SAVED 2024-09-02 09:48:32,506 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue: Application removed - appId: application_1725184331906_0005 user: super leaf-queue of parent: root #applications: 0 2024-09-02 09:48:32,506 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1725184331906_0005,name=importtsv_my_1,user=super,queue=root.default,state=FAILED,trackingUrl=http://_________________________:8088/cluster/app/application_1725184331906_0005,appMasterHost=dc1-apache-hbase._____________,submitTime=1725250705839,startTime=1725250705856,launchTime=1725250706283,finishTime=1725250712503,finalStatus=FAILED,memorySeconds=10734,vcoreSeconds=5,preemptedMemorySeconds=0,preemptedVcoreSeconds=0,preemptedAMContainers=0,preemptedNonAMContainers=0,preemptedResources=<memory:0\, vCores:0>,applicationType=MAPREDUCE,resourceSeconds=10734 MB-seconds\, 5 vcore-seconds,preemptedResourceSeconds=0 MB-seconds\, 0 vcore-seconds,applicationTags=,applicationNodeLabel=,diagnostics=Application application_1725184331906_0005 failed 2 times due to AM Container for appattempt_1725184331906_0005_000002 exited with exitCode: 1\nFailing this attempt.Diagnostics: [2024-09-02 09:48:32.496]Exception from container-launch.\nContainer id: container_1725184331906_0005_02_000001\nExit code: 1\n\n[2024-09-02 09:48:32.498]Container exited with a non-zero exit code 1. Error file: prelaunch.err.\nLast 4096 bytes of prelaunch.err :\nLast 4096 bytes of stderr :\nlog4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).\nlog4j:WARN Please initialize the log4j system properly.\nlog4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.\n\n\n[2024-09-02 09:48:32.499]Container exited with a non-zero exit code 1. Error file: prelaunch.err.\nLast 4096 bytes of prelaunch.err :\nLast 4096 bytes of stderr :\nlog4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).\nlog4j:WARN Please initialize the log4j system properly.\nlog4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.\n\n\nFor more detailed output\, check the application tracking page: http://_________________________________________________________--/cluster/app/application_1725184331906_0005 Then click on links to logs of each attempt.\n. Failing the application.,totalAllocatedContainers=2
... View more
Labels:
- Labels:
-
Apache HBase