hadoop smoke test mapreduce

[#1]

Hello, I am trying to run smoke test mapreduce on a cluster but I am getting some errors.

I follow these instructions, in text bold I marked the errors

I run the commands from a slavenode wihich has the client tools all installed

5.6. Smoke Test MapReduce

1. Browse to the ResourceManager:

http://$resourcemanager.full.hostname:8088/

2. Create a $CLIENT_USER in all of the nodes and add it to the users group.

useradd client

usermod -a -G users client

3. As the HDFS user, create a /user/$CLIENT_USER.

sudo su - $HDFS_USER

hdfs dfs -mkdir /user/$CLIENT_USER

hdfs dfs -chown $CLIENT_USER:$CLIENT_USER /user/$CLIENT_USER //error

hdfs dfs -chmod -R 755 /user/$CLIENT_USER

4. Run the smoke test as the $CLIENT_USER. Using Terasort, sort 10GB of data.

su - $CLIENT_USER

/usr/hdp/current/hadoop-client/bin/hadoop jar /usr/hdp/current/hadoopmapreduce-

client/hadoop-mapreduce-examples-*.jar teragen 10000 tmp/

teragenout

/usr/hdp/current/hadoop-client/bin/hadoop jar /usr/hdp/current/hadoopmapreduce-

client/hadoop-mapreduce-examples-*.jar terasort tmp/teragenout

tmp/terasortout //error there is no such as path user/root/...

[#2]

@spriv

There is an issue with your file path. I have just tried to run the same test successfully. My environment is kerberized so here is the output.

Created user opera1 and changed the password on the Linux box

# useradd opera1
# passwd opera1

Create the user home directory in hdfs

$ hdfs dfs -mkdir /user/opera1

changed the permissions

$ hdfs dfs -chmod -R 755 /user/opera1

Check the file system

$ hdfs dfs -ls /user
Found 9 items
drwx------   - accumulo  hdfs          0 2017-07-31 00:28 /user/accumulo
drwxr-xr-x   - admin     hdfs          0 2017-08-26 23:56 /user/admin
drwxrwx---   - ambari-qa hdfs          0 2017-08-24 16:08 /user/ambari-qa
drwxr-xr-x   - hbase     hdfs          0 2017-07-31 00:26 /user/hbase
drwxr-xr-x   - hcat      hdfs          0 2017-07-31 00:32 /user/hcat
drwxr-xr-x   - hive      hdfs          0 2017-08-27 00:20 /user/hive
drwxrwxr-x   - oozie     hdfs          0 2017-07-31 00:35 /user/oozie
drwxr-xr-x   - opera1    hdfs          0 2017-09-10 20:44 /user/opera1
drwxr-xr-x   - zeppelin  hdfs          0 2017-07-31 00:27 /user/zeppelin

Because of Kerberos I had to get a valid ticket

$ kinit -kt /etc/security/keytabs/opera1.keytab opera1@UK.COM

Then I run the command smoke test, I noticed you missed the "/" before the tmp

$ /usr/hdp/current/hadoop-client/bin/hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar teragen 10000 /tmp/teragenout

The output is below

17/09/10 20:44:37 INFO client.RMProxy: Connecting to ResourceManager at london.uk.com/192.168.0.80:8050
17/09/10 20:44:37 INFO client.AHSProxy: Connecting to Application History server at london.uk.com/192.168.0.80:10200
17/09/10 20:44:37 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 57 for opera1 on 192.168.0.80:8020
17/09/10 20:44:39 INFO security.TokenCache: Got dt for hdfs://london.uk.com:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.0.80:8020, Ident: (HDFS_DELEGATION_TOKEN token 57 for opera1)
17/09/10 20:44:39 INFO security.TokenCache: Got dt for hdfs://london.uk.com:8020; Kind: kms-dt, Service: 192.168.0.80:9292, Ident: (owner=opera1, renewer=yarn, realUser=, issueDate=1505069078794, maxDate=1505673878794, sequenceNumber=3, masterKeyId=4)
17/09/10 20:44:41 INFO terasort.TeraSort: Generating 10000 using 2
17/09/10 20:44:41 INFO mapreduce.JobSubmitter: number of splits:2
17/09/10 20:44:42 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504811137859_0003
17/09/10 20:44:42 INFO mapreduce.JobSubmitter: Kind: kms-dt, Service: 192.168.0.80:9292, Ident: (owner=opera1, renewer=yarn, realUser=, issueDate=1505069078794, maxDate=1505673878794, sequenceNumber=3, masterKeyId=4)
17/09/10 20:44:42 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.0.80:8020, Ident: (HDFS_DELEGATION_TOKEN token 57 for opera1)
17/09/10 20:44:42 INFO impl.TimelineClientImpl: Timeline service address: http://london.uk.com:8188/ws/v1/timeline/
17/09/10 20:44:45 INFO impl.YarnClientImpl: Application submission is not finished, submitted application application_1504811137859_0003 is still in NEW
17/09/10 20:44:46 INFO impl.YarnClientImpl: Submitted application application_1504811137859_0003
17/09/10 20:44:46 INFO mapreduce.Job: The url to track the job: http://london.uk.com:8088/proxy/application_1504811137859_0003/
17/09/10 20:44:46 INFO mapreduce.Job: Running job: job_1504811137859_0003
17/09/10 20:45:13 INFO mapreduce.Job: Job job_1504811137859_0003 running in uber mode : false
17/09/10 20:45:13 INFO mapreduce.Job:  map 0% reduce 0%
17/09/10 20:45:55 INFO mapreduce.Job:  map 100% reduce 0%
17/09/10 20:45:59 INFO mapreduce.Job: Job job_1504811137859_0003 completed successfully
17/09/10 20:45:59 INFO mapreduce.Job: Counters: 31
        File System Counters
                FILE: Number of bytes read=0
                FILE: Number of bytes written=308960
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=164
                HDFS: Number of bytes written=1000000
                HDFS: Number of read operations=8
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=4
        Job Counters
                Launched map tasks=2
                Other local map tasks=2
                Total time spent by all maps in occupied slots (ms)=153252
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=76626
                Total vcore-milliseconds taken by all map tasks=76626
                Total megabyte-milliseconds taken by all map tasks=117697536
        Map-Reduce Framework
                Map input records=10000
                Map output records=10000
                Input split bytes=164
                Spilled Records=0
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=25582
                CPU time spent (ms)=2570
                Physical memory (bytes) snapshot=310550528
                Virtual memory (bytes) snapshot=6513610752
                Total committed heap usage (bytes)=197132288
        org.apache.hadoop.examples.terasort.TeraGen$Counters
                CHECKSUM=21555350172850
        File Input Format Counters
                Bytes Read=0
        File Output Format Counters
                Bytes Written=1000000

And the Resource Manager UI shows the SUCCESS !!! see screenshot attached.

[#3]

@spriv

Has your problem been resolved?

[#4]

no, I am using centos 6.8

[#5]

@spriv

But running HDP on Centos 6 or 7 makes no difference, the above works for both.

What is the centos6 specific problem that you have ?