question Re: problem in moving the input text file from file system to hdfs dfs file system with following co in Archives of Support Questions (Read Only)

problem in moving the input text file from file system to hdfs dfs file system with following comman

subbu — Sat, 14 Apr 2018 15:03:51 GMT

1) [cloudera@localhost ~]$ sudo -u hdfs hdfs dfs -put /home/cloudera/ipf.txt /inputnew/
put: `/home/cloudera/ipf.txt': No such file or directory
2) [cloudera@localhost ~]$ hdfs dfs -put /home/cloudera/ipf.txt /inputnew/
put: Permission denied: user=cloudera, access=WRITE, inode="/inputnew":hdfs:supergroup:drwxr-xr-x

3) [cloudera@localhost ~]$ sudo -u cloudera hdfs dfs -put /home/cloudera/ipf.txt /inputnew/
put: Permission denied: user=cloudera, access=WRITE, inode="/inputnew":hdfs:supergroup:drwxr-xr-x

Re: problem in moving the input text file from file system to hdfs dfs file system with following co

Fawze — Sat, 14 Apr 2018 21:21:14 GMT

1) [cloudera@localhost ~]$ sudo -u hdfs hdfs dfs -put /home/cloudera/ipf.txt /inputnew/
put: `/home/cloudera/ipf.txt': No such file or directory

The file /home/cloudera/ipf.txt doesn't exist in you local host, you can check by ll /home/cloudera/

Below you are not using the sudo -u hdfs as you used in the above command.

** you faced the same issue in another post.

Please use sudo -u hdfs hdfs dfs -put /home/cloudera/ipf.txt /inputnew/
2) [cloudera@localhost ~]$ hdfs dfs -put /home/cloudera/ipf.txt /inputnew/
put: Permission denied: user=cloudera, access=WRITE, inode="/inputnew":hdfs:supergroup:drwxr-xr-x

3) [cloudera@localhost ~]$ sudo -u cloudera hdfs dfs -put /home/cloudera/ipf.txt /inputnew/
put: Permission denied: user=cloudera, access=WRITE, inode="/inputnew":hdfs:supergroup:drwxr-xr-x

Re: problem in moving the input text file from file system to hdfs dfs file system with following co

subbu — Sun, 15 Apr 2018 11:23:54 GMT

actually the txt file is already there in the folder and i am issuing the correct command but still i getting error.can u suggest me a solution

Re: problem in moving the input text file from file system to hdfs dfs file system with following co

Fawze — Sun, 15 Apr 2018 11:26:47 GMT

PLease send me the output of the ll command

Re: problem in moving the input text file from file system to hdfs dfs file system with following co

GeKas — Tue, 17 Apr 2018 08:36:16 GMT

1) User hdfs does not have access to the /home/cloudera directory

2) and 3) is actually the same because in both cases you try to upload the file as user cloudera.

You have two options:

1) grant read permissions to hdfs user in /home/cloudera and all sub-contents (directory access require also execute permission)

2) grant write permissions in "/inputnew/" directory in HDFS , to "cloudera" user.

example: sudo -u hdfs hdfs dfs -chown cloudera /inputnew

There are multiple ways to grant permissions (e.g. using ACLs), but keep it simple.

Re: problem in moving the input text file from file system to hdfs dfs file system with following co

subbu — Tue, 17 Apr 2018 16:30:33 GMT

put: Permission denied: user=cloudera, access=WRITE, inode="/inputnew":hdfs:supergroup:drwxr-xr-x

Re: problem in moving the input text file from file system to hdfs dfs file system with following co

subbu — Tue, 17 Apr 2018 16:31:40 GMT

how to give permission to cloudera

Re: problem in moving the input text file from file system to hdfs dfs file system with following co

GeKas — Tue, 17 Apr 2018 17:01:16 GMT

Use the example I wrote above. It will change the owner of /inputnew directory in hdfs to "cloudera"

sudo -u hdfs hdfs dfs -chown cloudera /inputnew

Re: problem in moving the input text file from file system to hdfs dfs file system with following co

subbu — Wed, 18 Apr 2018 15:04:46 GMT

thank u,its working

Excution of the following command gives warning and not executed

subbu — Wed, 18 Apr 2018 15:15:03 GMT

hadoop jar /home/cloudera/WordCount.jar WordCount /inputnew/inputfile.txt /outputnew

error message is:

[cloudera@localhost ~]$ hadoop jar /home/cloudera/WordCount.jar WordCount /inputnew/inputfile.txt /outputnew
18/04/18 07:57:24 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
18/04/18 07:57:25 INFO input.FileInputFormat: Total input paths to process : 1
18/04/18 07:57:25 INFO mapred.JobClient: Running job: job_201804180742_0002
18/04/18 07:57:26 INFO mapred.JobClient: map 0% reduce 0%
18/04/18 07:57:36 INFO mapred.JobClient: Task Id : attempt_201804180742_0002_m_000002_0, Status : FAILED
org.apache.hadoop.security.AccessControlException: Permission denied: user=cloudera, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:224)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:204)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:149)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4705)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4687)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:4661)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3032)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:2996)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:2977)

Re: Excution of the following command gives warning and not executed

Fawze — Wed, 18 Apr 2018 16:40:32 GMT

@subbu You are doing the same error all the time.

In order to write or read from and to the jdfs the user running the command need permissions.

You can solve this by always running with the super user "hdfs" and in order to do that you need to add for you command sudo -u hdfs, so your command should be :

sudo -u hdfs hadoop jar /home/cloudera/WordCount.jar WordCount /inputnew/inputfile.txt /outputnew

or as i see all the time you are running with cloudera user so you need to change the owner of / to cloudera or you can change the permissions to the root folder.

so:

1- use the super user:

sudo -u hdfs hadoop jar /home/cloudera/WordCount.jar WordCount /inputnew/inputfile.txt /outputnew

2- change root dir to cloudera-scm:

sudo -u hdfs hdfs -chown -R cloudera /

then run:

hadoop jar /home/cloudera/WordCount.jar WordCount /inputnew/inputfile.txt /outputnew

3- change the permissions:

sudo -u hdfs hdfs -chmod -R 777 / then run:

hadoop jar /home/cloudera/WordCount.jar WordCount /inputnew/inputfile.txt /outputnew

Re: Excution of the following command gives warning and not executed

subbu — Thu, 19 Apr 2018 09:17:13 GMT

sir while i am trying to exucute this getting error

cloudera@localhost ~]$ sudo -u hdfs hdfs -chown -R cloudera /
Unrecognized option: -chown
Could not create the Java virtual machine.

Re: Excution of the following command gives warning and not executed

Fawze — Thu, 19 Apr 2018 09:27:47 GMT

sudo -u hdfs hdfs dfs -chown -R cloudera /

Re: Excution of the following command gives warning and not executed

subbu — Fri, 20 Apr 2018 06:50:41 GMT

thank u very much sir,its working and i am able run and got an output also

Re: Excution of the following command gives warning and not executed

subbu — Sun, 22 Apr 2018 14:46:43 GMT

While executing this command I am getting error as
[cloudera@localhost ~]$ hadoop jar /home/cloudera/WordCount.jar WordCount /inputnew/inputfile.txt /op
18/04/22 07:40:40 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

Re: Excution of the following command gives warning and not executed

Fawze — Tue, 24 Apr 2018 21:06:01 GMT

Where did you get this command from?

Are you have the hadoop-example jar?

Check the jars under /home/cloudera/ by running ll /home/cloudera/

Runnigcloudera@localhost ~]$ hadoop jar WordCount.jar WordCount /inputnew2/inputfile.txt /output_new

subbu — Sat, 09 Jun 2018 10:00:47 GMT

While running a wordcount program i am getting the following error.

cloudera@localhost ~]$ hadoop jar WordCount.jar WordCount /inputnew2/inputfile.txt /output_new

18/06/09 00:29:06 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

18/06/09 00:29:07 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

18/06/09 00:29:08 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

18/06/09 00:29:09 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

18/06/09 00:29:10 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

18/06/09 00:29:11 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

18/06/09 00:29:12 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

18/06/09 00:29:13 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

18/06/09 00:29:14 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

18/06/09 00:29:15 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

18/06/09 00:29:15 ERROR security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to localhost.localdomain:8021 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

Exception in thread "main" java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to localhost.localdomain:8021 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)

at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)

at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729)

at org.apache.hadoop.ipc.Client.call(Client.java:1241)

at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)

at org.apache.hadoop.mapred.$Proxy10.getStagingAreaDir(Unknown Source)

at org.apache.hadoop.mapred.JobClient.getStagingAreaDir(JobClient.java:1324)

at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:102)

at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:951)

at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)

at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)

at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)

at WordCount.main(WordCount.java:132)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

Caused by: java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)

at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)

at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)

at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492)

at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:509)

at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:603)

at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:252)

at org.apache.hadoop.ipc.Client.getConnection(Client.java:1290)

at org.apache.hadoop.ipc.Client.call(Client.java:1208)

... 18 more