question Re: HDFS Cannot Create the directory of the user in Support Questions

HDFS Cannot Create the directory of the user

Justee — Sun, 20 Jun 2021 11:26:06 GMT

I installed hadoop-2.7.1 on a centos7 operating system.
when I execute the command # hdfs dfs -mkdir /user/Juste . I get an error

[hdfs@MASTER ~]$ hdfs dfs -mkdir /user/Juste
21/06/20 11:04:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
mkdir: `/user/Juste': No such file or directory

Re: HDFS Cannot Create the directory of the user

ChethanYM — Sun, 20 Jun 2021 13:41:00 GMT

Hi,

Hadoop has native implementations of certain components for both performance and non-availability of Java implementations. These components are available in a single, dynamically-linked native Linux library called the native hadoop library. On the *nix platforms the library is named libhadoop.so.

It is just a Warning but not an error and it can be ignored.

Please check if you have an issue with other hdfs commands as well...and check the hdfs logs and see if you can get exact errors.

Re: HDFS Cannot Create the directory of the user

Justee — Sun, 20 Jun 2021 14:04:12 GMT

When I execute this command this is the output

[hdfs@MASTER ~]$ hdfs dfs -mkdir /user
21/06/20 13:51:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
mkdir: Cannot create directory /user. Name node is in safe mode

Re: HDFS Cannot Create the directory of the user

ChethanYM — Sun, 20 Jun 2021 14:12:29 GMT

Safemode in Hadoop is a maintenance state of NameNode, during which NameNode doesn’t allow any modifications to the file system.

can you use below command to come out of the safemode and try to create a directory?

## hadoop dfsadmin -safemode leave

## hdfs dfs -mkdir /user/justee

Re: HDFS Cannot Create the directory of the user

Justee — Sun, 20 Jun 2021 18:46:07 GMT

Hi @ChethanYM , when i created hdfs user i didn't create with a password but when i execute this is what i'm asked. i used the root password but it doesn't pass either.

[hdfs@MASTER ~]$ hdfs dfsadmin -safemode leave
21/06/20 17:30:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
safemode: Access denied for user hdfs. Superuser privilege is required
[hdfs@MASTER ~]$ sudo hdfs dfsadmin -safemode leave
[sudo] password for hdfs:
sorry, try again.
[sudo] password for hdfs:
sorry, try again.
[sudo] password for hdfs:

Re: HDFS Cannot Create the directory of the user

ChethanYM — Mon, 21 Jun 2021 02:59:10 GMT

Hi,

Can you try with below command and see?

## sudo -u hdfs hdfs dfsadmin -safemode leave

If this doesn't work provide the output of below from the terminal:

## id hdfs

Re: HDFS Cannot Create the directory of the user

Justee — Mon, 21 Jun 2021 07:26:27 GMT

Hi @ChethanYM, I tried the following command again another error

[root@MASTER ~]# su - hdfs
Last login: Sun Jun 20 17:27:39 UTC 2021 on pts/5
[hdfs@MASTER ~]$ sudo -u hdfs hdfs dfsadmin -safemode leave
sudo: hdfs: command not found
[hdfs@MASTER ~]$ id hdfs
uid=30008(hdfs) gid=30009(hdfs) groups=30009(hdfs)
[hdfs@MASTER ~]$

Re: HDFS Cannot Create the directory of the user

RohitPathak — Mon, 21 Jun 2021 07:58:42 GMT

You don't need to switch to hdfs user to run below command.

Try to run the command when you are root/sudo user

sudo -u hdfs hdfs dfsadmin -safemode leave

Re: HDFS Cannot Create the directory of the user

Justee — Mon, 21 Jun 2021 08:59:00 GMT

Hi @RohitPathak, I tried with root/sudo user but still error,
[root@MASTER ~]# sudo -u hdfs hdfs dfsadmin -safemode leave
sudo: hdfs: command not found
[root@MASTER ~]#

Re: HDFS Cannot Create the directory of the user

ChethanYM — Mon, 21 Jun 2021 13:31:26 GMT

Seems to be hdfs is not installed/configured properly, Can you check the path of it by using "which hdfs" command.

If you are not able to see the path check the environment variable in "~/.bash_profile" file. set the path something like below and try:

## PATH=$PATH:$HADOOP_HOME/bin

then run -> source ~/.bash_profile

Below is the output from my test cluster:

[root@node2 bin]# which hdfs

/usr/bin/hdfs

[root@node2 bin]# sudo -u hdfs hdfs dfsadmin -safemode leave

Safe mode is OFF

[root@node2 bin]# id hdfs

uid=993(hdfs) gid=990(hdfs) groups=990(hdfs)

Re: HDFS Cannot Create the directory of the user

Justee — Mon, 21 Jun 2021 16:05:51 GMT

Hi @ChethanYM, I set the PATH like below and try:
When I run this command, I get the same error.

## PATH=$PATH:$HADOOP_HOME/bin

## source ~/.bash_profile
[root@MASTER ~]# which hdfs
/usr/local/hadoop-2.7.1/bin/hdfs
[root@MASTER ~]# sudo -u hdfs hdfs dfsadmin -safemode leave
sudo: hdfs: command not found
[root@MASTER ~]# id hdfs
uid=30008(hdfs) gid=30009(hdfs) groups=30009(hdfs)
[root@MASTER ~]#

Re: HDFS Cannot Create the directory of the user

ChethanYM — Mon, 21 Jun 2021 16:37:40 GMT

Hi,

Go to /etc/alterenatives provide the output for below commands: ( This is to check if linux alternatives subsystem is pointing to the binaries from an other old version of CDH that you are not using currently)

Example:

[root@node2 alternatives]# ls -lrth | grep hdfs

lrwxrwxrwx 1 root root 62 Aug 15 2020 hdfs -> /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.4951328/bin/hdfs

[root@node2 alternatives]# ls -lrth /usr/bin/hdfs

lrwxrwxrwx 1 root root 22 Aug 15 2020 /usr/bin/hdfs -> /etc/alternatives/hdfs

What is the CDH version you are using currently? Have you recently upgraded the cluster?
In my previous comment I have added “#” just for pointing the command if you have added same in the ~/.bash_profile do remove the “#” and try.
How you installed hdfs? is it possible for you to re install it?

Re: HDFS Cannot Create the directory of the user

Justee — Mon, 21 Jun 2021 17:23:41 GMT

Hi @ChethanYM, I installed hadoop-2.7.1 and it's during the installation that I created a hdfs user. but the hdfs command started working when I finished installing hadoop. here is the output of the command #ls -lrth | grep hdfs. the output does not give anything. Yes I can reinstall hadoop.

[root@MASTER alternatives]# ls -lrth | grep hdfs
[root@MASTER alternatives]#
[root@MASTER alternatives]# hadoop version
Hadoop 2.7.1
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 20fe5304904fc2f5a18053c389e43cd26f7a70fe
Compiled by vinodkv on 2017-06-02T06:14Z
Compiled with protoc 2.5.0
From source with checksum 60125541c2b3e266cbf3becc5bda666
This command was run using /usr/local/hadoop-2.7.1/share/hadoop/common/hadoop-common-2.7.1.jar
[root@MASTER alternatives]#

Re: HDFS Cannot Create the directory of the user

ChethanYM — Tue, 22 Jun 2021 03:23:21 GMT

Yes, It seems to be not properly installed.

May i know you are using plain Hadoop or CDH or HDP to manage it?

If you have followed any document for the hadoop installation provide the link here...

Re: HDFS Cannot Create the directory of the user

Justee — Tue, 22 Jun 2021 06:03:41 GMT

Hi @ChethanYM, I just use hadoop. I followed This tutorial , but I did the installation steps of hadoop only. https://www.youtube.com/watch?v=71EQblrUPRM&t=1375s