question Re: I have on cluster setup in hortonworks having ambari 2.1 and HDP 2.3. I have to create an automated Script for creating HDFS directories and Quota Setup. Anyone is having any kind of script related to this or any kind of suggestions, kindly post here. in Archives of Support Questions (Read Only)

I have on cluster setup in hortonworks having ambari 2.1 and HDP 2.3. I have to create an automated Script for creating HDFS directories and Quota Setup. Anyone is having any kind of script related to this or any kind of suggestions, kindly post here.

Yukti — Thu, 18 Aug 2016 17:53:27 GMT

Re: I have on cluster setup in hortonworks having ambari 2.1 and HDP 2.3. I have to create an automated Script for creating HDFS directories and Quota Setup. Anyone is having any kind of script related to this or any kind of suggestions, kindly post here.

myoung — Fri, 19 Aug 2016 19:55:38 GMT

You have the core of what you need to do this via a shell script. To do what you are looking for, you you just need to add some more logic. Why not prompt for the directory path, the owner/group, then the quota? You can always check the input for a special value like "-done-" to let the script know you are done.

while [ "$directory" != "-done-" ]
do
  echo "To exit, enter -done- or press ^C"
  read -p "Enter a directory : " directory
  echo $directory

  read -p "Enter permissions (ie: 755) : " permissions
  echo $permissions

  read -p "Enter ownership (ie: user:group) : " ownership
  echo $ownership

  read -p "Enter quota (ie: 1024m) : " quota
  echo $quota

echo "
  hdfs dfs -mkdir $directory
  hdfs dfs -chown $ownership $directory
  hdfs dfs -chmod $permissions $directory
  hdfs dfsadmin -setSpaceQuota $quota $directory"
done

You would still have to enter the info for each directory, which may not be ideal. An alternative would be read the input from a text file. The text file could be something like this (tab delimited):

/user/testuser1	testuser1:testgroup	775	1024M
/user/testuser2	testuser2:testgroup	775	1024M

Then all you need to do is load the file (say we call it myFile.txt) in and parse each line and execute the commands based on the values.

while IFS=$'\t' read -r -a myOptions
do
 echo "directory: ${myOptions[0]}"
 echo "ownership: ${myOptions[1]}"
 echo "permissions: ${myOptions[2]}"
 echo "quota: ${myOptions[3]}"
done < myFile.txt

Re: I have on cluster setup in hortonworks having ambari 2.1 and HDP 2.3. I have to create an automated Script for creating HDFS directories and Quota Setup. Anyone is having any kind of script related to this or any kind of suggestions, kindly post here.

Yukti — Tue, 23 Aug 2016 17:06:01 GMT

Thank you so much @Michael Young... I have created the script...now i want something like...whenever i will run that script it should login as super user which is hdfs in my case and then execute my script. can you please suggest how can i implement this functionality.

Re: I have on cluster setup in hortonworks having ambari 2.1 and HDP 2.3. I have to create an automated Script for creating HDFS directories and Quota Setup. Anyone is having any kind of script related to this or any kind of suggestions, kindly post here.

Yukti — Tue, 23 Aug 2016 18:53:33 GMT

#!/bin/bash

sudo su hdfs << EOF

while IFS=$'\t' read -r -a myOptions

echo "directory: ${myOptions[0]}"

echo "ownership: ${myOptions[1]}"

echo "permissions: ${myOptions[2]}"

echo "quota: ${myOptions[3]}"

hdfs dfs -mkdir ${myOptions[0]}

hdfs dfs -chown ${myOptions[1]} ${myOptions[0]}

hdfs dfs -chmod ${myOptions[2]} ${myOptions[0]}

hdfs dfsadmin -setSpaceQuota ${myOptions[3]} ${myOptions[0]}

done< myFile.txt

EOF

I have used the above script..but this is actually giving an error

bash: line 15: myFile.txt: Permission denied

Please suggest @Michael Young

Re: I have on cluster setup in hortonworks having ambari 2.1 and HDP 2.3. I have to create an automated Script for creating HDFS directories and Quota Setup. Anyone is having any kind of script related to this or any kind of suggestions, kindly post here.

myoung — Tue, 23 Aug 2016 20:45:45 GMT

@Yukti Agrawal

Where is myFile.txt located? What are the file permissions for that file? It is most likely that the hdfs user does not have read access to the file, or does not have access to the directory where the file is located.

Alternatives would be to run the entire script as the hdfs user from outside the script. With both of these methods, the permissions I mentioned above will also need to be fixed. You have a couple of options:

su - hdfs -c <your script name>
sudo -u hdfs <your script name>

Many systems will have sudo available, however you have to be configured with access to run commands via sudo. If you don't have access to sudo, then option number 1 will work for you. However, you have to know the hdfs user password.

If you feel my answer was helpful, don't forget to accept the answer. This helps other people find solutions.

Re: I have on cluster setup in hortonworks having ambari 2.1 and HDP 2.3. I have to create an automated Script for creating HDFS directories and Quota Setup. Anyone is having any kind of script related to this or any kind of suggestions, kindly post here.

Yukti — Wed, 24 Aug 2016 10:05:32 GMT

Unfortunately i dont have sudo access as well as the hdfs password. someone suggested me to do as follows:

#!/bin/bash

while IFS=$'\t' read -r -a myOptions

echo "directory: ${myOptions[0]}"

echo "ownership: ${myOptions[1]}"

echo "permissions: ${myOptions[2]}"

echo "quota: ${myOptions[3]}"

sudo su hdfs -c "hdfs dfs -mkdir ${myOptions[0]}"

sudo su hdfs -c "hdfs dfs -chown ${myOptions[1]} ${myOptions[0]}"

sudo su hdfs -c "hdfs dfs -chmod ${myOptions[2]} ${myOptions[0]}"

sudo su hdfs -c "hdfs dfsadmin -setSpaceQuota ${myOptions[3]} ${myOptions[0]}"

done< myFile.txt

myFile.txt /usr/t1 hive:hive 776 1T

/usr/t2 hdfs:hdfs 775 100G

But its also giving error as follows:

sh create_and_set_quota_alternative.sh

directory: /usr/t1

ownership: hive:hive

permissions: 776

quota: 1T

Error: JAVA_HOME is not set and could not be found.

directory: /usr/t2

ownership: hdfs:hdfs

permissions: 775

quota: 100G

Error: JAVA_HOME is not set and could not be found.

Re: I have on cluster setup in hortonworks having ambari 2.1 and HDP 2.3. I have to create an automated Script for creating HDFS directories and Quota Setup. Anyone is having any kind of script related to this or any kind of suggestions, kindly post here.

myoung — Thu, 25 Aug 2016 02:52:31 GMT

@Yukti Agrawal

When you use sudo, you should use the "-" option to get the user environment settings, etc.

sudo su - hdfs -c "hdfs dfs -mkdir ${myOptions[0]}

Re: I have on cluster setup in hortonworks having ambari 2.1 and HDP 2.3. I have to create an automated Script for creating HDFS directories and Quota Setup. Anyone is having any kind of script related to this or any kind of suggestions, kindly post here.

Yukti — Thu, 25 Aug 2016 14:44:36 GMT

Thanks for your help, Everything is working fine now. I just needed one more modification in this script, i wanted to take myFile.txt from command line, like suppose everytime i dont want to change the content of myFile.txt, instead i want to use different file, so i can do by command line. please suggest how to do that.

Re: I have on cluster setup in hortonworks having ambari 2.1 and HDP 2.3. I have to create an automated Script for creating HDFS directories and Quota Setup. Anyone is having any kind of script related to this or any kind of suggestions, kindly post here.

myoung — Thu, 25 Aug 2016 21:03:12 GMT

@Yukti Agrawal

If you want to get a parameter in a bash script, you use

$1 - first parameter
$2 - second parameter

So in the script you would have something like this:

myFile=$1

Anywhere you had "myFile.txt", you can replace with "$myFile". Naturally you would want to do some error checking make sure the file exists. There are a number of ways to do that.