How can I run linux commands after cloudera HDFS starts?

[#1]

Hello,

I'm using Ubuntu 16 and Cloudera 6.2.1. I need to startup a couple other agents only after CDH HDFS has completely started upon OS start, which can take several minutes. I am unsure how to do this.

Anyone have any pointers?  

Thank you.

[#2 · Accepted solution]

This will depend on what these other agents are to a large extent, but in the general case there are two options:

1) Run a cron job: one, check to see if the process is already running; two, check to see if HDFS is running; three start the process if neither of the above are true.

2) I believe Ubuntu 16 uses systemd. In systemd start order can be controlled via dependencies in the unit file of the process. You will have to reference it specifically for your version, but I believe there are two relevant settings: wants or requires determines which processes run together; before and after determines the order in which they are run. The most thorough solution is to set these other agents' unit files as requires HDFS and after HDFS.

[#3]

I think you can just run the linux command anytime. 

Your command will only be HDFS command once you use #hdfs dfs 

[#4]

Thanks for your reply... but what I need to do is run several applications only AFTER hdfs is fully started. I want this to happen automatically at OS start. I can get the applications to run at startup but they do not wait until hdfs is fully started. I need to somehow make sure hdfs is started before running the applications.
Anyone know how I might accomplish this? Thanks.

[#5]

Hello @HV_Training ,

thank you for reaching out to Community with your question on how to check whether HDFS is started.

Can you please clarify what do you exactly consider "HDFS is fully started"?

If we are happy with an active NameNode as a "HDFS service started" state, I would use the CM API to find out if I already have an active NameNode by a REST call:

https://archive.cloudera.com/cm6/6.3.0/generic/jar/cm_api/apidocs/json_ApiNameservice.html

Please let us know if it covers your inquiry.

Thank you:
Ferenc

[#6 · Accepted solution]

@muslihuddin , you can delay other agent startups using the “X-GNOME-Autostart-Delay” parameter under ~/.config/autostart/

You can add a delay by opening ~/.config/autostart and edit the related files. You should add the following commands:

X-GNOME-Autostart-enabled=true
X-GNOME-Autostart-Delay=100

 Ref: https://askubuntu.com/questions/48321/how-do-i-start-applications-automatically-on-login/48327#48327 

[#7]

Thanks to everyone to answered this question. Many were helpful!

[#8 · Accepted solution]

This will depend on what these other agents are to a large extent, but in the general case there are two options:

1) Run a cron job: one, check to see if the process is already running; two, check to see if HDFS is running; three start the process if neither of the above are true.

2) I believe Ubuntu 16 uses systemd. In systemd start order can be controlled via dependencies in the unit file of the process. You will have to reference it specifically for your version, but I believe there are two relevant settings: wants or requires determines which processes run together; before and after determines the order in which they are run. The most thorough solution is to set these other agents' unit files as requires HDFS and after HDFS.

[#9]

I used the systemd. It worked great. Again, I appreciate everyone's help!