Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How can I run linux commands after cloudera HDFS starts?

avatar
Explorer

Hello,

 

I'm using Ubuntu 16 and Cloudera 6.2.1. I need to startup a couple other agents only after CDH HDFS has completely started upon OS start, which can take several minutes. I am unsure how to do this.

 

Anyone have any pointers?  

 

Thank you.

2 ACCEPTED SOLUTIONS

avatar
Expert Contributor

This will depend on what these other agents are to a large extent, but in the general case there are two options:

 

1) Run a cron job: one, check to see if the process is already running; two, check to see if HDFS is running; three start the process if neither of the above are true.

 

2) I believe Ubuntu 16 uses systemd. In systemd start order can be controlled via dependencies in the unit file of the process. You will have to reference it specifically for your version, but I believe there are two relevant settings: wants or requires determines which processes run together; before and after determines the order in which they are run. The most thorough solution is to set these other agents' unit files as requires HDFS and after HDFS.

View solution in original post

avatar
Expert Contributor

@muslihuddin , you can delay other agent startups using the “X-GNOME-Autostart-Delay” parameter under ~/.config/autostart/

 

You can add a delay by opening ~/.config/autostart and edit the related files. You should add the following commands:

X-GNOME-Autostart-enabled=true
X-GNOME-Autostart-Delay=100

 Ref: https://askubuntu.com/questions/48321/how-do-i-start-applications-automatically-on-login/48327#48327 

View solution in original post

7 REPLIES 7

avatar
Rising Star

I think you can just run the linux command anytime. 

 

Your command will only be HDFS command once you use #hdfs dfs 

 

avatar
Explorer
Thanks for your reply... but what I need to do is run several applications only AFTER hdfs is fully started. I want this to happen automatically at OS start. I can get the applications to run at startup but they do not wait until hdfs is fully started. I need to somehow make sure hdfs is started before running the applications.
Anyone know how I might accomplish this? Thanks.

avatar
Moderator

Hello @HV_Training ,

 

thank you for reaching out to Community with your question on how to check whether HDFS is started.

 

Can you please clarify what do you exactly consider "HDFS is fully started"?

 

If we are happy with an active NameNode as a "HDFS service started" state, I would use the CM API to find out if I already have an active NameNode by a REST call:

https://archive.cloudera.com/cm6/6.3.0/generic/jar/cm_api/apidocs/json_ApiNameservice.html

 

Please let us know if it covers your inquiry.


Thank you:
Ferenc


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

avatar
Expert Contributor

@muslihuddin , you can delay other agent startups using the “X-GNOME-Autostart-Delay” parameter under ~/.config/autostart/

 

You can add a delay by opening ~/.config/autostart and edit the related files. You should add the following commands:

X-GNOME-Autostart-enabled=true
X-GNOME-Autostart-Delay=100

 Ref: https://askubuntu.com/questions/48321/how-do-i-start-applications-automatically-on-login/48327#48327 

avatar
Explorer

Thanks to everyone to answered this question. Many were helpful!

avatar
Expert Contributor

This will depend on what these other agents are to a large extent, but in the general case there are two options:

 

1) Run a cron job: one, check to see if the process is already running; two, check to see if HDFS is running; three start the process if neither of the above are true.

 

2) I believe Ubuntu 16 uses systemd. In systemd start order can be controlled via dependencies in the unit file of the process. You will have to reference it specifically for your version, but I believe there are two relevant settings: wants or requires determines which processes run together; before and after determines the order in which they are run. The most thorough solution is to set these other agents' unit files as requires HDFS and after HDFS.

avatar
Explorer

I used the systemd. It worked great. Again, I appreciate everyone's help!