Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How can I run linux commands after cloudera HDFS starts?

Solved Go to solution
Highlighted

How can I run linux commands after cloudera HDFS starts?

New Contributor

Hello,

 

I'm using Ubuntu 16 and Cloudera 6.2.1. I need to startup a couple other agents only after CDH HDFS has completely started upon OS start, which can take several minutes. I am unsure how to do this.

 

Anyone have any pointers?  

 

Thank you.

2 ACCEPTED SOLUTIONS

Accepted Solutions
Highlighted

Re: How can I run linux commands after cloudera HDFS starts?

Cloudera Employee

This will depend on what these other agents are to a large extent, but in the general case there are two options:

 

1) Run a cron job: one, check to see if the process is already running; two, check to see if HDFS is running; three start the process if neither of the above are true.

 

2) I believe Ubuntu 16 uses systemd. In systemd start order can be controlled via dependencies in the unit file of the process. You will have to reference it specifically for your version, but I believe there are two relevant settings: wants or requires determines which processes run together; before and after determines the order in which they are run. The most thorough solution is to set these other agents' unit files as requires HDFS and after HDFS.

View solution in original post

Highlighted

Re: How can I run linux commands after cloudera HDFS starts?

Cloudera Employee

@muslihuddin , you can delay other agent startups using the “X-GNOME-Autostart-Delay” parameter under ~/.config/autostart/

 

You can add a delay by opening ~/.config/autostart and edit the related files. You should add the following commands:

X-GNOME-Autostart-enabled=true
X-GNOME-Autostart-Delay=100

 Ref: https://askubuntu.com/questions/48321/how-do-i-start-applications-automatically-on-login/48327#48327 

View solution in original post

7 REPLIES 7
Highlighted

Re: How can I run linux commands after cloudera HDFS starts?

Explorer

I think you can just run the linux command anytime. 

 

Your command will only be HDFS command once you use #hdfs dfs 

 

Highlighted

Re: How can I run linux commands after cloudera HDFS starts?

New Contributor
Thanks for your reply... but what I need to do is run several applications only AFTER hdfs is fully started. I want this to happen automatically at OS start. I can get the applications to run at startup but they do not wait until hdfs is fully started. I need to somehow make sure hdfs is started before running the applications.
Anyone know how I might accomplish this? Thanks.

Re: How can I run linux commands after cloudera HDFS starts?

Moderator

Hello @HV_Training ,

 

thank you for reaching out to Community with your question on how to check whether HDFS is started.

 

Can you please clarify what do you exactly consider "HDFS is fully started"?

 

If we are happy with an active NameNode as a "HDFS service started" state, I would use the CM API to find out if I already have an active NameNode by a REST call:

https://archive.cloudera.com/cm6/6.3.0/generic/jar/cm_api/apidocs/json_ApiNameservice.html

 

Please let us know if it covers your inquiry.


Thank you:
Ferenc


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Highlighted

Re: How can I run linux commands after cloudera HDFS starts?

Cloudera Employee

@muslihuddin , you can delay other agent startups using the “X-GNOME-Autostart-Delay” parameter under ~/.config/autostart/

 

You can add a delay by opening ~/.config/autostart and edit the related files. You should add the following commands:

X-GNOME-Autostart-enabled=true
X-GNOME-Autostart-Delay=100

 Ref: https://askubuntu.com/questions/48321/how-do-i-start-applications-automatically-on-login/48327#48327 

View solution in original post

Highlighted

Re: How can I run linux commands after cloudera HDFS starts?

New Contributor

Thanks to everyone to answered this question. Many were helpful!

Highlighted

Re: How can I run linux commands after cloudera HDFS starts?

Cloudera Employee

This will depend on what these other agents are to a large extent, but in the general case there are two options:

 

1) Run a cron job: one, check to see if the process is already running; two, check to see if HDFS is running; three start the process if neither of the above are true.

 

2) I believe Ubuntu 16 uses systemd. In systemd start order can be controlled via dependencies in the unit file of the process. You will have to reference it specifically for your version, but I believe there are two relevant settings: wants or requires determines which processes run together; before and after determines the order in which they are run. The most thorough solution is to set these other agents' unit files as requires HDFS and after HDFS.

View solution in original post

Highlighted

Re: How can I run linux commands after cloudera HDFS starts?

New Contributor

I used the systemd. It worked great. Again, I appreciate everyone's help!

Don't have an account?
Coming from Hortonworks? Activate your account here