Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari setup not starting services

SOLVED Go to solution
Highlighted

Ambari setup not starting services

New Contributor

I am using ambari 2.2.1 and HDP 2.3.4 and ubuntu 12. I successfully completed the ambari setup through launch wizard but it never started the services. I am manually starting the services now through "start all" but it fails to start them.

I get the following error for all the services :

resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://xxxx:50070/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs'' returned status_code=503.

I checked that the node is reachable from the server and hdfs is working fine. What am I missing here? Thanks

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Ambari setup not starting services

New Contributor

I had the same issues and wanted to update with what I debugged and how I fixed it. I pulled back the environment that the user was running through the ambari-sudo script. The quick test is

/var/lib/ambari-agent/ambari-sudo.sh su hdfs -l -s /bin/bash -c 'env'
That showed me that there were http_proxy values being set which I found out was because of the default env variables that are being source from /etc/profile.d/ directory. For my fix, I just added the local machines to no_proxy and the install went forward. Hope that helps
7 REPLIES 7

Re: Ambari setup not starting services

Can you try to run just this command, make sure you get HTTP status 200 and no errors:

curl -iv 'http://xxxx:50070/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs

If /ats/done doesn't exist, replace it with /tmp in the command. Last time I saw this issue the reason was http proxy, so make sure you have no proxy to servers in the cluster.

Re: Ambari setup not starting services

New Contributor

@Predrag Minovic : Yes, I am running the cluster behind the firewall and have proxies. I added nodes to the no_proxy list and I'm now able to run curl command from the terminal with HTTP 200. I also updated no_proxy list on all the nodes and removed proxies from ambari-env.sh. But I still the error when trying to start services. I started with yarn service and see it fail. It seems like curl/ambari-server is still using proxies. Is there a way to force it to not use proxy ?

Re: Ambari setup not starting services

Well, not sure, if curl from cli works, it should work. Can you try to restart ambari-server and all ambari-agents.

Re: Ambari setup not starting services

New Contributor

Started server and agent but no luck. How can we ask ambari to not use proxy ? Which user is this process run as ? May be I can sudo as that and update its no_proxy list ?

Re: Ambari setup not starting services

New Contributor

Can we please confirm which user runs the service installation ? root ? I confirmed that root can run curl from cli successfully.

I created app-logs dir manually and ran

root:/home/ubuntu# curl -iv 'http://xxxx:50070/webhdfs/v1/app-logs?op=GETFILESTATUS&user.name=hdfs'
*   Trying 1.70.26.132... connected
> GET /webhdfs/v1/app-logs?op=GETFILESTATUS&user.name=hdfs HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Cache-Control: no-cache
Cache-Control: no-cache

Re: Ambari setup not starting services

New Contributor

I had the same issues and wanted to update with what I debugged and how I fixed it. I pulled back the environment that the user was running through the ambari-sudo script. The quick test is

/var/lib/ambari-agent/ambari-sudo.sh su hdfs -l -s /bin/bash -c 'env'
That showed me that there were http_proxy values being set which I found out was because of the default env variables that are being source from /etc/profile.d/ directory. For my fix, I just added the local machines to no_proxy and the install went forward. Hope that helps

Re: Ambari setup not starting services

New Contributor

Problem accessing /webhdfs/v1/ats/done. Reason