Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Ambari setup not starting services

Explorer

I am using ambari 2.2.1 and HDP 2.3.4 and ubuntu 12. I successfully completed the ambari setup through launch wizard but it never started the services. I am manually starting the services now through "start all" but it fails to start them.

I get the following error for all the services :

resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://xxxx:50070/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs'' returned status_code=503.

I checked that the node is reachable from the server and hdfs is working fine. What am I missing here? Thanks

1 ACCEPTED SOLUTION

New Contributor

I had the same issues and wanted to update with what I debugged and how I fixed it. I pulled back the environment that the user was running through the ambari-sudo script. The quick test is

/var/lib/ambari-agent/ambari-sudo.sh su hdfs -l -s /bin/bash -c 'env'
That showed me that there were http_proxy values being set which I found out was because of the default env variables that are being source from /etc/profile.d/ directory. For my fix, I just added the local machines to no_proxy and the install went forward. Hope that helps

View solution in original post

7 REPLIES 7

Can you try to run just this command, make sure you get HTTP status 200 and no errors:

curl -iv 'http://xxxx:50070/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs

If /ats/done doesn't exist, replace it with /tmp in the command. Last time I saw this issue the reason was http proxy, so make sure you have no proxy to servers in the cluster.

Explorer

@Predrag Minovic : Yes, I am running the cluster behind the firewall and have proxies. I added nodes to the no_proxy list and I'm now able to run curl command from the terminal with HTTP 200. I also updated no_proxy list on all the nodes and removed proxies from ambari-env.sh. But I still the error when trying to start services. I started with yarn service and see it fail. It seems like curl/ambari-server is still using proxies. Is there a way to force it to not use proxy ?

Well, not sure, if curl from cli works, it should work. Can you try to restart ambari-server and all ambari-agents.

Explorer

Started server and agent but no luck. How can we ask ambari to not use proxy ? Which user is this process run as ? May be I can sudo as that and update its no_proxy list ?

Explorer

Can we please confirm which user runs the service installation ? root ? I confirmed that root can run curl from cli successfully.

I created app-logs dir manually and ran

root:/home/ubuntu# curl -iv 'http://xxxx:50070/webhdfs/v1/app-logs?op=GETFILESTATUS&user.name=hdfs'
*   Trying 1.70.26.132... connected
> GET /webhdfs/v1/app-logs?op=GETFILESTATUS&user.name=hdfs HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Cache-Control: no-cache
Cache-Control: no-cache

New Contributor

I had the same issues and wanted to update with what I debugged and how I fixed it. I pulled back the environment that the user was running through the ambari-sudo script. The quick test is

/var/lib/ambari-agent/ambari-sudo.sh su hdfs -l -s /bin/bash -c 'env'
That showed me that there were http_proxy values being set which I found out was because of the default env variables that are being source from /etc/profile.d/ directory. For my fix, I just added the local machines to no_proxy and the install went forward. Hope that helps

Problem accessing /webhdfs/v1/ats/done. Reason

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.