Ambari setup not starting services

[#1]

I am using ambari 2.2.1 and HDP 2.3.4 and ubuntu 12. I successfully completed the ambari setup through launch wizard but it never started the services. I am manually starting the services now through "start all" but it fails to start them.

I get the following error for all the services :

resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://xxxx:50070/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs'' returned status_code=503.

I checked that the node is reachable from the server and hdfs is working fine. What am I missing here? Thanks

[#2]

Can you try to run just this command, make sure you get HTTP status 200 and no errors:

curl -iv 'http://xxxx:50070/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs

If /ats/done doesn't exist, replace it with /tmp in the command. Last time I saw this issue the reason was http proxy, so make sure you have no proxy to servers in the cluster.

[#3]

@Predrag Minovic : Yes, I am running the cluster behind the firewall and have proxies. I added nodes to the no_proxy list and I'm now able to run curl command from the terminal with HTTP 200. I also updated no_proxy list on all the nodes and removed proxies from ambari-env.sh. But I still the error when trying to start services. I started with yarn service and see it fail. It seems like curl/ambari-server is still using proxies. Is there a way to force it to not use proxy ?

[#4]

Well, not sure, if curl from cli works, it should work. Can you try to restart ambari-server and all ambari-agents.

[#5]

Started server and agent but no luck. How can we ask ambari to not use proxy ? Which user is this process run as ? May be I can sudo as that and update its no_proxy list ?

[#6]

Can we please confirm which user runs the service installation ? root ? I confirmed that root can run curl from cli successfully.

I created app-logs dir manually and ran

root:/home/ubuntu# curl -iv 'http://xxxx:50070/webhdfs/v1/app-logs?op=GETFILESTATUS&user.name=hdfs'
*   Trying 1.70.26.132... connected
> GET /webhdfs/v1/app-logs?op=GETFILESTATUS&user.name=hdfs HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Cache-Control: no-cache
Cache-Control: no-cache

[#7 · Accepted solution]

I had the same issues and wanted to update with what I debugged and how I fixed it. I pulled back the environment that the user was running through the ambari-sudo script. The quick test is

/var/lib/ambari-agent/ambari-sudo.sh su hdfs -l -s /bin/bash -c 'env'

That showed me that there were http_proxy values being set which I found out was because of the default env variables that are being source from /etc/profile.d/ directory. For my fix, I just added the local machines to no_proxy and the install went forward. Hope that helps

[#8]

Problem accessing /webhdfs/v1/ats/done. Reason