How to enable HTTP logging for Ambari Server Outbound connections?

[#1]

Sometimes it is desired to see what kind of Http request ambari server makes to the outside works for troubleshooting purpose and to know the value of various http headers (like: User-Agent, Accept, Last-Modified, Content-Type, Content-Length, Connection, ETag, Server, X-Cache ...etc) which ambari-server uses to make calls to Ambari Public repo ...etc.

So in order to achieve the Http Logging of ambari server made http requests we can do the following:

Step-1). Create a file "/etc/ambari-server/conf/http.logging.properties" with the following content:

.level=INFO
handlers=java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=ALL
sun.net.www.protocol.level=ALL
sun.net.www.protocol.http.HttpURLConnection.level=ALL
debug.level=ALL

.

Step-2). Inside the "/var/lib/ambari-server/ambari-env.sh" file variable "AMBARI_JVM_ARGS" make sure to add the following parameter "-Djava.util.logging.config.file=/etc/ambari-server/conf/http.logging.properties" to enable the http logging.

Example:

export AMBARI_JVM_ARGS="$AMBARI_JVM_ARGS -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=$ROOT/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Dcom.sun.jndi.ldap.connect.pool.protocol=\"plain ssl\" -Dcom.sun.jndi.ldap.connect.pool.maxsize=20 -Dcom.sun.jndi.ldap.connect.pool.timeout=300000  -Djava.util.logging.config.file=/etc/ambari-server/conf/http.logging.properties"

.

Step-3). Restart Ambari Server.

# ambari-server restart

.

Step-4). Now check the "/var/log/ambari-server/ambari-server.out" file to see the http logging.

Example Output:

Dec 14, 2018 4:52:29 AM sun.net.www.protocol.http.HttpURLConnection plainConnect0
FINEST: ProxySelector Request for http://public-repo-1.hortonworks.com/HDP/ubuntu14/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml
Dec 14, 2018 4:52:29 AM sun.net.www.http.HttpClient logFinest
FINEST: KeepAlive stream retrieved from the cache, sun.net.www.http.HttpClient(http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml)
Dec 14, 2018 4:52:29 AM sun.net.www.protocol.http.HttpURLConnection plainConnect0
FINEST: Proxy used: DIRECT
Dec 14, 2018 4:52:29 AM sun.net.www.protocol.http.HttpURLConnection writeRequests
FINE: sun.net.www.MessageHeader@5b3d7d395 pairs: {GET /HDP/ubuntu14/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml HTTP/1.1: null}{User-Agent: Java/1.8.0_112}{Host: public-repo-1.hortonworks.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}
Dec 14, 2018 4:52:29 AM sun.net.www.http.HttpClient logFinest
FINEST: KeepAlive stream used: http://public-repo-1.hortonworks.com/HDP/ubuntu14/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml
Dec 14, 2018 4:52:29 AM sun.net.www.protocol.http.HttpURLConnection getInputStream0
FINE: sun.net.www.MessageHeader@7854dac13 pairs: {null: HTTP/1.1 200 OK}{Content-Type: application/xml}{Content-Length: 2597}{Connection: keep-alive}{Last-Modified: Thu, 12 Jul 2018 23:45:36 GMT}{Accept-Ranges: bytes}{Server: AmazonS3}{Date: Thu, 13 Dec 2018 12:28:28 GMT}{ETag: "abcdefgh42ffdc96d0ab1de61e0dc36cd3"}{Age: 59042}{X-Cache: Hit from cloudfront}{Via: 1.1 frontend_cloud.example.net (CloudFront)}{X-Amz-Cf-Id: Sn-ABCDEFGH8xvcOmkqoeHg_UCYmwyRU9tgDonWfAd4TMXoPdMabcdefgh==}
Dec 14, 2018 4:52:29 AM sun.net.www.protocol.http.HttpURLConnection plainConnect0
FINEST: ProxySelector Request for http://public-repo-1.hortonworks.com/HDP/centos7-ppc/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml
Dec 14, 2018 4:52:29 AM sun.net.www.http.HttpClient logFinest
FINEST: KeepAlive stream retrieved from the cache, sun.net.www.http.HttpClient(http://public-repo-1.hortonworks.com/HDP/ubuntu14/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml)
Dec 14, 2018 4:52:29 AM sun.net.www.protocol.http.HttpURLConnection plainConnect0
FINEST: Proxy used: DIRECT
Dec 14, 2018 4:52:29 AM sun.net.www.protocol.http.HttpURLConnection writeRequests
FINE: sun.net.www.MessageHeader@32b9674a5 pairs: {GET /HDP/centos7-ppc/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml HTTP/1.1: null}{User-Agent: Java/1.8.0_112}{Host: public-repo-1.hortonworks.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}
Dec 14, 2018 4:52:29 AM sun.net.www.http.HttpClient logFinest
FINEST: KeepAlive stream used: http://public-repo-1.hortonworks.com/HDP/centos7-ppc/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml
Dec 14, 2018 4:52:29 AM sun.net.www.protocol.http.HttpURLConnection getInputStream0
FINE: sun.net.www.MessageHeader@606955ef13 pairs: {null: HTTP/1.1 200 OK}{Content-Type: application/xml}{Content-Length: 2609}{Connection: keep-alive}{Date: Mon, 10 Dec 2018 09:44:35 GMT}{Last-Modified: Fri, 13 Jul 2018 04:43:55 GMT}{ETag: "abcdefgh42ffdc96d0ab1de61e0dc36cd3"}{Accept-Ranges: bytes}{Server: AmazonS3}{Age: 59042}{X-Cache: Hit from cloudfront}{Via: 1.1 frontend_cloud.example.net (CloudFront)}{X-Amz-Cf-Id: ABCDEFGHb3LeLucDbVt5BgHNJmhZRjQPUYPMZS7zHO7oqR1Kabcdefgh==}

.