Created on 12-14-2018 05:11 AM
Sometimes it is desired to see what kind of Http request ambari server makes to the outside works for troubleshooting purpose and to know the value of various http headers (like: User-Agent, Accept, Last-Modified, Content-Type, Content-Length, Connection, ETag, Server, X-Cache ...etc) which ambari-server uses to make calls to Ambari Public repo ...etc.
So in order to achieve the Http Logging of ambari server made http requests we can do the following:
Step-1). Create a file "/etc/ambari-server/conf/http.logging.properties" with the following content:
.level=INFO handlers=java.util.logging.ConsoleHandler java.util.logging.ConsoleHandler.level=ALL sun.net.www.protocol.level=ALL sun.net.www.protocol.http.HttpURLConnection.level=ALL debug.level=ALL
.
Step-2). Inside the "/var/lib/ambari-server/ambari-env.sh" file variable "AMBARI_JVM_ARGS" make sure to add the following parameter "-Djava.util.logging.config.file=/etc/ambari-server/conf/http.logging.properties" to enable the http logging.
Example:
export AMBARI_JVM_ARGS="$AMBARI_JVM_ARGS -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=$ROOT/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Dcom.sun.jndi.ldap.connect.pool.protocol=\"plain ssl\" -Dcom.sun.jndi.ldap.connect.pool.maxsize=20 -Dcom.sun.jndi.ldap.connect.pool.timeout=300000 -Djava.util.logging.config.file=/etc/ambari-server/conf/http.logging.properties"
.
Step-3). Restart Ambari Server.
# ambari-server restart
.
Step-4). Now check the "/var/log/ambari-server/ambari-server.out" file to see the http logging.
Example Output:
Dec 14, 2018 4:52:29 AM sun.net.www.protocol.http.HttpURLConnection plainConnect0 FINEST: ProxySelector Request for http://public-repo-1.hortonworks.com/HDP/ubuntu14/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml Dec 14, 2018 4:52:29 AM sun.net.www.http.HttpClient logFinest FINEST: KeepAlive stream retrieved from the cache, sun.net.www.http.HttpClient(http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml) Dec 14, 2018 4:52:29 AM sun.net.www.protocol.http.HttpURLConnection plainConnect0 FINEST: Proxy used: DIRECT Dec 14, 2018 4:52:29 AM sun.net.www.protocol.http.HttpURLConnection writeRequests FINE: sun.net.www.MessageHeader@5b3d7d395 pairs: {GET /HDP/ubuntu14/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml HTTP/1.1: null}{User-Agent: Java/1.8.0_112}{Host: public-repo-1.hortonworks.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive} Dec 14, 2018 4:52:29 AM sun.net.www.http.HttpClient logFinest FINEST: KeepAlive stream used: http://public-repo-1.hortonworks.com/HDP/ubuntu14/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml Dec 14, 2018 4:52:29 AM sun.net.www.protocol.http.HttpURLConnection getInputStream0 FINE: sun.net.www.MessageHeader@7854dac13 pairs: {null: HTTP/1.1 200 OK}{Content-Type: application/xml}{Content-Length: 2597}{Connection: keep-alive}{Last-Modified: Thu, 12 Jul 2018 23:45:36 GMT}{Accept-Ranges: bytes}{Server: AmazonS3}{Date: Thu, 13 Dec 2018 12:28:28 GMT}{ETag: "abcdefgh42ffdc96d0ab1de61e0dc36cd3"}{Age: 59042}{X-Cache: Hit from cloudfront}{Via: 1.1 frontend_cloud.example.net (CloudFront)}{X-Amz-Cf-Id: Sn-ABCDEFGH8xvcOmkqoeHg_UCYmwyRU9tgDonWfAd4TMXoPdMabcdefgh==} Dec 14, 2018 4:52:29 AM sun.net.www.protocol.http.HttpURLConnection plainConnect0 FINEST: ProxySelector Request for http://public-repo-1.hortonworks.com/HDP/centos7-ppc/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml Dec 14, 2018 4:52:29 AM sun.net.www.http.HttpClient logFinest FINEST: KeepAlive stream retrieved from the cache, sun.net.www.http.HttpClient(http://public-repo-1.hortonworks.com/HDP/ubuntu14/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml) Dec 14, 2018 4:52:29 AM sun.net.www.protocol.http.HttpURLConnection plainConnect0 FINEST: Proxy used: DIRECT Dec 14, 2018 4:52:29 AM sun.net.www.protocol.http.HttpURLConnection writeRequests FINE: sun.net.www.MessageHeader@32b9674a5 pairs: {GET /HDP/centos7-ppc/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml HTTP/1.1: null}{User-Agent: Java/1.8.0_112}{Host: public-repo-1.hortonworks.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive} Dec 14, 2018 4:52:29 AM sun.net.www.http.HttpClient logFinest FINEST: KeepAlive stream used: http://public-repo-1.hortonworks.com/HDP/centos7-ppc/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml Dec 14, 2018 4:52:29 AM sun.net.www.protocol.http.HttpURLConnection getInputStream0 FINE: sun.net.www.MessageHeader@606955ef13 pairs: {null: HTTP/1.1 200 OK}{Content-Type: application/xml}{Content-Length: 2609}{Connection: keep-alive}{Date: Mon, 10 Dec 2018 09:44:35 GMT}{Last-Modified: Fri, 13 Jul 2018 04:43:55 GMT}{ETag: "abcdefgh42ffdc96d0ab1de61e0dc36cd3"}{Accept-Ranges: bytes}{Server: AmazonS3}{Age: 59042}{X-Cache: Hit from cloudfront}{Via: 1.1 frontend_cloud.example.net (CloudFront)}{X-Amz-Cf-Id: ABCDEFGHb3LeLucDbVt5BgHNJmhZRjQPUYPMZS7zHO7oqR1Kabcdefgh==}
.