Many times we see some repeated logging inside our log files. For example in case of ambari-server.log we see the following kind of repeated logging inside the log. WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.ambari.server.api.services.StacksService.getStackArtifacts(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String), should not consume any entity. We might see the above kind of warning messages repeated many times. # grep 'public javax.ws.rs.core.Response org.apache.ambari.server.api.services.RequestService.getRequests' /var/log/ambari-server/ambari-server.log 150 - These are actually harmless WARNING messages, but many times it is desired to make sure that they are not logged, That way we can save some disk space issues and have a clean log. - Every time it is not possible to change the rootLogger to "ERROR" like following to avoid printing some INFO/WARNING messages, Because it will cause suppressing other useful INFO/WARNING messages not t be logged. log4j.rootLogger=ERROR,file - In order to avoid logging of few specific log entries based on the Strings irrespective of the various different logging level (INFO/WARNING/ERROR/DEBUG) those entries are coming from. - In this case suppose, if we do not want to log the line which has "public javax.ws.rs.core.Response" entry in it at any place then we can make use of StringMatchFilter feature of log4j as following: . Step-1). Edit the "/etc/ambari-serevr/conf/log4j.properties" and add the following 3 lines in it Just below to the "file" log appender. log4j.appender.file.filter.01=org.apache.log4j.varia.StringMatchFilter log4j.appender.file.filter.01.StringToMatch=public javax.ws.rs.core.Response log4j.appender.file.filter.01.AcceptOnMatch=false Now the log4j.properties audit log appender will look like following: # Direct log messages to a log file log4j.appender.file=org.apache.log4j.RollingFileAppender log4j.appender.file.File=${ambari.log.dir}/${ambari.log.file} log4j.appender.file.MaxFileSize=80MB log4j.appender.file.MaxBackupIndex=60 log4j.appender.file.layout=org.apache.log4j.PatternLayout log4j.appender.file.layout.ConversionPattern=%d{DATE} %5p [%t] %c{1}:%L - %m%n log4j.appender.file.filter.01=org.apache.log4j.varia.StringMatchFilter log4j.appender.file.filter.01.StringToMatch=public javax.ws.rs.core.Response log4j.appender.file.filter.01.AcceptOnMatch=false NOTE: we can use as many filters we want. We will only need to change the filter number like "log4j.appender.file.filter.01", "log4j.appender.file.filter.02", "log4j.appender.file.filter.03" with different "StringToMatch" values. Step-2). Move the OLD ambari-server logs and restart the ambari-server # mv /var/log/ambari-server /var/log/ambari-server_OLD # ambari-server restart . Step-3). Put the ambari-server.log in tail and then restart ambari server to see if the following line entry is gone from the ambari-server.log now and you should not see those lines again. # grep 'public javax.ws.rs.core.Response org.apache.ambari.server.api.services.RequestService.getRequests' /var/log/ambari-server/ambari-server.log . ... View more

In order to setup HTTPs access for LogSearch Portal UI first of all we will need to create the keystore. We will also create truststore as well using the "$JAVA_HOME/bin/keytool" utility. In this example we will be using the self signed certificates. # mkdir -p /etc/security/serverKeys/ # keytool -genkey -v -alias logsearch -keyalg RSA -keysize 1024 -keystore /etc/security/serverKeys/logsearch.keyStore.jks -validity 3650 -keypass logsearch -storepass logsearch -dname 'CN=erie1.example.com, OU=hwx, O=test, L=Pune, S=Maharashtra, C=IN' # keytool -genkeypair -alias logsearch -keyalg RSA -keysize 2048 -keypass logsearch -storepass logsearch -validity 3650 -keystore /etc/security/serverKeys/logsearch.trustStore.jks -dname 'CN=erie1.example.com, OU=hwx, O=test, L=Pune, S=Maharashtra, C=IN' -rfc . - In the above example i used CN=erie1.example.com as my LogSearch service is running on the same host. - Now we will need to make sure that those files has proper read permissions and ownership set correctly. # chown -R logsearch:hadoop /etc/security/serverKeys/ # ls -la /etc/security/serverKeys/ -rw-r--r--. 1 logsearch hadoop 1399 Feb 24 07:19 logsearch.keyStore.jks -rw-r--r--. 1 logsearch hadoop 2245 Feb 24 07:38 logsearch.trustStore.jks . - Now we can login to ambari UI and navigate to "Log Search" --> "Configs" --> "Advanced" --> "Advanced logsearch-env" and then provide the following details there: Log Search UI Protocol: https Log Search UI Port: 61888 Log Search trust store location: /etc/security/serverKeys/logsearch.trustStore.jks Log Search trust store type: JKS Log Search trust store password: logsearch Log Search key store location: /etc/security/serverKeys/logsearch.keyStore.jks Log Search key store type: logsearch Log Search key store password: logsearch . - Once we enter the above details then we can "Save" the settings and then restart the "Log Search" service. - Now we should be able to access the "Log Search" using the HTTPs protocol . And then the Log Search UI dashboard: . Troubleshooting Common Issues: If we encounter any issue while while running starting up the Log Search after enabling the https then we can look at the file "/var/log/ambari-logsearch-portal/logsearch/logsearch.err", For example if we enter incorrect credentials of store then we might see the following kind of error in our logs: [main] WARN org.eclipse.jetty.util.component.AbstractLifeCycle (AbstractLifeCycle.java:212) - FAILED ServerConnector@1cb37ee4{SSL-http/1.1}{0.0.0.0:61888}: java.io.IOException: Keystore was tampered with, or password was incorrect java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:225) at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) at java.security.KeyStore.load(KeyStore.java:1445) at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:55) at org.eclipse.jetty.util.ssl.SslContextFactory.loadTrustStore(SslContextFactory.java:884) at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:274) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:64) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:256) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.server.Server.doStart(Server.java:366) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.ambari.logsearch.LogSearch.run(LogSearch.java:89) at org.apache.ambari.logsearch.LogSearch.main(LogSearch.java:73) Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778) ... 22 more . ... View more

In this article we will talk about one of the best feature of Ambari 2.4 in which we will be doing the ambari "setup-security" in non interactive mode to enable the HTTPs in ambari server. I assume that you have alredy created the ambari-server keys and certificate using open ssl. So we will see how can we do the HTTPs setup using a single line command: ambari-server setup-security --security-option=setup-https --api-ssl=true --api-ssl-port=8443 --import-cert-path=/etc/ambari-server/certs/sandbox.hortonworks.com.crt --import-key-path=/etc/ambari-server/certs/sandbox.hortonworks.com.key --pem-password=hadoop Output of the above command can be as following, Notice that it will run in non interactive mode. Using python /usr/bin/python Security setup options... Do you want to configure HTTPS [y/n] (y)? SSL port [8443] ? Please enter password for Private Key: Importing and saving Certificate...done. Ambari server URL changed. To make use of the Tez View in Ambari please update the property tez.tez-ui.history-url.base in tez-site Adjusting ambari-server permissions and ownership... NOTE: Restart Ambari Server to apply changes ("ambari-server restart|stop+start") . Now in order to see the configuration changes we will need to restart ambari server. ambari-server restart Now we can simply access ambari server using the HTTPs port 8443 as following: https://localhost:8443/#/main/dashboard/metrics . - This feature in ambari is added as part of JIRA: Ability to automate setup-security and setup-ldap/sync-ldap: https://issues.apache.org/jira/browse/AMBARI-14627 - We can see similar examples there (there might be little changes in the argument names though) so use the "ambari-server --help" to explore those arguments. Examples from AMBARI-14627, I have not tested the following options so you might see some small argumant changes there in the actual implementations: 1.) LDAP setup: ambari-server setup-ldap --ldap-url="ldap.apache.org389" --ldap-secondary-url="" --ldap-ssl="false" --ldap-user-class="person" --ldap-user-attr="sAMAccountName" --ldap-group-class="group" --ldap-group-attr="cn" --ldap-member-attr="member" --ldap-dn="distunguishedName" --ldap-base-dn="dc=ambari01,dc=local" --ldap-referral="" --ldap-bind-anonym=false --ldap-manager-dn="cn=hdfs,ou=ambari,dc=ambari01,dc=local" --ldap-manager-password="myldappassword" --ldap-save-settings --truststore-type="jks" --truststore-path="/var/lib/ambari-server/keys/jkskeystore.jks" --truststore-password="mypass" 2.) Ldap sync: ambari-server sync-ldap --groups=groups.txt --ldap-sync-admin-name=admin --ldap-sync-admin-password=admin 3.) Setup Https: ambari-server setup-security \ --security-option=setup-https --api-ssl=true --client-api-ssl-port=8443 \ --import-cert-path=/var/lib/ambari-server/keys/my.crt \ --import-key-path=/var/lib/ambari-server/keys/my.key --pem-password=password 4.) Encrypt passwords: ambari-server setup-security --security-option=encrypt-passwords --master-key=masterkey --master-key-persist=true 5.) Setup Kerberos JAAS: ambari-server setup-security --security-option=setup-kerberos-jaas --jaas-principal="ambari@EXAMPLE.COM" --jaas-keytab="/etc/security/keytabs/ambari.keytab" 6.) Setup TrustStore: ambari-server setup-security --security-option=setup-truststore \ --truststore-path=/var/lib/ambari-server/keys/keystore.p12 --truststore-type=pkcs12 \ --truststore-password=password --truststore-reconfigure 7.) Import certificate to TrustStore: ambari-server setup-security \ --security-option=import-certificate \ --truststore-path=/var/lib/ambari-server/keys/keystore.p12 \ --truststore-type=pkcs12 \ --truststore-password=password \ --import-cert-path=/var/lib/ambari-server/my.crt \ --import-cert-alias=myalias \ --truststore-reconfigure . . ... View more

- We can see that ambari has it's image files and web contents present inside the "/usr/lib/ambari-server/web/" directory. This directory contains all the static stuff that are needed by the UI. . - Suppose we want to make changes to ambari UI logo. That can be accessed from the URL: http://localhost:8080/img/logo.png http://localhost:8080/img/logo-white.png - In this example we will try changing the "logo-white.png". So in order to do that we will need to get our own logo like http://test.example.com/jboss/wp-content/uploads/2015/09/MM-Banner-logo.png Now we want to use the above image as ambari "logo-white.png". So in order to do that in need to do the following: # mkdir /tmp/images # cd /tmp/images # wget http://test.example.com/jboss/wp-content/uploads/2015/09/MM-Banner-logo.png # mv MM-Banner-logo.png logo-white.png # gzip logo-white.png - We have converted out images in compressed format. We can see the file as following, which we will need to move inside the "/usr/lib/ambari-server/web/img" directory. # ls -l /tmp/images/logo-white.png.gz -rw-r--r-- 1 root root 41532 Nov 13 05:46 ./logo-white.png.gz # cp /tmp/images/logo-white.png.gz /usr/lib/ambari-server/web/img/ mv: overwrite `/usr/lib/ambari-server/web/img/logo-white.png.gz'? y . Now we should be able to open the ambari UI after refreshing the browser. Refresh the browser (make sure to clear the old cache data from browser) Or open ambari UI in (Google chrome menu "File --> New Incognito Window")   . Notice: The top left corner of the page that ambari UI where the logo is changed. Same way we can also make changes in the Style sheets (css) as well. . . ... View more