In order to setup HTTPs access for LogSearch Portal UI first of all we will need to create the keystore. We will also create truststore as well using the "$JAVA_HOME/bin/keytool" utility. In this example we will be using the self signed certificates. # mkdir -p /etc/security/serverKeys/ # keytool -genkey -v -alias logsearch -keyalg RSA -keysize 1024 -keystore /etc/security/serverKeys/logsearch.keyStore.jks -validity 3650 -keypass logsearch -storepass logsearch -dname 'CN=erie1.example.com, OU=hwx, O=test, L=Pune, S=Maharashtra, C=IN' # keytool -genkeypair -alias logsearch -keyalg RSA -keysize 2048 -keypass logsearch -storepass logsearch -validity 3650 -keystore /etc/security/serverKeys/logsearch.trustStore.jks -dname 'CN=erie1.example.com, OU=hwx, O=test, L=Pune, S=Maharashtra, C=IN' -rfc . - In the above example i used CN=erie1.example.com as my LogSearch service is running on the same host. - Now we will need to make sure that those files has proper read permissions and ownership set correctly. # chown -R logsearch:hadoop /etc/security/serverKeys/ # ls -la /etc/security/serverKeys/ -rw-r--r--. 1 logsearch hadoop 1399 Feb 24 07:19 logsearch.keyStore.jks -rw-r--r--. 1 logsearch hadoop 2245 Feb 24 07:38 logsearch.trustStore.jks . - Now we can login to ambari UI and navigate to "Log Search" --> "Configs" --> "Advanced" --> "Advanced logsearch-env" and then provide the following details there: Log Search UI Protocol: https Log Search UI Port: 61888 Log Search trust store location: /etc/security/serverKeys/logsearch.trustStore.jks Log Search trust store type: JKS Log Search trust store password: logsearch Log Search key store location: /etc/security/serverKeys/logsearch.keyStore.jks Log Search key store type: logsearch Log Search key store password: logsearch . - Once we enter the above details then we can "Save" the settings and then restart the "Log Search" service. - Now we should be able to access the "Log Search" using the HTTPs protocol . And then the Log Search UI dashboard: . Troubleshooting Common Issues: If we encounter any issue while while running starting up the Log Search after enabling the https then we can look at the file "/var/log/ambari-logsearch-portal/logsearch/logsearch.err", For example if we enter incorrect credentials of store then we might see the following kind of error in our logs: [main] WARN org.eclipse.jetty.util.component.AbstractLifeCycle (AbstractLifeCycle.java:212) - FAILED ServerConnector@1cb37ee4{SSL-http/1.1}{0.0.0.0:61888}: java.io.IOException: Keystore was tampered with, or password was incorrect java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:225) at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) at java.security.KeyStore.load(KeyStore.java:1445) at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:55) at org.eclipse.jetty.util.ssl.SslContextFactory.loadTrustStore(SslContextFactory.java:884) at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:274) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:64) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:256) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.server.Server.doStart(Server.java:366) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.ambari.logsearch.LogSearch.run(LogSearch.java:89) at org.apache.ambari.logsearch.LogSearch.main(LogSearch.java:73) Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778) ... 22 more . ... View more

In this article we will talk about one of the best feature of Ambari 2.4 in which we will be doing the ambari "setup-security" in non interactive mode to enable the HTTPs in ambari server. I assume that you have alredy created the ambari-server keys and certificate using open ssl. So we will see how can we do the HTTPs setup using a single line command: ambari-server setup-security --security-option=setup-https --api-ssl=true --api-ssl-port=8443 --import-cert-path=/etc/ambari-server/certs/sandbox.hortonworks.com.crt --import-key-path=/etc/ambari-server/certs/sandbox.hortonworks.com.key --pem-password=hadoop Output of the above command can be as following, Notice that it will run in non interactive mode. Using python /usr/bin/python Security setup options... Do you want to configure HTTPS [y/n] (y)? SSL port [8443] ? Please enter password for Private Key: Importing and saving Certificate...done. Ambari server URL changed. To make use of the Tez View in Ambari please update the property tez.tez-ui.history-url.base in tez-site Adjusting ambari-server permissions and ownership... NOTE: Restart Ambari Server to apply changes ("ambari-server restart|stop+start") . Now in order to see the configuration changes we will need to restart ambari server. ambari-server restart Now we can simply access ambari server using the HTTPs port 8443 as following: https://localhost:8443/#/main/dashboard/metrics . - This feature in ambari is added as part of JIRA: Ability to automate setup-security and setup-ldap/sync-ldap: https://issues.apache.org/jira/browse/AMBARI-14627 - We can see similar examples there (there might be little changes in the argument names though) so use the "ambari-server --help" to explore those arguments. Examples from AMBARI-14627, I have not tested the following options so you might see some small argumant changes there in the actual implementations: 1.) LDAP setup: ambari-server setup-ldap --ldap-url="ldap.apache.org389" --ldap-secondary-url="" --ldap-ssl="false" --ldap-user-class="person" --ldap-user-attr="sAMAccountName" --ldap-group-class="group" --ldap-group-attr="cn" --ldap-member-attr="member" --ldap-dn="distunguishedName" --ldap-base-dn="dc=ambari01,dc=local" --ldap-referral="" --ldap-bind-anonym=false --ldap-manager-dn="cn=hdfs,ou=ambari,dc=ambari01,dc=local" --ldap-manager-password="myldappassword" --ldap-save-settings --truststore-type="jks" --truststore-path="/var/lib/ambari-server/keys/jkskeystore.jks" --truststore-password="mypass" 2.) Ldap sync: ambari-server sync-ldap --groups=groups.txt --ldap-sync-admin-name=admin --ldap-sync-admin-password=admin 3.) Setup Https: ambari-server setup-security \ --security-option=setup-https --api-ssl=true --client-api-ssl-port=8443 \ --import-cert-path=/var/lib/ambari-server/keys/my.crt \ --import-key-path=/var/lib/ambari-server/keys/my.key --pem-password=password 4.) Encrypt passwords: ambari-server setup-security --security-option=encrypt-passwords --master-key=masterkey --master-key-persist=true 5.) Setup Kerberos JAAS: ambari-server setup-security --security-option=setup-kerberos-jaas --jaas-principal="ambari@EXAMPLE.COM" --jaas-keytab="/etc/security/keytabs/ambari.keytab" 6.) Setup TrustStore: ambari-server setup-security --security-option=setup-truststore \ --truststore-path=/var/lib/ambari-server/keys/keystore.p12 --truststore-type=pkcs12 \ --truststore-password=password --truststore-reconfigure 7.) Import certificate to TrustStore: ambari-server setup-security \ --security-option=import-certificate \ --truststore-path=/var/lib/ambari-server/keys/keystore.p12 \ --truststore-type=pkcs12 \ --truststore-password=password \ --import-cert-path=/var/lib/ambari-server/my.crt \ --import-cert-alias=myalias \ --truststore-reconfigure . . ... View more

- We can see that ambari has it's image files and web contents present inside the "/usr/lib/ambari-server/web/" directory. This directory contains all the static stuff that are needed by the UI. . - Suppose we want to make changes to ambari UI logo. That can be accessed from the URL: http://localhost:8080/img/logo.png http://localhost:8080/img/logo-white.png - In this example we will try changing the "logo-white.png". So in order to do that we will need to get our own logo like http://test.example.com/jboss/wp-content/uploads/2015/09/MM-Banner-logo.png Now we want to use the above image as ambari "logo-white.png". So in order to do that in need to do the following: # mkdir /tmp/images # cd /tmp/images # wget http://test.example.com/jboss/wp-content/uploads/2015/09/MM-Banner-logo.png # mv MM-Banner-logo.png logo-white.png # gzip logo-white.png - We have converted out images in compressed format. We can see the file as following, which we will need to move inside the "/usr/lib/ambari-server/web/img" directory. # ls -l /tmp/images/logo-white.png.gz -rw-r--r-- 1 root root 41532 Nov 13 05:46 ./logo-white.png.gz # cp /tmp/images/logo-white.png.gz /usr/lib/ambari-server/web/img/ mv: overwrite `/usr/lib/ambari-server/web/img/logo-white.png.gz'? y . Now we should be able to open the ambari UI after refreshing the browser. Refresh the browser (make sure to clear the old cache data from browser) Or open ambari UI in (Google chrome menu "File --> New Incognito Window")   . Notice: The top left corner of the page that ambari UI where the logo is changed. Same way we can also make changes in the Style sheets (css) as well. . . ... View more

- Ambari manages it's Http Session information's using "AMBARISESSIONID". So if we want to logout the logged in user then we will need to find out it's "AMBARISESSIONID" cookie value. We can get it using browser debug tools like for Google chrome we can use Menu -> More Tools --> Developer Tools The we can go to the "Network" tab of the developer tool and then on the right hand side we should see the Cookies and other request headers. Here i have found that the value for the "AMBARISESSIONID" Http Session Cookie is "1t4o6q0ph8j23etnl3e6ig5b3" hence now we can simply make a curl call as following in order to get this user logged out: $ curl -u admin:admin -i -H 'X-Requested-By:ambari' -H "Cookie: AMBARISESSIONID=1t4o6q0ph8j23etnl3e6ig5b3" -X GET http://c6401.ambari.apache.org:8080/api/v1/logout HTTP/1.1 200 OK X-Frame-Options: DENY X-XSS-Protection: 1; mode=block User: admin Content-Type: text/plain Content-Length: 0 Server: Jetty(8.1.19.v20160209) - As soon as the above curl command is executed we will see that the user is logged out from the ambari UI on the browser. . We can also enable the Http Session logging in the ambari server log by editing the file "/etc/ambari-server/conf/log4j.properties" and adding the following line at the end. log4j.logger.org.eclipse.jetty.server.session=DEBUG Now When we will restart ambari server then we can see that the logged in users sessions are printed there: 07 Feb 2017 05:21:20,283 DEBUG [ambari-client-thread-29] session:275 - Got Session ID 1t4o6q0ph8j23etnl3e6ig5b3 from cookie With the above approach we can get all the active Session IDs and can force logout them all. However enabling DEBUG logging will causes excessive log event generation so we need to be careful there. ... View more

While accessing the Zeppelin View is shows the "zeppelin service is not running". But when we check at the back end then we find that the Zeppelin server was running. PID file has the correct PID information mentioned in it. - As we see that Zepplin Notebok is running via ambari as well. But still when we try to access the Zeppelin View it shows failure saying "Zeppelin service is not running". We will need to check the Zeppelin log to see if it shows any error? We see the following kind of error: INFO [2016-12-01 09:42:08,926] ({main} ZeppelinServer.java[setupWebAppContext]:266) - ZeppelinServer Webapp path: /usr/hdp/current/zeppelin-server/webapps INFO [2016-12-01 09:42:09,331] ({main} ZeppelinServer.java[main]:114) - Starting zeppelin server INFO [2016-12-01 09:42:09,333] ({main} Server.java[doStart]:327) - jetty-9.2.15.v20160210 WARN [2016-12-01 09:42:09,367] ({main} WebAppContext.java[doStart]:514) - Failed startup of context o.e.j.w.WebAppContext@69b794e2{/,null,null}{/usr/hdp/current/zeppelin-server/lib/zeppelin-web-0.6.0.2.5.0.0-1133.war} java.lang.IllegalStateException: Failed to delete temp dir /usr/hdp/2.5.0.0-1133/zeppelin/webapps at org.eclipse.jetty.webapp.WebInfConfiguration.configureTempDirectory(WebInfConfiguration.java:372) at org.eclipse.jetty.webapp.WebInfConfiguration.resolveTempDirectory(WebInfConfiguration.java:260) at org.eclipse.jetty.webapp.WebInfConfiguration.preConfigure(WebInfConfiguration.java:69) at org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:468) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:504) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:163) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) at org.eclipse.jetty.server.Server.start(Server.java:387) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) at org.eclipse.jetty.server.Server.doStart(Server.java:354) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.zeppelin.server.ZeppelinServer.main(ZeppelinServer.java:116) . . INFO [2016-12-01 09:42:09,390] ({main} AbstractConnector.java[doStart]:266) - Started ServerConnector@28cab3cc{HTTP/1.1}{0.0.0.0:9995} INFO [2016-12-01 09:42:09,391] ({main} Server.java[doStart]:379) - Started @1094ms . However the port was opened fine but the the Zeppelin WebAppContext was not initialized properly due to the above error. So Zeppeline View was showing "Service check failed" error with message "zeppelin service is not running". In this case check we will need to check if the "/usr/hdp/2.5.0.0-1133/zeppelin/webapps" has proper permission/ownership as "zeppelin:hadoop" the user who is running zeppelin something like following: INCORRECT: # ls -l /usr/hdp/2.5.0.0-1133/zeppelin/web* drwxr-xr-x. 3 root root 4096 Dec 1 09:37 webapps . CORRECT: # ls -l /usr/hdp/2.5.0.0-1133/zeppelin/web* drwxr-xr-x. 10 zeppelin hadoop 4096 Dec 1 09:59 webapp . ... View more

Ambari allows its users to have different configurations for different hosts for different components via configuration groups. Ambari initially assigns all hosts in your cluster to one, default configuration group for each service you install. When using Configuration groups, it enforces configuration properties that allow override, based on installed components for the selected service and group. For more information on this please refer to: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/using_host_config_groups.html Here we will try to make a very small change in "flume-conf" content and will apply it to specific host ("erie3.example.com") where as all the other hosts will be using the default "flume-conf" configuration. We can simply use the following ambari API to list all the config_groups. http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME/config_groups Example: http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups . Step-1). ======== Lets see the current "flume-conf" from ambari which looks something like following and applied to all the Ambari Hosts: # Flume agent config sandbox.sources = eventlog sandbox.channels = file_channel sandbox.sinks = sink_to_hdfs # Define / Configure source sandbox.sources.eventlog.type = exec sandbox.sources.eventlog.command = tail -F /var/log/eventlog-demo.log sandbox.sources.eventlog.restart = true sandbox.sources.eventlog.batchSize = 1000 #sandbox.sources.eventlog.type = seq # HDFS sinks sandbox.sinks.sink_to_hdfs.type = hdfs sandbox.sinks.sink_to_hdfs.hdfs.fileType = DataStream sandbox.sinks.sink_to_hdfs.hdfs.path = /flume/events sandbox.sinks.sink_to_hdfs.hdfs.filePrefix = eventlog sandbox.sinks.sink_to_hdfs.hdfs.fileSuffix = .log sandbox.sinks.sink_to_hdfs.hdfs.batchSize = 1000 # Use a channel which buffers events in memory sandbox.channels.file_channel.type = file sandbox.channels.file_channel.checkpointDir = /var/flume/checkpoint sandbox.channels.file_channel.dataDirs = /var/flume/data # Bind the source and sink to the channel sandbox.sources.eventlog.channels = file_channel sandbox.sinks.sink_to_hdfs.channel = file_channel . Step-2). ======== Now suppose for Host ("erie3.example.com") we want to run the flume with a slightly different properly like [sandbox.sources.eventlog.command = tail -F /var/log/eventlog-demo-new-location.log] So in order to achieve that we will need to create a "config_group" json data which we will need to push to Ambari. Here we will create a file like "/tmp/erie3_flume_conf.json" [ { "ConfigGroup": { "cluster_name": "ErieCluster", "group_name": "cfg_group_test1", "tag": "FLUME", "description": "FLUME configs for Changes", "hosts": [ { "host_name": "erie3.example.com" } ], "desired_configs": [ { "type": "flume-conf", "tag": "nextgen1", "properties": { "content": " # Flume agent config\r\n sandbox.sources = eventlog \r\n sandbox.channels = file_channel \r\n sandbox.sinks = sink_to_hdfs \r\n \r\n # Define / Configure source \r\n sandbox.sources.eventlog.type = exec \r\n sandbox.sources.eventlog.command = tail -F /var/log/eventlog-demo-new-location.log \r\n sandbox.sources.eventlog.restart = true \r\n sandbox.sources.eventlog.batchSize = 1000 \r\n #sandbox.sources.eventlog.type = seq \r\n \r\n # HDFS sinks \r\n sandbox.sinks.sink_to_hdfs.type = hdfs \r\n sandbox.sinks.sink_to_hdfs.hdfs.fileType = DataStream \r\n sandbox.sinks.sink_to_hdfs.hdfs.path = /flume/events \r\n sandbox.sinks.sink_to_hdfs.hdfs.filePrefix = eventlog \r\n sandbox.sinks.sink_to_hdfs.hdfs.fileSuffix = .log \r\n sandbox.sinks.sink_to_hdfs.hdfs.batchSize = 2000 \r\n # Use a channel which buffers events in memory \r\n sandbox.channels.file_channel.type = file \r\n sandbox.channels.file_channel.checkpointDir = /var/flume/checkpoint \r\n sandbox.channels.file_channel.dataDirs = /var/flume/data \r\n # Bind the source and sink to the channel \r\n sandbox.sources.eventlog.channels = file_channel \r\n sandbox.sinks.sink_to_hdfs.channel = file_channel \r\n " } }, { "type": "flume-env", "tag": "nextgen1" } ] } } ] **Notice:** If our configuration contains new line then the json data should use "\r\n" characters sequence. - Also notice that the above JSON configuration is specified for host "erie3.example.com" as following, (However we can have more comma separated hosts there): . "hosts": [ { "host_name": "erie3.example.com" } ] - We have also specified the name for our config group ["group_name": "cfg_group_test1",] . Step-3). ======== Lets now PUT these changes to ambari and see if it works: curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d @/Users/jsensharma/Cases/Articles/Flume_Config_Group/erie3_flume_conf.json http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups OUTPUT: $ curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d @/Users/jsensharma/Cases/Articles/Flume_Config_Group/erie3_flume_conf.json http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups { "resources" : [ { "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups/252", "ConfigGroup" : { "id" : 252 } } ] } . ... View more