Microservice applications prefer OpenID SSO by existing IDP, for example Keycloak. When these application access Hadoop Services with JWT, Knox should be able to verify and proxy to internal services. This article describe how to configure Knox JWT Provider and customize for your IDP requirement. There are two options. KnoxSSO direct integrate with IDP KnoxSSO, Knox Single Sign On service, integrates with IDP, for example Keycloak. After authenticated by IDP, Knox signs a JWT token for all microservices and Hadoop RESTful services. This solution is complex to configure and Knox could become the performance bottleneck. Knox JWTProvider accept jwt signed by other IDP Microservices still use current IDP for single sign on, and configure an extra Knox topology to accept other IDP signed JWT token. The second solution has less impact on existing architecture and Knox is only used when access Hadoop services. Here we will focus on this solution, and discuss how to configure Knox and what is the limitation. Add a JWTProvider Knox topology JWTProvider is explained https://knox.apache.org/books/knox-0-12-0/user-guide.html#JWT+Provider Current Ambari can't edit extra Knox topology, have to add it in command line. # ssh Knox node $ sudo su - knox # Add the new topology named jwt $ vim jwt.xml <topology> <gateway> <provider> <role>identity-assertion</role> <name>Default</name> <enabled>true</enabled> </provider> <provider> <role>authorization</role> <name>XASecurePDPKnox</name> <enabled>true</enabled> </provider> <provider> <role>federation</role> <name>JWTProvider</name> <enabled>true</enabled> <param> <!-- knox.token.audiences is optional --> <name>knox.token.audiences</name> <value>tokenbased</value> </param> </provider> </gateway> <!-- Add Hadoop Services allowed jwt access, here use Yarn UI as an example --> <service> <role>YARNUI</role> <url>http://{MASTER_NODE_1}:8088</url> <url>http://{MASTER_NODE_2}:8088</url> </service> <service> <role>RESOURCEMANAGER</role> <url>http://{MASTER_NODE_1}:8088/ws</url> <url>http://{MASTER_NODE_2}:8088/ws</url> </service> </topology> Add Knox Token Service in knoxsso toplogy for test <service> <role>KNOXTOKEN</role> <param> <name>knox.token.ttl</name> <value>600000</value> </param> <!-- knox.token.audiences is optional, must meet what's configured in JWTProvider --> <param> <name>knox.token.audiences</name> <value>tokenbased</value> </param> </service> Test Knox jwt toplogy Enable Knox demo ldap as knoxsso source accounts. Get a knoxsso jwt token. $ curl -ivku guest:guest-password https://{KNOX_NODE}:8443/gateway/knoxsso/knoxtoken/api/v1/token {"access_token":"eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJndWVzdCIsImF1ZCI6InRva2VuYmFzZWQiLCJpc3MiOiJLTk9YU1NPIiwiZXhwIjoxNTI4MjgxODQ3fQ.i2Y3MMGbTW9M-wHStL7TuwjmL_rYmTGRjN_7QK0KB8EfLxKJzL2zRFEU8USxyFAchDJ-3vDdLaU8UPsTCVufo9UT5p8ywSlBgulFsOzIYuq-YVIqATpJZVZIJWCnoGHjXuTZHXeRreyjAs6cFsiiqsDwL8rCxnAmtBQeoX9fsAI","token_type":"Bearer ","expires_in":1528281847804} Use this token to access Yarn UI $ curl -ivk -H "Authorization: Bearer eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJndWVzdCIsImF1ZCI6InRva2VuYmFzZWQiLCJpc3MiOiJLTk9YU1NPIiwiZXhwIjoxNTI4MjgxODQ3fQ.i2Y3MMGbTW9M-wHStL7TuwjmL_rYmTGRjN_7QK0KB8EfLxKJzL2zRFEU8USxyFAchDJ-3vDdLaU8UPsTCVufo9UT5p8ywSlBgulFsOzIYuq-YVIqATpJZVZIJWCnoGHjXuTZHXeRreyjAs6cFsiiqsDwL8rCxnAmtBQeoX9fsAI" https://hdp-e.field.hortonworks.com:8443/gateway/jwt/yarn/ admin should be able to view the UI html contents, and guest would be blocked. Also can find the entries in Ranger Audit. Decode knoxsso jwt token Decode the jwt https://jwt.io/ HEADER { "alg": "RS256" } PAYLOAD { "sub": "guest", "aud": "tokenbased", "iss": "KNOXSSO", "exp": 1528281847 } Microservice Application JWT Verification Synchronise certificates between Knox and IDP IDP use private certificate to sign JWT, and Knox use public certificate to verify the JWT. The certificate pair must match for the verification. In current Knox version, public key is not supported, have to configure public certificate as following example. Knox only supports keypair JWT signature algorithms, RS256, RS384, RS512, PS256, PS384, PS512. RS256 is default and not configurable in HDP2.6. Get public certificate from IDP. $ cat knox-pub.pem -----BEGIN CERTIFICATE----- MIIENTCCAx2gAwIBAgIJAP4/owzmw1t4MA0GCSqGSIb3DQEBCwUAMIGlMQswCQYD VQQGEwJVSzEABC0GA1UECAwGTG9uZG9uMQ8wDQYDVQQHDAZMb25kb24xFDASBgNV BAoMC0hvbWUgT2ZmaWNlMUQwQgYDVQQDDDtwcmltYXJ5LXNlY3VyaXR5MC5ub25w cm9kZi51ay5zZHAuZGlnaXRhbC5ob21lb2ZmaWNlLmdvdi51azEYMBYGA1UdEQwP RE5TLjE9bG9jYWxob3N0MB4XDTE4MDUzMTE2MTA0MFoXDTI4MDUyODE2MTA0MFow gaUxCzAJBgNVBAYTAlVLMQ8wDQYDVQQIDAZMb25kb24xDzANBgNVBAcMBkxvbmRv bjEUMBIGA1UECgwLSG9tZSBPZmZpY2UxRDBCBgNVBAMMO3ByaW1hcnktc2VjdXJp dHkwLm5vbnByb2RmLnVrLnNkcC5kaWdpdGFsLmhvbWVvZmZpY2UuZ292LnVrMRgw FgYDVR0RDA9ETlMuMT1sb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQDNb8qkTY0afOAUauLWbLZfF/1kys6Il/aRwKLaf0m+nTiaYowCGBKB XLbzkXXDNSCOC2b6qPRIul2yF1fd8hAmnfUiVJY2hKbbfhdvbPp1oF9cC6QL+28x ppj+utmyf2YNYpXGCMKHh7wZHNTTt28jqB/+Co8RC2xQgQ6FX6rCmUSB62/VuMHp JBCb4Sf3Y9qgiZKyqtK9B9UiNIo4bJqeyF3Ql3qCFKmXIoni4k/3wcKvw9wNiWPy MQweK0Q7G542K02Q3h7+caRNnZibFSTC/Qvs5jsRrxmkw4vH3npEJGASVr+y9JGq dCjsi+ocrEdN9SMnGlQpWWtuB8LRlsitAgMBAAGjZjBkMBQGA1UdEQQNMAuCCWxv Y2FsaG9zdDAdBgNVHQ4EFgQUPlLDl7837zNPS2Dvn2u7mtqDTZgwHwYDVR0jBBgw FoAUPlLDl7837zNPS2Dvn2u7mtqDTZgwDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0B AQsFAAOCAQEAZa7ZJfc8MzwmYCCmVt9xcGQqFeAxC4saqKEFuS6PJAJqlZIK+D/A y+3AT2qJ84Rs3nWFdnIsEmGzWwLbfve/xyFFAizM8d1tYF5DxXWp+7f2c1Ssah+S t+ua80N9Q2EwdZekQrtnfba58gW5RdTORNGAVjXJjQvHSytwkn1YlRsVQjhvv4Cy R6LBb5Xdd0R9DIqu2mpp0bGvX6hlx0yPJrsiYxd1DsHl+aFdTnQ3OkZVvxe2MfWi yhTIWQoLfHrMwc2l1qjn2c3x4AIRsqLiLkMTfgrgUWC+T2IL1oO5jFBjbeV3ljOY fOfkGmG6TCsdF38qkB/fl869jUGXIBAHjQ== -----END CERTIFICATE----- In Knox JWT topology, configure the verification public certificate. <provider> <role>federation</role> <name>JWTProvider</name> <enabled>true</enabled> <param> <name>knox.token.verification.pem</name> <value> MIIENTCCAx2gAwIBAgIJAP4/owzmw1t4MA0GCSqGSIb3DQEBCwUAMIGlMQswCQYD VQQGEwJVSzEABC0GA1UECAwGTG9uZG9uMQ8wDQYDVQQHDAZMb25kb24xFDASBgNV BAoMC0hvbWUgT2ZmaWNlMUQwQgYDVQQDDDtwcmltYXJ5LXNlY3VyaXR5MC5ub25w cm9kZi51ay5zZHAuZGlnaXRhbC5ob21lb2ZmaWNlLmdvdi51azEYMBYGA1UdEQwP RE5TLjE9bG9jYWxob3N0MB4XDTE4MDUzMTE2MTA0MFoXDTI4MDUyODE2MTA0MFow gaUxCzAJBgNVBAYTAlVLMQ8wDQYDVQQIDAZMb25kb24xDzANBgNVBAcMBkxvbmRv bjEUMBIGA1UECgwLSG9tZSBPZmZpY2UxRDBCBgNVBAMMO3ByaW1hcnktc2VjdXJp dHkwLm5vbnByb2RmLnVrLnNkcC5kaWdpdGFsLmhvbWVvZmZpY2UuZ292LnVrMRgw FgYDVR0RDA9ETlMuMT1sb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQDNb8qkTY0afOAUauLWbLZfF/1kys6Il/aRwKLaf0m+nTiaYowCGBKB XLbzkXXDNSCOC2b6qPRIul2yF1fd8hAmnfUiVJY2hKbbfhdvbPp1oF9cC6QL+28x ppj+utmyf2YNYpXGCMKHh7wZHNTTt28jqB/+Co8RC2xQgQ6FX6rCmUSB62/VuMHp JBCb4Sf3Y9qgiZKyqtK9B9UiNIo4bJqeyF3Ql3qCFKmXIoni4k/3wcKvw9wNiWPy MQweK0Q7G542K02Q3h7+caRNnZibFSTC/Qvs5jsRrxmkw4vH3npEJGASVr+y9JGq dCjsi+ocrEdN9SMnGlQpWWtuB8LRlsitAgMBAAGjZjBkMBQGA1UdEQQNMAuCCWxv Y2FsaG9zdDAdBgNVHQ4EFgQUPlLDl7837zNPS2Dvn2u7mtqDTZgwHwYDVR0jBBgw FoAUPlLDl7837zNPS2Dvn2u7mtqDTZgwDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0B AQsFAAOCAQEAZa7ZJfc8MzwmYCCmVt9xcGQqFeAxC4saqKEFuS6PJAJqlZIK+D/A y+3AT2qJ84Rs3nWFdnIsEmGzWwLbfve/xyFFAizM8d1tYF5DxXWp+7f2c1Ssah+S t+ua80N9Q2EwdZekQrtnfba58gW5RdTORNGAVjXJjQvHSytwkn1YlRsVQjhvv4Cy R6LBb5Xdd0R9DIqu2mpp0bGvX6hlx0yPJrsiYxd1DsHl+aFdTnQ3OkZVvxe2MfWi yhTIWQoLfHrMwc2l1qjn2c3x4AIRsqLiLkMTfgrgUWC+T2IL1oO5jFBjbeV3ljOY fOfkGmG6TCsdF38qkB/fl869jUGXIBAHjQ== </value> </param> </provider> JWT Issuer By default Knox only accept JWT with "iss": "KNOXSSO". Need to verify if IDP, for example Keycloak can customize it. Alternatively, can override this in Knox configuration. Add following parameters in Ambari Knox Custom gateway-site jwt.expected.issuer=CUSTOM_SSO jwt.expected.sigalg=RS256 However this new feature is only available in HDP3.x. Before upgrade to HDP3, have to hard code jwt issuer as "KNOXSSO". User Account ID Current Knox can only parse user account from jwt parameter "sub": "guest". If IDP, for example Ping Federation, uses "client_id", it would be a problem. And the value should be consistent with account id synchronized into Ranger. Normally it is uid in OpenLDAP. JWT audiences This is optional. It must meet what's configured in knox.token.audiences Known Issues Knox json parser issue Before HDP2.6.5, Knox json parser doesn't support complex json raw format with multiple keys. Knox treats it as invalid json format, and throw NullpointException. Workaround JSON Issue For HDP2.6.3, manually replace dependant JSON jar files. SSH into every Knox node. $ cd /usr/hdp/2.6.3.0-235/knox/dep $ sudo wget http://repo1.maven.org/maven2/com/nimbusds/nimbus-jose-jwt/4.41.2/nimbus-jose-jwt-4.41.2.jar $ sudo wget http://repo1.maven.org/maven2/com/jayway/jsonpath/json-path/2.4.0/json-path-2.4.0.jar $ sudo wget http://repo1.maven.org/maven2/net/minidev/json-smart/2.3/json-smart-2.3.jar $ sudo wget http://repo1.maven.org/maven2/net/minidev/asm/1.0.2/asm-1.0.2.jar $ sudo mv json-path-0.9.1.jar json-path-0.9.1.jar.bak $ sudo mv nimbus-jose-jwt-4.11.jar nimbus-jose-jwt-4.11.jar.bak $ sudo mv json-smart-1.2.jar json-smart-1.2.jar.bak Restart Knox from Ambari ... View more

Short Description How to setup popular SQL development tools, like DbVisualizer, to access HDP Hive in secured network zone via Knox. Article Most customers setup Hadoop/Hive cluster in secured network Zone. There's no direct network connection between office and Hadoop network. So Knox is always setup to proxy Hive connections from office network. This article will setup DbVisualizer to connect to Knox Hive URL step by step. Download Hive Driver for HDP2.6.1 Download hive-jdbc.jar https://github.com/timveil/hive-jdbc-uber-jar/releases The latest for HDP2.6.1 is https://github.com/timveil/hive-jdbc-uber-jar/releases/download/v1.6-2.6.1/hive-jdbc-uber-2.6.1.0-129.jar Create a new Database Driver use the hive-jdbc.jar URL Format: jdbc:hive2://<server>:<port10000>/<database> The URL Format is for standard hive2 jdbc, will change later in actual connection for Knox. Create a new Connection with this new Driver Use Database URL, and edit the Database URL as jdbc:hive2://<KNOX_NODE_FQDN>:<port>/;ssl=true;transportMode=http;httpPath=<KNOX_HIVE_HTTPPATH> Default knox hive http path is gateway/default/hive, but please double check with your system admin. Add Knox node certificate or CA certificate into DbVisualizer JVM truststore, otherwise will get SSL Exception during DB connection. Check the Java Home of DbVisualizer. Then add Knox node certificate or CA certificate into the truststore $ sudo keytool -import -alias knox -file wb-e.crt.pem -keystore /Library/InternetPlug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/security/cacerts Enter keystore password:changeit Enjoy DbVisualizer ... View more

Hi, I know we can setup 2 HiveServer2 instances with different protocol, binary and http, for different clients. But how to enable HA in this scenario? My plan is to setup 4 HiveServer2 instances, 2 with binary and HA each other, another 2 with http HA each other. Not sure is it possible, and how to setup this? Cheers, Wendell ... View more

LDAP security options LDAP over SSL Traditionally, LDAP connection is wrapped with SSL/TLS, and via a separate service port, normally 636. The protocol is ldaps://. To be aware, it’s not just over SSL, depends on configuration, can over TLS as well. LDAP with StartTLS Alternatively, StartTLS extended operation enable encrypted connection after the initial unencrypted connection, via the same plain ldap port, normally 389. If you don’t configure properly, it could be over SSL, not always with TLS. This protocol is new, normally is used to upgrade the plain ldap port 389 to be secure. But not all customer have configured it in AD/LDAP server. Most common cases are still use ldaps for secure ldap connection. So this article is focusing on LDAPS. Components support LDAPS Most of HDP/HDF components support LDAPS. Ambari, NiFi, Ranger Usersync, Ranger Admin, Zeppelin, Knox. But Atlas still only supports LDAP. General Configure LDAPS Steps Verify LDAPS certificate To connect to ldaps service, first thing we need to get the AD/LDAPS CA certificate. And should verify it’s the correct certificate. $ openssl s_client -CAfile ${LDAPS_SERVER_CA_CERTIFICATE_FILE} -connect ${LDAPS_SERVER_FQDN}:636 -showcerts -verify 32 For example $ openssl s_client -CAfile ./AD-CA.crt -connect ad1.example.com:636 -showcerts -verify 32 Expect return "Verify return code: 0 (ok)". Otherwise need to check with the AD/LDAP team to find the correct certificate file or check the certificate file format. Test ldapsearch Then we need to use the certificate and ldap bind account to test ldapsearch, to verify we have the correct credential of the binding account. $ LDAPTLS_CACERT=${LDAPS_SERVER_CA_CERTIFICATE_FILE} -H ldaps://${LDAPS_SERVER_FQDN}:636 -D ${LDAP_BIND_USER_ACCOUNT} -b ${BASE_OU} "(&(objectclass=person)(sAMAccountName=*))" For example $ LDAPTLS_CACERT=./AD-CA.crt ldapsearch -H ldaps://ad1.example.com:636 -D "hadoopldapuser@datalake.hadoop.internal" -b "OU=Hadoop,DC=DataLake,DC=Hadoop,DC=Internal" "(&(objectclass=person)(sAMAccountName=*))" Expect return some AD/LDAP accounts. Add the certificate into Truststore $ keytool -import -file ${LDAPS_SERVER_CA_CERTIFICATE_FILE} -alias LDAPS-CA -keystore truststore.jks For example $ keytool -import -file ./AD-CA.crt -alias LDAPS-CA -keystore truststore.jks Configure components truststore and ldaps Components have slightly different configure for the ldaps. But normally need to configure the truststore and truststore password, and base OU, bind account, search filters. Detail configurations will be discussed in other articles. ... View more

When NiFi flow runs in mission critical production environment, customer would concern about how to change NiFi flow logic but don't impact business. One solution was implemented in customer production is as follow steps: 1. Add an extra non-logic processor, for example "UpdateAttribute", between ListenPort and main ProcessGroup 2. Updated NewProcessGroup, add into canvas 3. Stop "UpdateAttribute" process, make sure RunningProcessGroup consume all flowfiles in the previous queue. But client components can still send messages to Listen Port 4. After all queued flowfiles are processed by RunningProcessGroup, move the queue end connection to the NewProcessGroup 5. Start "UPdateAttribute" process and NewProcessGroup 6. At the end, remove the old RunningProcessGroup Follow these steps, the production flow change is transparent for client components, and guarantee no data lost. ... View more

Hi, I'm integrate Atlas with AD. ldap works fine, but ldaps doesn't work. Anyone did this before? Regards ... View more

I Know Ranger Admin can implement HA behind a load balancer. How about Ranger User Sync? Regards, Wendell ... View more

Background After Kerberos HDP/HDF cluster, some native admin UI will return "401 Authentication required". For example, Ambari Infra Solr Admin UI This is because the web UI is configured for SPNEGO (Simple and Protected GSSAPI Negotiation Mechanism) to extend Kerberos to HTTP. Client must have a valid Kerberos ticket and send by browser. Mac Kerberos Client Configuration Kerberos software is installed by default in Mac OS, but need to add configure file to access your KDC server. $ sudo vi /Library/Preferences/edu.mit.Kerberos [domain_realm] .hortonworks.com = "HORTONWORKS.COM" hortonworks.com = "HORTONWORKS.COM" [libdefaults] default_realm = "HORTONWORKS.COM" dns_fallback = "yes" noaddresses = "TRUE" [realms] HORTONWORKS.COM = { admin_server = "ad.hortonworks.com" default_domain = "hortonworks.com" kdc = "ad.hortonworks.com" } Firefox Configuration type about:config search negotiate-auth Option1 Command Line command line to init a kerberos ticket $ kinit hadoopadmin@HORTONWORKS.COM hadoopadmin@HORTONWORKS.COM's password: list the ticket $ klist Option2 Ticket Viewer Go to Ticket Viewer Folder CoreServices Go to the folder: /System/Library/CoreServices/ Find Ticket Viewer Add Identity Then you can see the identity Access UI again ... View more

Background Customer attached bigger disks to expand the data node storage. If one disk physical fail, can also use this solution. Step by Step Decommission HDFS DataNode component on the host It takes hours to finish, depends on your existing data size DataNode is decommissioned Turn the host in maintenance Stop all components on the host Change linux /etc/fstab mount new disks to existing mount points. If possible use uuid rather than disk device. uuid is much stable especially in cloud environment. Manually create the yarn log and local folders in the mount points. Because we don't reprovision the host, yarn won't create create these dirs in your configure, but try to reuse them. # for disk in /hadoop/disk-sd{d..j}/Hadoop > do > mkdir ${disk}/yarn/log && chown yarn:hadoop ${disk}/yarn/log > mkdir ${disk}/yarn/local && chown yarn:hadoop ${disk}/yarn/local > done After change Linux disk mount configuration, start all components on the host Recommission DateNode Turn off maintenance Check hdfs blocks $ hdfs fsck / | egrep -v '^\.+$' | grep -v eplica FSCK started by hdfs (auth:KERBEROS_SSL) from /192.168.141.39 for path / at Tue Nov 29 10:42:34 UTC 20161.............................................................................................Status: HEALTHY Total size: 769817156313 B (Total open files size: 75189484 B) Total dirs: 4934 Total files: 23693 Total symlinks: 0 (Files currently being written: 30) Total blocks (validated): 27536 (avg. block size 27956753 B) (Total open file blocks (not validated): 24) Corrupt blocks: 0 Number of data-nodes: 7 Number of racks: 1FSCK ended at Tue Nov 29 10:42:34 UTC 2016 in 433 millisecondsThe filesystem under path '/' is HEALTHY ... View more

Background When we use NiFi flow to load Adobe ClickStream tsv file into hive, we found around 3% rows are in wrong format or missed. Source Data Quality $ awk -F "\t" '{print NF}' 01-weblive_20161014-150000.tsv | sort | uniq -c | sort 1 154 1 159 1 162 1 164 1 167 1 198 1 201 1 467 2 446 2 449 2 569 6 13 10 3 13 146 13 185 15 151 16 54 18 433 21 432 22 238 23 102 26 2 34 138 179 1 319412 670 After clean the tsv $ awk -F "\t" 'NF == 670' 01-weblive_20161014-150000.tsv >> cleaned.tsv $ awk -F "\t" '{print NF}' cleaned.tsv | sort | uniq -c | sort 319412 670 Still missed a few percent rows. Root Cause and Solution We are using ConvertCSVToAvro and ConvertAvroToORC. The clickstrem tsv files have " in them and the ConvertCSVtoAvro processor uses " as the value for the "CSV quote Character" processor configuration property by default. As a result many tabbed fields end up in the same record. We can get good output by changing this configuration property to another character that is not used in input files anywhere. We used ¥ So when use CSV related processor, double check the contents don't have the quote character. ... View more

Error Ranger Tagsync shows lots of KafkaException in log file, which causes disk space alert in Ambari. Also it used out all of the client port. /var/log/ranger/tagsync/tagsync.log 10 Nov 2016 11 : 46 : 43 ERROR TagSynchronizer [main] - 262 tag-source:atlas initialization failed with javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, b ut the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user kafka.common.KafkaException: fetching topic metadata for topics [Set(ATLAS_ENTITIES)] from broker [ArrayBuffer(BrokerEndPoint( 1001 ,host.domain , 6667 ))] failed at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala: 73 ) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala: 96 ) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala: 67 ) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala: 63 ) Caused by: java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala: 122 ) at kafka.producer.SyncProducer.liftedTree1$ 1 (SyncProducer.scala: 82 ) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$doSend(SyncProducer.scala: 81 ) at kafka.producer.SyncProducer.send(SyncProducer.scala: 126 ) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala: 59 ) ... 3 more Background The Atlas was installed after the HDP2.5.0 was kerberosed. Ambari2.4.1 doesn't create the kerberos principal for Ranger Tagsync, and distributed to the node. Could find the hint from Tagsync log: /var/log/ranger/tagsync/tagsync.log 10 Nov 2016 11 : 46 : 41 WARN SecureClientLogin [main] - 119 /etc/security/keytabs/rangertagsync.service.keytab doesn't exist. 10 Nov 2016 11 : 46 : 41 WARN SecureClientLogin [main] - 130 Can't find principal : rangertagsync/host.domain @REALM Fix Manually create rangertagsync principal and keytab. kadmin.local: add_principal -randkey rangertagsync/ <code>rangertagsync/host.domain @REALM kadmin.local: xst -k rangertagsync.service.keytab rangertagsync/<code>rangertagsync/host.domain @REALM Deploy keytab to the node $ sudo cp rangertagsync.service.keytab /etc/security/keytabs/ $ sudo chown ranger:hadoop /etc/security/keytabs/rangertagsync.service.keytab $ sudo chmod 440 /etc/security/keytabs/rangertagsync.service.keytab No errors in the Ranger Tagsync log. ... View more