Ranger Tagsync KafkaException and run out of the client ports

[#1]

Error

Ranger Tagsync shows lots of KafkaException in log file, which causes disk space alert in Ambari. Also it used out all of the client port.

/var/log/ranger/tagsync/tagsync.log

10 Nov 2016 11:46:43 ERROR TagSynchronizer [main] - 262 tag-source:atlas initialization failed with javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, b ut the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user kafka.common.KafkaException: fetching topic metadata for topics [Set(ATLAS_ENTITIES)] from broker [ArrayBuffer(BrokerEndPoint(1001,host.domain,6667))] failed at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:73) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:96) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:67) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) Caused by: java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:122) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$doSend(SyncProducer.scala:81) at kafka.producer.SyncProducer.send(SyncProducer.scala:126) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59) ... 3 more

Background

The Atlas was installed after the HDP2.5.0 was kerberosed. Ambari2.4.1 doesn't create the kerberos principal for Ranger Tagsync, and distributed to the node. Could find the hint from Tagsync log:

/var/log/ranger/tagsync/tagsync.log

10 Nov 2016 11:46:41 WARN SecureClientLogin [main] - 119 /etc/security/keytabs/rangertagsync.service.keytab doesn't exist. 10 Nov 2016 11:46:41 WARN SecureClientLogin [main] - 130 Can't find principal : rangertagsync/host.domain@REALM

Fix

Manually create rangertagsync principal and keytab.

kadmin.local: add_principal -randkey rangertagsync/<code>rangertagsync/host.domain@REALM kadmin.local: xst -k rangertagsync.service.keytab rangertagsync/<code>rangertagsync/host.domain@REALM

Deploy keytab to the node

$ sudo cp rangertagsync.service.keytab /etc/security/keytabs/ $ sudo chown ranger:hadoop /etc/security/keytabs/rangertagsync.service.keytab

$ sudo chmod 440 /etc/security/keytabs/rangertagsync.service.keytab

No errors in the Ranger Tagsync log.