@Michael Bronson Do you have a working cluster of same HDP version? Is it a kerberized environment? ... View more

@Bhushan Kandalkar Can you start with the following checks to investigate the SaslTransport issues, first the hive keytab # ll /etc/security/keytabs/hive.service.keytab Desired output see ownership and permission bits !!! -r--r----- 1 hive hadoop 353 Oct 11 10:49 /etc/security/keytabs/hive.service.keytab Check the Hive--->Configs--->Advanced hive-site check the hive.server2.authentication.kerberos.principal Desired output hive/_HOST@REALM This should match the entry in the Kerberos database, validate by running on the KDC server the below command as root user # kadmin.local kadmin.local: listprincs Desired output hive/$FQDN@REALM Lastly, Can you regenerate the keytabs using the Ambari Kerberos wizard the restart the cluster HTH ... View more

@Siva A Try using the import control options, you can find them here - https://sqoop.apache.org/docs/1.4.3/SqoopUserGuide.html e.g The --null-string and --null-non-string arguments are optional.\ If not specified, then the string "null" will be used. HTH ... View more

@Bhushan Kandalkar This looks like an authorization issue in ranger, Atlas need the hbase DB for the backend ensure all hbase components are running, Check the in Ranger that Atlas has the correct permissions to 2 important tables Atlas_titan and ATLAS_ENTITY_AUDIT_EVENTS see attached screenshots Log on the Zookeeper CLI and check the ACL's you should have a similar output [zk: localhost:2181(CONNECTED) 0] getAcl /hbase-unsecure/table/ATLAS_ENTITY_AUDIT_EVENTS 'world,'anyone : cdrwa [zk: localhost:2181(CONNECTED) 1] getAcl /hbase-unsecure/table/atlas_titan 'world,'anyone : cdrwa [zk: localhost:2181(CONNECTED)2] getAcl /hbase-unsecure/table/hbase:meta 'world,'anyone : cdrwa [zk: localhost:2181(CONNECTED) 3] getAcl /hbase-unsecure/table/hbase:namespace 'world,'anyone : cdrwa Please revert ... View more

@Michael Bronson Have a look at these 2 tools Kafka check Kafka Tool Happy hadooping ... View more

@Michael Bronson Sorry, I should have misunderstood you. I thought your primary worry was how to balance the topics, including replica and Isr as you depicted in your --describe examples with missing Isr's if not you want to validate the output kafka-topics.sh --describe against what metrics? Please help me understand your request. A Kafka administrator's biggest worry is NOT to have the Isr's in sync. I am happy you got some interesting content in this thread ... View more

@Michael Bronson How to Rebalance Topics in a Kafka Cluster Here is a superb reference by Upendra Mutori to sort out your current issue rebalancing kafka topics Kafka Monitoring Tools Any monitoring tools with JMX support should be able to monitor a Kafka cluster. Here are some monitoring tools : First one is check_kafka.pl from Hari Sekhon. It performs a complete end to end test, i.e. it inserts a message in Kafka as a producer and then extracts it as a consumer. This makes our life easier when measuring service times. Another useful tool is KafkaOffsetMonitor for monitoring Kafka consumers and their position (offset) in the queue. It aids our understanding of how our queue grows and which consumers groups are lagging behind. Last but not least, the LinkedIn folks have developed what I think is the smartest tool out there: Burrow. It analyzes consumer offsets and lags over a window of time and determines the consumer status. You can retrieve this status over an HTTP endpoint and then plug it into your favourite monitoring tool (Server Density for example). There is also Yahoo’s Kafka-Manager. While it does include some basic monitoring, it is more of a management tool. If you are just looking for a Kafka management tool, check out AirBnb’s kafkat. HTH ... View more