I'm using Cloudera Enterprise v5.10.1 including Kafka.  We're developing some Kafka streams applications that consumer from Kafka topics, are robust, scalable and performant.  Currently these applications are being deployed using an Oozie workflow and essentially run forever (ie. I don't use a coordinator).  Essentially if I want to scale it up I can just launch another instance using Oozie.   My question is there a better way to deploy Kafka streams application in Cloudera v5.10.1?  I know there are frameworks like Mesos, Docker, etc... but not sure how well it would fit in with Cloudera. ... View more

We are using Cloudera 5.10.1 enterprise.  We have Cloudera Navigator setup and generally it's working and audit events are getting to the Cloudera Navigator Audit Server and are published to Kafka.   For HUE and HIVE audit log files (ie. /var/log/hue/audit and /var/log/hive/audit) cloudera-scm-agent on that node tries to consume the audit log files but gets an error for some of the entries and stops processor the audit files.   I see an error like this...   cloudera-scm-agent.log:... Audit-Plugin navigator_logs ERROR   Stopping processing of hue-HUE_SERVER-... because of corrupt navigator log line: {"username":"user1",..."operation":"USER_LOGOUT",{"username":"user1",...}   Essentially it looks like 2 JSON events are being combined and the 1st event isn't completely written out (ie. there is no '}' ) before the next event starts (ie. '{').   I tried deleted the /var/log/hue/audit files and restarting the service but it just occurs again.  Any ideas?   Also I looked at the timestamps on the audit events causing the issues and they are old.  So I'm unclear why deleting the /var/log/hue/audit files and restarting caused the audit events to be regenerated with the old timestamps.   Maybe to work around this issue there is a cloudera-scm-agent options to skip those corrupt lines and continue?   Richard   ... View more

I'm writing an HBase coprocessor.  It's working well but I'm wondering what's available in the coprocessor environment.  I'm able to retrieve the authenticated client's username no problem (VisibilityUtils.getActiveUser()).  But what about... - client's IP address - client program (eg. MapReduce, HBase shell, etc...) - other info   ... View more

I'm using Cloudera v5.4.9 enterprise edition.   I've enabled Kerberos on HBase, Kafka and Zookeeper and it's working great.  I've created an HBase coprocessor that uses KafkaProducer to send events to a Kafka topic.  Once again I got it working but I had to add a 'KafkaClient' block in the jaas.conf file to allow the HBase coprocessor to connect to Kafka using Kerberos.   ie. KafkaClient {   com.sun.security.auth.module.Krb5LoginModule required   useKeyTab=true   keyTab="hbase.keytab"   principal="hbase/<node>"; };   I had to manually add this on each HBase node and point HBase to use this jaas.conf file rather than use the generated one located in /var/run/cloudera/process/<HBase process directory>/jaas.conf.   ie. Add the following to Java Configuration Options for HBase Master and Region -Djava.security.auth.login.config=/etc/hbase/jaasClientAndKafkaClient.conf   Is there a way to hook into when the jaas.conf file is generated and add this 'KafkaClient' into the jaas.conf file automatically?   ... View more

I'm using Cloudera 5.4.9 enterprise edition.   I've created an HBase coprocessor that is packaged up into a simple JAR.  I add the coprocessor to both the Master and Region servers.  What's the best way to deploy this JAR?  Right now I'm uploading the JAR file to HDFS and then running a -get on each node to put the JAR file in the HBase lib directory.  This doesn't scale well.   ... View more