About tseader

Harsh J · ‎04-03-2019

For CDH / CDK Kafka users, the command is already in your PATH as "kafka-consumer-groups".

GYD · ‎03-18-2019

Your job.properties serves future launches and very hand when any of your cluster parameters changes due to upgrades or other issues. It is not meant for launching only but can be used for testing as well if you configure your files appropriately.

laurakia · ‎08-07-2018

What I did is to use a Semaphore to wait for the Callback and then within the Callback itself I set the Exception to a local variable in the main thread and then it throws it to the client: @Override public int send_(String data) throws Exception { logger.trace("Entering KafkaMQDeviceService.send_() data:" + data); Semaphore semaphore = new Semaphore(0); producer.send(new ProducerRecord<String, String>(topic, data), new Callback() { @Override public void onCompletion(RecordMetadata meta, Exception e) { if ( null != e ) setExceptionOnSend(e); semaphore.release(); } }); producer.flush(); try { semaphore.acquire(); } catch (InterruptedException e) {} if ( null != exceptionOnSend ) throw exceptionOnSend; logger.trace("Leaving KafkaMQDeviceService.send_()"); return StatusConstants.DEVICE_SEND_SUCCESSFUL; } Hope it helps.

Genthandsome · ‎07-10-2018

@MughilWhat did you do,,,, How did you diagnose the problem? Where is the directory index?

tseader · ‎04-03-2017

Try addig some arguments into your Oozie run command like so: $ oozie job -oozie http://localhost:11000/oozie -config job.properties -run If those changes don't work for you might try the following: Put your job.properties out in HDFS in the same directory as your workflow, then use Hue FileBrowser to execute the workflow and see if that works. To do that, just checkmark the workflow.xml and a button will appear for you to take action like a submit. Reduce your workflow down to a simple email, then test... add the SSH, then test... keep adding and testing along the way. If things fail at the first and most simple test (email action), then we've eliminated the other actions as being the culprit, and likely quite a few of your job.properties variables too.

pdvorak · ‎03-27-2017

meta.properties was likely left over from a previous instance of the kafka broker. Did you remove the broker role instances and recreate them on the same machine? When that happens, the broker.id changes and kafka won't start properly as a safety measure. A good rule of thumb is if you need to remove the role instances and recreate, then delete everything from the kafka data dir (unless you need to use the partitions), in which case, you'll have to update that meta.properties file manually. -pd

tseader · ‎08-22-2016

yes, thanks for the reply! I figured out the same thing earlier today as I went back to the Flume User Guide and started copying and pasting the properties in again... When I reviewed my config initiall, i didn't look before the attribute name to even see I was missing "hdfs". Definitely an ID10T and PEBKAC error. 🙂 Thanks for keeping me honest!

rspwilliam · ‎04-08-2016

Thanks for the response , I have passed the keytb file in shell action section of workflow that solved the issue. Workflow.xml <file>${keytabaccount}#${keytabaccount}</file> In script : kinit ${keytabaccount}@xxx.xxx.com-k -t ${keytabaccount}.keytab The above approach resolved the problem, Where i can access hive tables in shell script through hive2 beeline. Thanks

fmorcamp · ‎04-06-2016

Hi tseader, Sorry I wasn't avaiable ! For update, It works. The problem was the "Dynamic ressrouce pool". I create a resource pool for my username, and now the job is starting and runing. It was different from our Cloudera 4 in how it works... So now the job is runing, doing the sqoop and the hive job, and terminate successfuly ! Great news! But it very slow for a small table import, I think there is something to do in Dynamic resource pool or yarn setting to use more resource cause, during the job, cpu/emory of my 2 datanode was very less... Maybe you can give me some informations on how to calculate the the max container possible ? To give you some answer: - Yes sqoop was working alone. - Yes our analytics use <args> cause sometime in CDH4 with <command>, they were some error with specific caracters. - Now yes, sqoop/oozie/hive works now. We will try Impala now - No we doesn't try to create a workflow since Hue. I will see with our dev about that. - Not, didn't try with another db. As you thinking, the problem wasn't come from the workflow but the configuration. I'm new in Cloudera/Hadoop, so I learn! I discover the configuration with time! Now I've to find the best configuration to a better usage of our datanode... Thanks again tseader!

tseader · ‎03-30-2016

Good catch! I did notice that in an example I had done, I didn't have the memory configurations even set and things worked fine. I do recall those memory settings being a bit guess and check for me. Something I should probably work on. 😉

Online	Offline
Last Visited	‎07-25-2019 04:57 PM

Member Since	‎08-05-2015 06:23 PM
Last Visited	‎07-25-2019 04:57 PM
Posts	50
Kudos received	14

Cloudera Community

Re: Kafka Producer Async Send with Callback - Erro...

Re: kafka-consumer-groups.sh is missing missing

Re: I am not able to run cloudera VM in VM player ...

Re: kafka-consumer-groups.sh is missing missing

Re: Oozie job.properties - Local or HDFS

Re: Kafka Producer Async Send with Callback - Erro...

Re: Sqoop Import Faile "No Space Left on Device"

Re: Trying to run an oozie job receiving a null po...

Re: kafka producer not accepecting messages

Re: Flume HDFS Sink - File Roll Settings not Worki...

Re: Running hive commands using oozie shell action

Re: Oozie batch to import in Hive from Mysql fail

Re: Problems with Flume and Morphline Solr Sink