About asasvari

asasvari · ‎05-23-2018

I'm glad to hear you were able to drain messages with the new consumer group. Does it fail with the same reason (coordinator dead)? Please note consumers in the "prod-abc-events" consumer group have already established offset to consume from; if there are no new messages produced, they would look like as if they were hanging. Actually coordinator for the consumer group / designated broker is derived from the group.id (note: in the consumer, the request is sent from sendGroupCoordinatorRequest()). So, the second time you start the consumer with the same group id, then it would go to the same broker. If you don't specify group.id for the kafka-console-consumer, it will be generated.

asasvari · ‎05-23-2018

"The cluster was back online and all brokers are up with no underreplicated partitons." Is the problem still present (consumers mark coordinators as dead) or it was being observed only during the time period while the brokers were offline (around 2018-05-21 22:56:20,126) . "can we drain all messages from this topic ? " If you create a new consumer group, you shall be able to poll all messages (quick test: kafka-console-consumer --group test_group_id --bootstrap-server $(hostname):9092 --from-beginning --topic messages-events ). If you use the new Java client / KafkaConsumer in your consumers, you can also seek() to a given offset and start consuming messages from that point.

asasvari · ‎05-22-2018

If min.insync.replicas is 1 and some brokers went offline, then it can be root cause of the issue (assuming minISR is 1 for assuming __consumer_offsets too). In this case, the broker is not alive that is the coordinator for the given consumer group (i.e. there is no partition leader for partition: XY of the internal __consumer_offsets topic). - Can you verify this by running kafka-topics --describe --zookeeper $(hostname -f):2181 - What is offsets.topic.replication.factor set to for__consumer_offsets? It is recommended to set it to 3.

asasvari · ‎05-22-2018

This can occur for example when there are network communication errors between a consumer and the consumer group coordinator (a designated Kafka broker that is the leader for the underlying internal offset topic used for tracking consumers' progress in the consumer group). If that broker is down for some, consumer will mark it as dead. In this case a new consumer coordinator will be selected from the ISR set (assuming offsets.topic.replication.factor=3 and min.insync.replicas for the internal topic is 2). Some questions: - How did you configure for session.timeout.ms, heartbeat.interval.ms, request.timeout.ms? - Does your consumer poll and send heart beat the coordinator Kafka broker on time? - How do you assign partitions in your consumer group? - How do you commit the offsets? - Can you share the Kafka version you are using?

asasvari · ‎03-09-2017

Hi anirbandd, It's likely that you ran into OOZIE-2739. There is a workaround that requires to override the value for oozie.action.shell.setup.hadoop.conf.dir.log4j.content in oozie-site.xml. If you are using Cloudera Manager, you can easiliy do it via using a Safety valve for Oozie. Go to Cloudera Manager -> Oozie -> Configuration page and search for oozie-site.xml. Add the following to "Oozie Server Advanced Configuration Snippet (Safety Valve) for oozie-site.xml" using the XML editor that is better for a newline value carrying config such as the following. <property> <name>oozie.action.shell.setup.hadoop.conf.dir.log4j.content</name> <value> log4j.rootLogger=INFO,console log4j.appender.console=org.apache.log4j.ConsoleAppender log4j.appender.console.target=System.err log4j.appender.console.layout=org.apache.log4j.PatternLayout </value> </property> Then click Save changes and Re-deploy client configuration (restart Oozie).

asasvari · ‎09-30-2016

Hi Rekon As gezapeti suggests the patch is contained by CDH5.7.0+ If you are interested, you can also look it up at Cloudera's Oozie repository, https://github.com/cloudera/oozie . Here, switch to branch 'cdh5-4.1.0_5.7.0' and click commits. You can find the patch in commit commit 830171c6514aad2e20ad6d0bf629a2cc6ba30b6c: https://github.com/cloudera/oozie/commit/830171c6514aad2e20ad6d0bf629a2cc6ba30b6c

Online	Offline
Last Visited	‎12-20-2018 08:41 AM

Member Since	‎06-21-2016 01:39 AM
Last Visited	‎12-20-2018 08:41 AM
Posts	9
Kudos received	2

Cloudera Community

Re: Failing Oozie Launcher, Main class [org.apache...

Re: Error in kafka consumer

Re: Error in kafka consumer

Re: Error in kafka consumer

Re: Error in kafka consumer

Re: Failing Oozie Launcher, Main class [org.apache...

Re: Oozie Action character limit expanded to 128 -...