About ashsskum

ashsskum · ‎11-15-2016

It gets stuck at the very first polling and never retrieves any messages.

ashsskum · ‎11-15-2016

Adding the below had no effect in the nifi-app.log. <appender name="APP_FILE"> . <logger name="org.apache.nifi.processors.kafka.pubsub.ConsumeKafka" level="DEBUG" /> <logger name="org.apache.nifi.processors.kafka.pubsub.PublishKafka" level="DEBUG" /> . </appender>

ashsskum · ‎11-15-2016

Hi, NiFi has not been able to retrieve the messages from Kafka, within few secs after the start it throws the below error. The below log detail doesn't provide adequate detail to troubleshoot. Could you please shed some light on tracing the below issue? Kafka version : 0.9.0.2.4.2.0-258 HDF Version: 2.0.1 NiFi log: 22:44:40 EST DEBUG b24213d2-1008-1158-cec5-cd8961a6f7bc ConsumeKafka[id=b24213d2-1008-1158-cec5-cd8961a6f7bc] Rebalance Alert: Paritions '[]' revoked for lease 'org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@6f4d8702' with consumer 'org.apache.kafka.clients.consumer.KafkaConsumer@31c1244d' 22:44:40 EST DEBUG b24213d2-1008-1158-cec5-cd8961a6f7bc ConsumeKafka[id=b24213d2-1008-1158-cec5-cd8961a6f7bc] Rebalance Alert: Paritions '[olem-0, olem-1]' assigned for lease 'org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@6f4d8702' with consumer 'org.apache.kafka.clients.consumer.KafkaConsumer@31c1244d' Kafka - server log: [2016-11-14 22:44:40,707] INFO [GroupCoordinator 1001]: Preparing to restabilize group olem_group with old generation 0 (kafka.coordinator.GroupCoordinator) [2016-11-14 22:44:40,708] INFO [GroupCoordinator 1001]: Stabilized group olem_group generation 1 (kafka.coordinator.GroupCoordinator) [2016-11-14 22:44:40,709] INFO [GroupCoordinator 1001]: Assignment received from leader for group olem_group for generation 1 (kafka.coordinator.GroupCoordinator) [2016-11-14 22:45:10,711] INFO [GroupCoordinator 1001]: Preparing to restabilize group olem_group with old generation 1 (kafka.coordinator.GroupCoordinator) [2016-11-14 22:45:10,711] INFO [GroupCoordinator 1001]: Group olem_group generation 1 is dead and removed (kafka.coordinator.GroupCoordinator) [2016-11-14 22:47:17,332] INFO [Group Metadata Manager on Broker 1001]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)

ashsskum · ‎11-12-2016

Have an external HIVE table that is based on an avro files and one of the column description is as below. When trying to do SELECT on that with header.timestamp, it fails with the error below. Is there way to query columns with the keywords? col_name data_type header struct<versionnum:binary, timestamp:bigint,uuid:binary> Error: Error while compiling statement: FAILED: ParseException line 1:80 Failed to recognize predicate 'timestamp'. Failed rule: 'identifier' in expression specification

ashsskum · ‎11-10-2016

Yes, but if we need to track how many times the consumer has been going in retries, and notify whenever that happens? Since I do not see a failure relationship for the ConsumeKafka processor, do we have any other workaround/options to achieve this? This is to ensure that we prevent duplicates by identifying the cause and notify whenever that occurs.

ashsskum · ‎11-10-2016

For e.g. PutHDFS processor, if the failure relationship is connected to itself 1) How to control the number of retries of a single flowfile? 2) I see "FlowFile Expiration" can be used to expire the message (meaning discard the Flowfile content if it cannot be reprocessed within a specified period of time). Is there a way to retain the FlowFile content after specified number of retries, that is to persist on a different channel say local file system (PutFile) and perhaps be able to send email notification ?

ashsskum · ‎11-10-2016

I only see success relationship available for ConsumeKafka processor. In case of a failure, that is processor failed to commit the batch due to rebalance do we have any workaround/option to redirect to a PutEmail processor?

ashsskum · ‎11-10-2016

In NiFi - ConsumeKafka processor you have the schedule interval: In Kafka, let's consider the following properties: session.timeout.ms = 300000 (5 mins) heartbeat.interval.ms = 60000 (1 min) If the processor scheduler interval is set to say 600 sec (10 min), would the processor still continue to run to maintain the heartbeat? Would the session timeout be specific to each run, per scheduled interval?

ashsskum · ‎11-02-2016

Thank you. Avro - merge format matches our requirement!

ashsskum · ‎11-01-2016

As MergeContent is just concatenating the binary content of the files, the resultant Avro file can no longer be parsed because there would be more than one header line with the schema defined. Is there an option/workaround to just strip the schema header from each message binary content, before they can be merged? We have the schema (assume just same schema) in a static file that can be referenced anytime, but wanted the final merged file just to have the content and not the header details with schema.

Online	Offline
Last Visited	‎03-06-2019 11:26 PM

Member Since	‎07-09-2016 06:23 AM
Last Visited	‎03-06-2019 11:26 PM
Posts	83
Kudos received	17

Cloudera Community

Re: NiFi: ReplaceTextWithMapping processor

Re: NiFi: Hive dynamic variable

Re: NiFi: ConsumeKafka fails to retrieve from Kafk...

Re: NiFi: ConsumeKafka fails to retrieve from Kafk...

NiFi: ConsumeKafka fails to retrieve from Kafka

Hive: Query keyword column name

Re: NiFi-Kafka: Commit failure notification due to...

NiFi: Flowfile retries

NiFi-Kafka: Commit failure notification due to reb...

NiFi-Kafka: How are they connected?

Re: NiFi: Avro message processing from Kafka

Re: NiFi: Avro message processing from Kafka