@dieden9    NiFi provides a number of Kafka processors based off the Kafka Client they are using.  The original ConsumeKafka processor (no number) used the old Kafka 0.8 client.  The 0.8 client processor does not offer the ability to specify a regex for the topic names.  You should be using the Kafka client version processors that match the Kafka server version you are consuming from. From ConsumeKafka_0_10 [1] on, you have the ability to configure the processor to use "names" or "pattern" for the topic name(s). The pattern is a java regular expression.   [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-1-0-nar/1.13.2/org.apache.nifi.processors.kafka.pubsub.ConsumeKafka_1_0/index.html   If you found this help with your query, please take a moment to login and click accept on this solution. Thank you, Matt ... View more

Hi Simon,   I am facing the same issue but my case might be different. Are you using temporary credentials? i.e. assuming role. If so, you can't provide aws_session_token property in nifi unfortunately and this will throw the error you are facing. There is an open issue here https://issues.apache.org/jira/browse/NIFI-7900    /Mahmoud ... View more

Hi @balajip  thanks for the reply. excuse my ignorace as I am still new to cloudera platform.  Is there a config that I can set to override the bucket limit? or should I apply that patch? ... View more