Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

TimeoutException while sending data from Nifi to Kafka topic

TimeoutException while sending data from Nifi to Kafka topic

New Contributor

Hello All,

We are sending some data from Nifi to Kafka topic using PublishKafka processor. We are getting the following error:

PublishKafka Failed to send all messages for StandardFlowFileRecord to Kafka; routing to failure due to org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s) for test_topic:30011 ms has passed since batch creation plus linger time

We have tried increasing the linger time and request.timeout.ms but still facing the issue.

Any help would be appreciated.

9 REPLIES 9

Re: TimeoutException while sending data from Nifi to Kafka topic

Rising Star

@R Patel

Could you please attach the broker server log covering the timestamp when you saw this error. Also what is the current value for linger time and request.timeout.ms?

Also please provide properties for PublishKafka processor. Are you seeing same issue if you publish to any other Kafka topic?

Thanks!

Re: TimeoutException while sending data from Nifi to Kafka topic

New Contributor

Hi dbains,

Thanks for your answer.

As of now, the request.timeout.ms=30000 and linger.ms=1

PublishKafka processor properties are as following:

  • Kafka Brokers: 10.2.2.10:6667
  • Security Protocol: PLAINTEXT
  • Topic Name: test_topic
  • Delivery Guarantee: Best Effort
  • Key Attribute Encoding: UTF-8 Encoded
  • Max Request Size: 1 MB
  • Acknowledgement Wait Time: 5 secs
  • Max Metadata Wait Time: 5 secs

Yes, we see same issue for all the topics we tried.

Thanks.

Re: TimeoutException while sending data from Nifi to Kafka topic

New Contributor

@dbains

Thanks for your answer.

As of now, the request.timeout.ms=30000 and linger.ms=1

PublishKafka processor properties are as following:

  • Kafka Brokers: 10.2.2.10:6667
  • Security Protocol: PLAINTEXT
  • Topic Name: test_topic
  • Delivery Guarantee: Best Effort
  • Key Attribute Encoding: UTF-8 Encoded
  • Max Request Size: 1 MB
  • Acknowledgement Wait Time: 5 secs
  • Max Metadata Wait Time: 5 secs

Yes, we see same issue for all the topics we tried.

Thanks.

Re: TimeoutException while sending data from Nifi to Kafka topic

Rising Star

@R Patel

  • Security Protocol: PLAINTEXT , looks like your Kafka cluster is not secure, could you please confirm?
  • Could you please tell me your Kafka version as well.

Thanks!

Re: TimeoutException while sending data from Nifi to Kafka topic

New Contributor

Hi @dbains

I am getting the same issue while pulling the data into PublishKafka procesor.

My conf for PublishKafka is :

  • Kafka Brokers: ip:6667
  • Security Protocol: PLAINTEXT
  • Topic Name: test
  • Delivery Guarantee: Best Effort
  • Key Attribute Encoding: UTF-8 Encoded
  • Max Request Size: 1 MB
  • Acknowledgement Wait Time: 5 secs
  • Max Metadata Wait Time: 5 secs

It throws the error Failed to update metadata 5000 ms.


Could you provide me any solution for this.

Thanks you.

Re: TimeoutException while sending data from Nifi to Kafka topic

Rising Star

@Surendra Shringi

Hi,

Could you please try passing the FQDN of the kafka broker in the 'Kafka Brokers property of the processor?

Also may I know what version of Kafka are you trying to publish to?

Thanks!

Re: TimeoutException while sending data from Nifi to Kafka topic

New Contributor

Hi @dbains, thanks for your quick response.

I am using PublishKafka_0_10_1.0.0 processor. and i have given FQDN in kafka property but still getting an error.

Do you have any demo template or any further help.

Highlighted

Re: TimeoutException while sending data from Nifi to Kafka topic

Rising Star

Hi @Surendra Shringi,

May I know if you are connecting to a Kerberized Kafka or unsecure?

For template , kindly refer: https://community.hortonworks.com/articles/57262/integrating-apache-nifi-and-apache-kafka.html

Its an old article but it covers the basics in detail. Also, is Kafka in the same HDF cluster or are you using a separate Kafka cluster?

Thanks!

Re: TimeoutException while sending data from Nifi to Kafka topic

New Contributor

We are using unsecure Kafka connection and not using MiNiFi, instead of that we are using MQTT.

this is our NiFi flow ..... MQTT --> PublishKafka --> ConsumeKafka --> PutCassandraQL.

What you suggest to use NiFi flow for below process ...

MQTT --> PublishKafka --> ConsumeKafka --> PutCassandraQL

Don't have an account?
Coming from Hortonworks? Activate your account here