How to configure MiNiFi PublishKafka Processor to inject data to Kafka

[#1]

1) Enable librdkafka extension

Please reference https://github.com/apache/nifi-minifi-cpp/blob/master/README.md

cmake -DENABLE_LIBRDKAFKA ..

2) Configure PublishKafka Processor with below properties

Known Brokers: A comma-separated list of known Kafka Brokers in the format <host>:<port> which map to kafka configuration bootstrap.servers

Topic Name: The Kafka Topic of interest

Delivery Guarantee: Specifies the requirement for guaranteeing that a message is sent to Kafka like all/1/0 which map to kafka configuration request.required.acks

Max Request Size: Maximum Kafka protocol request message size which map to kafka configuration message.max.bytes

Request Timeout: The ack timeout of the producer request in milliseconds which map to request.timeout.ms

Client Name: Client Name to use when communicating with Kafka which map to kafka configuration client.id

Batch Size: Maximum number of messages batched in one MessageSet which map to kafka configuration batch.num.messages

Queue Buffering Max Time: Delay to wait for messages in the producer queue to accumulate before constructing message batches which map to kafka configuration queue.buffering.max.ms

Queue Max Buffer Size: Maximum total message size sum allowed on the producer queue which map to kafka configuration queue.buffering.max.kbytes

Queue Max Message: Maximum number of messages allowed on the producer queue which map to kafka configuration queue.buffering.max.messages

Compress Codec: compression codec to use for compressing message sets like none/gzip/snappy

Max Flow Segment Size: Maximum flow content payload segment size for the kafka record, flow will be fragmented to multiple kafka records based on that

Security Protocol: Protocol used to communicate with brokers like plaintext/ssl

Security CA: File or directory path to CA certificate(s) for verifying the broker's key

Security Cert: Path to client's public key (PEM) used for authentication

Security Private Key: Path to client's private key (PEM) used for authentication

Security Pass Phrase: Private key passphrase

3) Relationships

success: Any FlowFile that is successfully sent to Kafka will be routed to this Relationship

failure: Any FlowFile that cannot be sent to Kafka will be routed to this Relationship

4) Kafka Key

if flow file has attribute kafka.key, it will use that attribute value as kafka key, else it will use the UUID of the flow record as kafka key

5) Please refer to http://kafka.apache.org/quickstart for kafka setup

6) Please refer to https://github.com/edenhill/librdkafka for librdkafak extension