question Re: use of nifi with Kafka and Ranger in Support Questions

use of nifi with Kafka and Ranger

Peruvian81 — Mon, 07 Oct 2019 15:08:16 GMT

Hey, folks,

I have a Kerberized server, and for the police I'm using Ranger, via console I can publish and consume the messages of Kafka, But now I must do through Nifi and that I'm very lost someone could help me.

Thank you

Re: use of nifi with Kafka and Ranger

ManuelCalvo — Mon, 07 Oct 2019 18:59:18 GMT

@Peruvian81

You can start testing a flow like below:

tailFile --> PublishKafka_1_0(2_0 depending on your Kafka version)

In publishKafka you can use a configuration example like below:

Ensure that the principal has Ranger authorization to publish data to the topic.
In Kafka brokers, provide the brokers FQDM, do not use localhost or IPs

Re: use of nifi with Kafka and Ranger

Peruvian81 — Tue, 08 Oct 2019 07:26:48 GMT

Hello ManuelCalvo

I don't know if it's asking you too much, as I would have to do a relationship of publisher and consumer. I'm a little new to the subject of Big data. Thank you very much for the info.

Greetings

Re: use of nifi with Kafka and Ranger

ManuelCalvo — Tue, 08 Oct 2019 17:00:29 GMT

@Peruvian81

You can try below flow which is just for testing purposes:

Basically I have a tailFile processor passing data through splitText then these messages are sent to PublishKafka_1_0(use this processor for this test), finally I created a consumer to consume data from the same topic configured in PublishKafka_1_0 storing the data in the file system with putFile.

In putFile I have configured Maximum File Count to 10, to avoid excessive space usage in the file system.