Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

use of nifi with Kafka and Ranger

avatar
Explorer

Hey, folks,

I have a Kerberized server, and for the police I'm using Ranger, via console I can publish and consume the messages of Kafka, But now I must do through Nifi and that I'm very lost someone could help me.

Thank you

1 ACCEPTED SOLUTION

avatar
Expert Contributor

@Peruvian81 

 

You can try below flow which is just for testing purposes:

 

Screen Shot 2019-10-08 at 1.56.03 PM.png

 

Basically I have a tailFile processor passing data through splitText then these messages are sent to PublishKafka_1_0(use this processor for this test), finally I created a consumer to consume data from the same topic configured in PublishKafka_1_0 storing the data in the file system with putFile.

 

In putFile I have configured Maximum File Count to 10, to avoid excessive space usage in the file system.

View solution in original post

3 REPLIES 3

avatar
Expert Contributor

@Peruvian81 

 

You can start testing a flow like below:

 

tailFile --> PublishKafka_1_0(2_0 depending on your Kafka version) 

 

In publishKafka you can use a configuration example like below:

 

Screen Shot 2019-10-07 at 3.58.07 PM.png

 

  • Ensure that the principal has Ranger authorization to publish data to the topic.
  • In Kafka brokers, provide the brokers FQDM, do not use localhost or IPs

 

avatar
Explorer

 

Hello ManuelCalvo

 

I don't know if it's asking you too much, as I would have to do a relationship of publisher and consumer. I'm a little new to the subject of Big data. Thank you very much for the info.

 

Greetings

avatar
Expert Contributor

@Peruvian81 

 

You can try below flow which is just for testing purposes:

 

Screen Shot 2019-10-08 at 1.56.03 PM.png

 

Basically I have a tailFile processor passing data through splitText then these messages are sent to PublishKafka_1_0(use this processor for this test), finally I created a consumer to consume data from the same topic configured in PublishKafka_1_0 storing the data in the file system with putFile.

 

In putFile I have configured Maximum File Count to 10, to avoid excessive space usage in the file system.