Support Questions
Find answers, ask questions, and share your expertise

use of nifi with Kafka and Ranger

Explorer

Hey, folks,

I have a Kerberized server, and for the police I'm using Ranger, via console I can publish and consume the messages of Kafka, But now I must do through Nifi and that I'm very lost someone could help me.

Thank you

1 ACCEPTED SOLUTION

Expert Contributor

@Peruvian81 

 

You can try below flow which is just for testing purposes:

 

Screen Shot 2019-10-08 at 1.56.03 PM.png

 

Basically I have a tailFile processor passing data through splitText then these messages are sent to PublishKafka_1_0(use this processor for this test), finally I created a consumer to consume data from the same topic configured in PublishKafka_1_0 storing the data in the file system with putFile.

 

In putFile I have configured Maximum File Count to 10, to avoid excessive space usage in the file system.

View solution in original post

3 REPLIES 3

Expert Contributor

@Peruvian81 

 

You can start testing a flow like below:

 

tailFile --> PublishKafka_1_0(2_0 depending on your Kafka version) 

 

In publishKafka you can use a configuration example like below:

 

Screen Shot 2019-10-07 at 3.58.07 PM.png

 

  • Ensure that the principal has Ranger authorization to publish data to the topic.
  • In Kafka brokers, provide the brokers FQDM, do not use localhost or IPs

 

Explorer

 

Hello ManuelCalvo

 

I don't know if it's asking you too much, as I would have to do a relationship of publisher and consumer. I'm a little new to the subject of Big data. Thank you very much for the info.

 

Greetings

Expert Contributor

@Peruvian81 

 

You can try below flow which is just for testing purposes:

 

Screen Shot 2019-10-08 at 1.56.03 PM.png

 

Basically I have a tailFile processor passing data through splitText then these messages are sent to PublishKafka_1_0(use this processor for this test), finally I created a consumer to consume data from the same topic configured in PublishKafka_1_0 storing the data in the file system with putFile.

 

In putFile I have configured Maximum File Count to 10, to avoid excessive space usage in the file system.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.