Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Connecting nifi to HDFS

avatar
Explorer

Hello

 

We have a HDFS cluster configure to work with Kerberos and we want to use nifi processor PutParquet to write file into hadoop cluster.

do we need to configure nifi to work with kerberos too in order to connect to Hadoop?

what are the steps i need to take in order to do the connection between the two clusters (nifi --> HDFS)

 

Thanks

1 ACCEPTED SOLUTION

avatar
Master Collaborator
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
4 REPLIES 4

avatar
Master Collaborator
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Explorer

Thanks for your reply,

step 2 - a. where can i find the user principle ? b. there are several keytabs for each service . which one do i need ?

step 3 - its not very clear , do i need to install kerberos in nifi cluster in order to

           connect to hdfs cluster ?  do i need to copy the krb5.conf file to nifi cluster ?

 

Thanks

avatar
Master Collaborator

Regarding step 2, You have to determine the HDFS directory where NiFi PutParquet will write the files, and who has access to this directory path on HDFS, that user's user principal and associated keytab is required. I assume if HDFS is secured by Kerberos then the users has to obtain the Kerberos ticket by running kinit with user principal and Keytab to access it at the HDFS  side.  

 

About step 3. No need to install Kerberos service, NiFi needs a Kerberos client on NiFi hosts which is by default installed on most Linux OS.

client config files located at /etc/krb5.conf , to which Kerberos server NiFi PutParquet should connect in order to obtain kerbeors ticket using configured user pric/keytab details, user has updated Krb5.conf file with Kerberos Server details. I mean KDC realm details.

 

If you found this additional response assisted with your issue, please take a moment and click on "Accept as Solution" below this post.

Thank you

 

 

avatar
Explorer

Thanks,

i am getting the following error " ERROR org.apache.nifi.processors.parquet.PutParquet: PutParquet[id=c6dee132-cb63-3b8b-9148-ec10de8044c4] HDFS Configuration error - java.lang.IllegalArgumentException: Can't get Kerberos realm: java.lang.IllegalArgumentException: KrbException: Cannot locate default realm
↳ causes: java.lang.IllegalArgumentException: Can't get Kerberos realm
"