question Re: Configure Nifi to mutliple kerberized HDP Cluster in Support Questions

Configure Nifi to mutliple kerberized HDP Cluster

dupuy_gregory — Thu, 12 Aug 2021 12:55:16 GMT

Hi everyone,

I'm working to a new feature with an existing nifi cluster to provide a new service to add an interface with serveral kerberized HDP Cluster.

I would like to know if a single Nifi cluster can use several realms in the same krb5 file.

Reading official documentation, nifi can do it (https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#kerberos_properties) : " If necessary the krb5 file can support multiple realms."

It seems ok but because at this time I have no way to have several hadoop cluster for testing (to notice : my cluster is working already with one kerberized cluster hadoop ), is anybody can confirm or reject this design: one cluster Nifi with different realms to communicate with multiple kerberized hdp cluster.

Thanks for your help and as soon as I have several kerberized cluster hadoop for testing, I will update this article.

Re: Configure Nifi to mutliple kerberized HDP Cluster

MattWho — Fri, 13 Aug 2021 14:42:14 GMT

@dupuy_gregory

Yes, a krb5.conf can contain multiple realms in it.

The HDFS processor components offered through NiFi can each be configured with different hadoop configuration resources (core-site.xml, hdfs-site.xml, etc..) and different Kerberos credentials (keytab and principal).

This will allow various NiFi dataflows to interact with different target Hadoop clusters.

If you found this response addressd yoru query, please take a moment to login and click on "Accept as Solution"

Thank you,

Matt

Re: Configure Nifi to mutliple kerberized HDP Cluster

dupuy_gregory — Fri, 13 Aug 2021 16:50:52 GMT

Thanks for your anwser !

Waiting for several days to have an another Hadoop Cluster, it's now provided.
After some configuration and check, it's working with some trouble.

Let me explain my issue.

I have two Hadoop kerberized cluster : romulus and remus.
For each one, kerberos configuration wad added in the krb5.conf

Find an extract of this configuration :

[libdefaults] default_realm = romulus dns_lookup_realm = false dns_lookup_kdc = true rdns = false dns_canonicalize_hostname = false ticket_lifetime = 168h 0m 0s renew_lifetime = 90d forwardable = true udp_preference_limit = 0 ... [realms] romulus = { admin_server = <...> kdc = <...> } remus = { admin_server = <...> kdc = <...> }

If I use the kinit client , it's working fine.

I configure two GetHDFS processor, each one with the core-site.xml and hdfs-site.xml for Hadoop Configuration Resources, the keytab and the principal associated.

Case one : default_realm is romulus and I start the GetHDFS processor romulus.

=> Ok nice, I get the hdfs file (you can see the flowfile in the queue)

Case two: default_realm is still romulus and I start the GetHDFS processor remus

=> KO :

Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Server has invalid Kerberos principal: nn/<ip namenode active>@<ROMULUS>, expecting: nn/<ip namenode active>@<REMUS>

Nifi trying to connect using the default realm romulus :

Server has invalid Kerberos principal: nn/<ip namenode active>@<ROMULUS>

And with the correct realm remus :

expecting: nn/<ip namenode active>@<REMUS>

To sum up : Nifi working well with the default realm but have an issue with the other realms.

Is there a configuation I missed ?

Thanks

Re: Configure Nifi to mutliple kerberized HDP Cluster

MattWho — Fri, 13 Aug 2021 19:27:02 GMT

@dupuy_gregory

Can you share the configurations for both your GetHDFS processors?

Re: Configure Nifi to mutliple kerberized HDP Cluster

dupuy_gregory — Sat, 14 Aug 2021 09:15:02 GMT

For sur !

For your information because I haven't written : if I switch the default realm under krb5.conf from romulus to remus, romulus not working and remus is ok. it's like Nifi, when the default realm it's not the same as the principal realm is lost and try to use an another realm.

Further more, remus and romulus are not the true name and I need to change path, server name, ip and other objets to share with you 😉

Remus :

Romulus :

Re: Configure Nifi to mutliple kerberized HDP Cluster

dupuy_gregory — Sat, 14 Aug 2021 09:18:36 GMT

I made some changes just to try

- added under /etc/hosts name node IP and server name

- updated dns_lookup_kdc from true to false

- udp_preference_limit from 0 to 1

No effect. It's still not working

Re: Configure Nifi to mutliple kerberized HDP Cluster

Wynner — Wed, 22 Sep 2021 14:23:00 GMT

@dupuy_gregory

When using multiple realms, the KDC servers have to have cross-realm trust setup.

Here is an article with links to good resources that explain in detail: How does a cross realm trust work?

One of the links is broken in the article, here is a good link for the setup of cross-realm trust: Setting up Cross-Realm Kerberos Trusts

As a test to make sure it is working, run the kinit command in a terminal window on the NiFi node for the principal in the non-default realm, for example, kinit hdfs-romulus@ROMULUS

Re: Configure Nifi to mutliple kerberized HDP Cluster

dupuy_gregory — Mon, 20 Dec 2021 18:22:53 GMT

Hello !

Sorry I was out during few months.

No need to have a cross-realm trust setup because it's just a single one direction.

Solution and it's now running :

[realms] romulus = { admin_server = <...> kdc = <...> } remus = { admin_server = <...> kdc = <...> } [domain_realm] <IP Name Node 1 romulus cluster> = romulus <IP Name Node 2 romulus cluster> = romulus <IP Name Node 1 remus cluster> = remus <IP Name Node 2 remus cluster> = remus

Let me explain :

Nifi needs a default realm. the default realm is not used to communicate with project Hadoop cluster kerberised (romus and remulus).

To help Nifi you must maps the name node hostnames to Kerberos realms in the section domain_realm.

In this case, Nifi will try to use the default realm and the realm of the main kerberos defined in the HDFS processor of the project and will failed.

It was a little bit tricky 😉

Re: Configure Nifi to mutliple kerberized HDP Cluster

christopher — Mon, 20 Dec 2021 18:28:47 GMT

@dupuy_gregory - from your last comment, it seems you have a solution, is that correct? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

Re: Configure Nifi to mutliple kerberized HDP Cluster

dupuy_gregory — Mon, 20 Dec 2021 22:15:45 GMT

Hi Christopher,

It's done ! I hope this post can help other people !

Re: Configure Nifi to mutliple kerberized HDP Cluster

christopher — Mon, 20 Dec 2021 23:25:29 GMT

Thank you so much, @dupuy_gregory !

Re: Configure Nifi to mutliple kerberized HDP Cluster

TB_19 — Mon, 27 Mar 2023 07:13:31 GMT

Do i need to config nifi kerberos in order to connect to HDFS that already configure with Kerberos?

Thanks

Re: Configure Nifi to mutliple kerberized HDP Cluster

VidyaSargur — Mon, 27 Mar 2023 09:09:03 GMT

@TB_19 as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.