Support Questions

vg27 · ‎10-16-2024

We have configured the nifi two nodes cluster on Azure environment with the below nifi properties, we are using private ip address of NIFI, currently the NIFI application is running on ubuntu system without any error, But could not able to access it from web url, can you please guide us what are the steps we need to follow in order to access these from web uRL.

we tried through load balancer, The problem in the load balancer is that we are not able to get the Vm's traffic to reach the load balancer.

Also followed the configuration steps from the below blog
Apache NiFi Walkthroughs

Nifi Properties.

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Core Properties #
nifi.flow.configuration.file=./conf/flow.xml.gz
nifi.flow.configuration.json.file=./conf/flow.json.gz
nifi.flow.configuration.archive.enabled=true
nifi.flow.configuration.archive.dir=./conf/archive/
nifi.flow.configuration.archive.max.time=30 days
nifi.flow.configuration.archive.max.storage=500 MB
nifi.flow.configuration.archive.max.count=
nifi.flowcontroller.autoResumeState=true
nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowservice.writedelay.interval=500 ms
nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=10 millis
nifi.queue.backpressure.count=10000
nifi.queue.backpressure.size=1 GB

nifi.authorizer.configuration.file=./conf/authorizers.xml
nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
nifi.templates.directory=./conf/templates
nifi.ui.banner.text=
nifi.ui.autorefresh.interval=30 sec
nifi.nar.library.directory=./lib
nifi.nar.library.autoload.directory=./extensions
nifi.nar.working.directory=./work/nar/
nifi.documentation.working.directory=./work/docs/components
nifi.nar.unpack.uber.jar=false

####################
# State Management #
####################

nifi.state.management.configuration.file=./conf/state-management.xml
# The ID of the local state provider
nifi.state.management.provider.local=local-provider
# The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster.
nifi.state.management.provider.cluster=zk-provider
# Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server
nifi.state.management.embedded.zookeeper.start=true
# Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties

# Database Settings
nifi.database.directory=./database_repository

# Repository Encryption properties override individual repository implementation properties
nifi.repository.encryption.protocol.version=
nifi.repository.encryption.key.id=
nifi.repository.encryption.key.provider=
nifi.repository.encryption.key.provider.keystore.location=
nifi.repository.encryption.key.provider.keystore.password=

# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=./flowfile_repository
nifi.flowfile.repository.checkpoint.interval=20 secs
nifi.flowfile.repository.always.sync=false
nifi.flowfile.repository.retain.orphaned.flowfiles=true

nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=20000

# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=50 KB
nifi.content.repository.directory.default=./content_repository
nifi.content.repository.archive.max.retention.period=7 days
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=../nifi-content-viewer/

# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository

# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=./provenance_repository
nifi.provenance.repository.max.storage.time=30 days
nifi.provenance.repository.max.storage.size=10 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2

# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=100000

# Component and Node Status History Repository
nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository

# Volatile Status History Repository Properties
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min

# QuestDB Status History Repository Properties
nifi.status.repository.questdb.persist.node.days=14
nifi.status.repository.questdb.persist.component.days=3
nifi.status.repository.questdb.persist.location=./status_repository

# Site to Site properties
nifi.remote.input.host=NiFiTest01
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443
nifi.remote.input.http.enabled=true

nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs

# web properties #
#############################################

# For security, NiFi will present the UI on 127.0.0.1 and only be accessible through this loopback interface.
# Be aware that changing these properties may affect how your instance can be accessed without any restriction.
# We recommend configuring HTTPS instead. The administrators guide provides instructions on how to do this.

nifi.web.http.host=
nifi.web.http.port=
nifi.web.http.network.interface.default=

#############################################

#nifi.web.https.host=NiFiTest01
nifi.web.https.host=0.0.0.0
#nifi.web.https.port=9443
nifi.web.https.port=9443
nifi.web.https.network.interface.default=
nifi.web.https.application.protocols=http/1.1
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=
nifi.web.max.content.size=
nifi.web.max.requests.per.second=30000
nifi.web.max.access.token.requests.per.second=25
nifi.web.request.timeout=60 secs
nifi.web.request.ip.whitelist=
nifi.web.should.send.server.version=true
nifi.web.request.log.format=%{client}a - %u %t "%r" %s %O "%{Referer}i" "%{User-Agent}i"

# Filter JMX MBeans available through the System Diagnostics REST API
nifi.web.jmx.metrics.allowed.filter.pattern=

# Include or Exclude TLS Cipher Suites for HTTPS
nifi.web.https.ciphersuites.include=
nifi.web.https.ciphersuites.exclude=

# security properties #
nifi.sensitive.props.key=@Rfs]HjNl=r(z0&ocSsTrrR8rm?/7qMP

nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
nifi.sensitive.props.additional.keys=

nifi.security.autoreload.enabled=false
nifi.security.autoreload.interval=10 secs
nifi.security.keystore=./conf/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=rxwq+NNjYrSq31UpAoJ/Ox/VLPrX8KfCAm1k6VMo8Oo
nifi.security.keyPasswd=rxwq+NNjYrSq31UpAoJ/Ox/VLPrX8KfCAm1k6VMo8Oo
nifi.security.truststore=./conf/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=VmfdekZSn5N8cNoCF059nUuu3Xqzg+w9DdkHmn9rNyg
nifi.security.user.authorizer=single-user-authorizer
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=single-user-provider
nifi.security.user.jws.key.rotation.period=PT1H
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate=

# OpenId Connect SSO Properties #
nifi.security.user.oidc.discovery.url=
nifi.security.user.oidc.connect.timeout=5 secs
nifi.security.user.oidc.read.timeout=5 secs
nifi.security.user.oidc.client.id=
nifi.security.user.oidc.client.secret=
nifi.security.user.oidc.preferred.jwsalgorithm=
nifi.security.user.oidc.additional.scopes=offline_access
nifi.security.user.oidc.claim.identifying.user=
nifi.security.user.oidc.fallback.claims.identifying.user=
nifi.security.user.oidc.claim.groups=groups
nifi.security.user.oidc.truststore.strategy=JDK
nifi.security.user.oidc.token.refresh.window=60 secs

# Apache Knox SSO Properties #
nifi.security.user.knox.url=
nifi.security.user.knox.publicKey=
nifi.security.user.knox.cookieName=hadoop-jwt
nifi.security.user.knox.audiences=

# SAML Properties #
nifi.security.user.saml.idp.metadata.url=
nifi.security.user.saml.sp.entity.id=
nifi.security.user.saml.identity.attribute.name=

nifi.security.user.saml.group.attribute.name=
nifi.security.user.saml.request.signing.enabled=false
nifi.security.user.saml.want.assertions.signed=true
nifi.security.user.saml.signature.algorithm=http://www.w3.org/2001/04/xmldsig-more#rsa-sha256
nifi.security.user.saml.authentication.expiration=12 hours
nifi.security.user.saml.single.logout.enabled=false
nifi.security.user.saml.http.client.truststore.strategy=JDK
nifi.security.user.saml.http.client.connect.timeout=30 secs
nifi.security.user.saml.http.client.read.timeout=30 secs

# Identity Mapping Properties #
# These properties allow normalizing user identities such that identities coming from different identity providers
# (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing
# DNs from certificates and principals from Kerberos into a common identity string:
#
# nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$
# nifi.security.identity.mapping.value.dn=$1@$2
# nifi.security.identity.mapping.transform.dn=NONE
# nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
# nifi.security.identity.mapping.value.kerb=$1@$2
# nifi.security.identity.mapping.transform.kerb=UPPER

# Group Mapping Properties #
# These properties allow normalizing group names coming from external sources like LDAP. The following example
# lowercases any group name.
#
# nifi.security.group.mapping.pattern.anygroup=^(.*)$
# nifi.security.group.mapping.value.anygroup=$1
# nifi.security.group.mapping.transform.anygroup=LOWER

# Listener Bootstrap properties #
# This property defines the port used to listen for communications from NiFi Bootstrap. If this property
# is missing, empty, or 0, a random ephemeral port is used.
nifi.listener.bootstrap.port=0

# cluster common properties (all nodes must have same values) #
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.heartbeat.missable.max=8
nifi.cluster.protocol.is.secure=true

# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=NiFiTest01
nifi.cluster.node.protocol.port=11443

nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=1 mins
nifi.cluster.flow.election.max.candidates=2

# cluster load balancing properties #
nifi.cluster.load.balance.host=NiFiTest01
nifi.cluster.load.balance.port=6342
nifi.cluster.load.balance.connections.per.node=1
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.comms.timeout=30 sec

# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=NiFiTest01:2181,NiFiTest02:2181
nifi.zookeeper.connect.timeout=10 secs
nifi.zookeeper.session.timeout=10 secs
nifi.zookeeper.root.node=/nifi
nifi.zookeeper.client.secure=false
nifi.zookeeper.security.keystore=
nifi.zookeeper.security.keystoreType=
nifi.zookeeper.security.keystorePasswd=
nifi.zookeeper.security.truststore=
nifi.zookeeper.security.truststoreType=
nifi.zookeeper.security.truststorePasswd=
nifi.zookeeper.jute.maxbuffer=

# Zookeeper properties for the authentication scheme used when creating acls on znodes used for cluster management
# Values supported for nifi.zookeeper.auth.type are "default", which will apply world/anyone rights on znodes
# and "sasl" which will give rights to the sasl/kerberos identity used to authenticate the nifi node
# The identity is determined using the value in nifi.kerberos.service.principal and the removeHostFromPrincipal
# and removeRealmFromPrincipal values (which should align with the kerberos.removeHostFromPrincipal and kerberos.removeRealmFromPrincipal
# values configured on the zookeeper server).
nifi.zookeeper.auth.type=
nifi.zookeeper.kerberos.removeHostFromPrincipal=
nifi.zookeeper.kerberos.removeRealmFromPrincipal=

# kerberos #
nifi.kerberos.krb5.file=

# kerberos service principal #

nifi.kerberos.service.principal=
nifi.kerberos.service.keytab.location=

# kerberos spnego principal #
nifi.kerberos.spnego.principal=
nifi.kerberos.spnego.keytab.location=
nifi.kerberos.spnego.authentication.expiration=12 hours

# external properties files for variable registry
# supports a comma delimited list of file locations
nifi.variable.registry.properties=

# analytics properties #
nifi.analytics.predict.enabled=false
nifi.analytics.predict.interval=3 mins
nifi.analytics.query.interval=5 mins
nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares
nifi.analytics.connection.model.score.name=rSquared
nifi.analytics.connection.model.score.threshold=.90

# runtime monitoring properties
nifi.monitor.long.running.task.schedule=
nifi.monitor.long.running.task.threshold=

# Enable automatic diagnostic at shutdown.
nifi.diagnostics.on.shutdown.enabled=false

# Include verbose diagnostic information.
nifi.diagnostics.on.shutdown.verbose=false

# The location of the diagnostics folder.
nifi.diagnostics.on.shutdown.directory=./diagnostics

# The maximum number of files permitted in the directory. If the limit is exceeded, the oldest files are deleted.
nifi.diagnostics.on.shutdown.max.filecount=10

# The diagnostics folder's maximum permitted size in bytes. If the limit is exceeded, the oldest files are deleted.
nifi.diagnostics.on.shutdown.max.directory.size=10 MB

# Performance tracking properties
## Specifies what percentage of the time we should track the amount of time processors are using CPU, reading from/writing to content repo, etc.
## This can be useful to understand which components are the most expensive and to understand where system bottlenecks may be occurring.
## The value must be in the range of 0 (inclusive) to 100 (inclusive). A larger value will produce more accurate results, while a smaller value may be
## less expensive to compute.

## The value must be in the range of 0 (inclusive) to 100 (inclusive). A larger value will produce more accurate results, while a smaller value may be
## less expensive to compute.
## Results can be obtained by running "nifi.sh diagnostics <filename>" and then inspecting the produced file.
nifi.performance.tracking.percentage=0

# NAR Provider Properties #
# These properties allow configuring one or more NAR providers. A NAR provider retrieves NARs from an external source
# and copies them to the directory specified by nifi.nar.library.autoload.directory.
#
# Each NAR provider property follows the format:
# nifi.nar.library.provider.<identifier>.<property-name>
#
# Each NAR provider must have at least one property named "implementation".
#
# Example HDFS NAR Provider:
# nifi.nar.library.provider.hdfs.implementation=org.apache.nifi.flow.resource.hadoop.HDFSExternalResourceProvider
# nifi.nar.library.provider.hdfs.resources=/path/to/core-site.xml,/path/to/hdfs-site.xml
# nifi.nar.library.provider.hdfs.storage.location=hdfs://hdfs-location
# nifi.nar.library.provider.hdfs.source.directory=/nars
# nifi.nar.library.provider.hdfs.kerberos.principal=nifi@NIFI.COM
# nifi.nar.library.provider.hdfs.kerberos.keytab=/path/to/nifi.keytab
# nifi.nar.library.provider.hdfs.kerberos.password=
#
# Example NiFi Registry NAR Provider:
# nifi.nar.library.provider.nifi-registry.implementation=org.apache.nifi.registry.extension.NiFiRegistryNarProvider
# nifi.nar.library.provider.nifi-registry.url=http://localhost:18080

************************************************************************************************************

Authorizers.xml

<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>

<property name="Initial User Identity 1">CN=PAVANBL</property>
<property name="Initial User Identity 2">CN=NiFiTest01, OU=NIFI</property>
<property name="Initial User Identity 3">CN=NiFiTest02, OU=NIFI</property>
</userGroupProvider>

*********************************************************************************************************

users.xml

**********************************************************************************************************

Attaching the nifi logs which is running

web url unable to access by using private ip address which was given 0.0.0.0.0

Request you to kindly check and update me.

MattWho · ‎10-17-2024

@vg27

A few issues I see with what has been shared:

Issue 1:

IPV4 addresses like 0.0.0.0 and 127.0.0.1 are special reserved network addresses.

0.0.0.0 is not going to be resolvable to any network host. 0.0.0.0 is typically used by server applications running on multi-homed networks. Meaning the the server has multiple NIC cards that are connected. For example: one NIC connected to a companies internal private network and another NIC on same server connected to a public network. BY configuring the server with 0.0.0.0, the special addresses tells the server to listen and accept connection from all NICs. From the browser you would still need to use a resolvable IP address for accessing that server via one of the NICs.

NiFi offers a specific configuration property if you want to define multiple network interfaces you want NiFi to listen for connections on.

127.0.0.1 is the loopback address (localhost), which if used in NiFi, means NiFi is only listening on that loopback address.

So you should be configuring your "nifi.web.https.host" property for the hostname of the server on which it is running. This hostname would need to be resolvable and reachable by your browser.

--------------------------
Issue 2:

You are setting up. NiFi cluster, but are still using the single-user providers for authentication and authorization:

nifi.security.user.authorizer=single-user-authorizer
nifi.security.user.login.identity.provider=single-user-provider

These providers were created so that users could launch NiFi standalone instances out-of-the-box for ease of product evaluation. These providers are not suitable for clustered NiFi setups or multi-users environments.

The authorizers.xml you shared is incomplete and only shows a "file-user-group-provider". The authorizers.xml is easiest to read from bottom of the file upward. At the very bottom you will find the authorizer (for example: "single-user-authorizer" or "managed-authorizer"). Depending on type of authorizer, the authorizer may utilize one or more additional providers ("file-access-policy-provider", composite-user-group-provider, ldap-user-group-provider", "file-user-group-provider", etc...) already loaded further up in the authorizers.xml. So while you may have configured the "file-user-group-provider" and the that provider executes on NiFi creating a users.xml file, the configured single-user-authorizer would never use it.

---------------------------
Issue 3:

Did you generate your own NiFi keystores and truststore for your 2 NiFi nodes or are you using the NiFi auto-generated keystore and truststore? I encourage you to manage your own certificates, keystores, and truststores in a production setup. The certificates generated automatically are self-signed and only contain very specific Subject Alternative names. In a cluster setup, NiFi nodes act as both clients and servers since they communicate with one another. And the owner of those self-signed certifcates is going to be "localhost". This means that all nodes will report as same identity "localhost", which is not good for security.

-------------------------------
Issue 4:

I see you are using the embedded zookeeper (ZK). ZK requires quorum in order to be useable. With only 2 nodes, you only have 2 ZK nodes which is not a quorum. IF either one of the nodes becomes unreachable or goes down, you will not be able to access the one remaining node either since ZK has lost quorum. Quorum consists of an odd number of ZK nodes (typically 3 or 5) depending on how much loss tolerance you want to have. While embedded ZK makes things easy with setup, you'll want to have at least 3 NiFi nodes (this allows you to lose up to one node and still retain access to the NiFi UI of the other 2 nodes. Of course using an external ZK is even better and the best production option as stopping/starting or loss of NiFi nodes has no impact on quorum.

---------------------------------

These are the areas to address first.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

vg27 · ‎10-18-2024

HI Matt,

thanks for the update,

please find the full details of authorizers.xml

</html>

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<!--
This file lists the userGroupProviders, accessPolicyProviders, and authorizers to use when running securely. In order
to use a specific authorizer it must be configured here and it's identifier must be specified in the nifi.properties file.
If the authorizer is a managedAuthorizer, it may need to be configured with an accessPolicyProvider and an userGroupProvider.
This file allows for configuration of them, but they must be configured in order:

...
all userGroupProviders
all accessPolicyProviders
all Authorizers
...
-->
<authorizers>

<!--
The FileUserGroupProvider will provide support for managing users and groups which is backed by a file
on the local file system.

- Users File - The file where the FileUserGroupProvider will store users and groups.

- Legacy Authorized Users File - The full path to an existing authorized-users.xml that will be automatically
be used to load the users and groups into the Users File.

- Initial User Identity [unique key] - The identity of a users and systems to seed the Users File. The name of
each property must be unique, for example: "Initial User Identity A", "Initial User Identity B",
"Initial User Identity C" or "Initial User Identity 1", "Initial User Identity 2", "Initial User Identity 3"

NOTE: Any identity mapping rules specified in nifi.properties will also be applied to the user identities,
so the values should be the unmapped identities (i.e. full DN from a certificate).
-->
<userGroupProvider>

<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>

<property name="Initial User Identity 1">CN=PAVANBL</property>
<property name="Initial User Identity 2">CN=NiFiTest01, OU=NIFI</property>
<property name="Initial User Identity 3">CN=NiFiTest02, OU=NIFI</property>
</userGroupProvider>

<!--
The LdapUserGroupProvider will retrieve users and groups from an LDAP server. The users and groups
are not configurable.

'Authentication Strategy' - How the connection to the LDAP server is authenticated. Possible
values are ANONYMOUS, SIMPLE, LDAPS, or START_TLS.

'Manager DN' - The DN of the manager that is used to bind to the LDAP server to search

for users.
'Manager Password' - The password of the manager that is used to bind to the LDAP server to
search for users.

'TLS - Keystore' - Path to the Keystore that is used when connecting to LDAP using LDAPS or START_TLS.
'TLS - Keystore Password' - Password for the Keystore that is used when connecting to LDAP
using LDAPS or START_TLS.
'TLS - Keystore Type' - Type of the Keystore that is used when connecting to LDAP using
LDAPS or START_TLS such as PKCS12.
'TLS - Truststore' - Path to the Truststore that is used when connecting to LDAP using LDAPS or START_TLS.
'TLS - Truststore Password' - Password for the Truststore that is used when connecting to
LDAP using LDAPS or START_TLS.
'TLS - Truststore Type' - Type of the Truststore that is used when connecting to LDAP using
LDAPS or START_TLS such as PKCS12.
'TLS - Client Auth' - Client authentication policy when connecting to LDAP using LDAPS or START_TLS.
Possible values are REQUIRED, WANT, NONE.
'TLS - Protocol' - Protocol to use when connecting to LDAP using LDAPS or START_TLS. (i.e. TLS,
TLSv1.1, TLSv1.2, etc).
'TLS - Shutdown Gracefully' - Specifies whether the TLS should be shut down gracefully
before the target context is closed. Defaults to false.

'Referral Strategy' - Strategy for handling referrals. Possible values are FOLLOW, IGNORE, THROW.
'Connect Timeout' - Duration of connect timeout. (i.e. 10 secs).
'Read Timeout' - Duration of read timeout. (i.e. 10 secs).

'Url' - Space-separated list of URLs of the LDAP servers (i.e. ldap://<hostname>:<port>).
'Page Size' - Sets the page size when retrieving users and groups. If not specified, no paging is performed.
50,912%
org.apache.nifi.processors.stand

'Page Size' - Sets the page size when retrieving users and groups. If not specified, no paging is performed.
'Sync Interval' - Duration of time between syncing users and groups (i.e. 30 mins). Minimum allowable value is 10 secs.
'Group Membership - Enforce Case Sensitivity' - Sets whether group membership decisions are case sensitive. When a user or group
is inferred (by not specifying or user or group search base or user identity attribute or group name attribute) case sensitivity
is enforced since the value to use for the user identity or group name would be ambiguous. Defaults to false.

'User Search Base' - Base DN for searching for users (i.e. ou=users,o=nifi). Required to search users.
'User Object Class' - Object class for identifying users (i.e. person). Required if searching users.
'User Search Scope' - Search scope for searching users (ONE_LEVEL, OBJECT, or SUBTREE). Required if searching users.
'User Search Filter' - Filter for searching for users against the 'User Search Base' (i.e. (memberof=cn=team1,ou=groups,o=nifi) ). Optional.
'User Identity Attribute' - Attribute to use to extract user identity (i.e. cn). Optional. If not set, the entire DN is used.
'User Group Name Attribute' - Attribute to use to define group membership (i.e. memberof). Optional. If not set
group membership will not be calculated through the users. Will rely on group membership being defined
through 'Group Member Attribute' if set. The value of this property is the name of the attribute in the user ldap entry that
associates them with a group. The value of that user attribute could be a dn or group name for instance. What value is expected
is configured in the 'User Group Name Attribute - Referenced Group Attribute'.
'User Group Name Attribute - Referenced Group Attribute' - If blank, the value of the attribute defined in 'User Group Name Attribute'
is expected to be the full dn of the group. If not blank, this property will define the attribute of the group ldap entry that
the value of the attribute defined in 'User Group Name Attribute' is referencing (i.e. name). Use of this property requires that
'Group Search Base' is also configured.

'Group Search Base' - Base DN for searching for groups (i.e. ou=groups,o=nifi). Required to search groups.
'Group Object Class' - Object class for identifying groups (i.e. groupOfNames). Required if searching groups.
'Group Search Scope' - Search scope for searching groups (ONE_LEVEL, OBJECT, or SUBTREE). Required if searching groups.
'Group Search Filter' - Filter for searching for groups against the 'Group Search Base'. Optional.
'Group Name Attribute' - Attribute to use to extract group name (i.e. cn). Optional. If not set, the entire DN is used.
'Group Member Attribute' - Attribute to use to define group membership (i.e. member). Optional. If not set
group membership will not be calculated through the groups. Will rely on group membership being defined
through 'User Group Name Attribute' if set. The value of this property is the name of the attribute in the group ldap entry that
associates them with a user. The value of that group attribute could be a dn or memberUid for instance. What value is expected
is configured in the 'Group Member Attribute - Referenced User Attribute'. (i.e. member: cn=User 1,ou=users,o=nifi vs. memberUid: user1)
'Group Member Attribute - Referenced User Attribute' - If blank, the value of the attribute defined in 'Group Member Attribute'
is expected to be the full dn of the user. If not blank, this property will define the attribute of the user ldap entry that
the value of the attribute defined in 'Group Member Attribute' is referencing (i.e. uid). Use of this property requires that
'User Search Base' is also configured. (i.e. member: cn=User 1,ou=users,o=nifi vs. memberUid: user1)

NOTE: Any identity mapping rules specified in nifi.properties will also be applied to the user identities.
Group names are not mapped.
-->
<!-- To enable the ldap-user-group-provider remove 2 lines. This is 1 of 2.
<userGroupProvider>
<identifier>ldap-user-group-provider</identifier>
<class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
<property name="Authentication Strategy">START_TLS</property>

<property name="Authentication Strategy">START_TLS</property>

<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>

<property name="Url"></property>
<property name="Page Size"></property>
<property name="Sync Interval">30 mins</property>
<property name="Group Membership - Enforce Case Sensitivity">false</property>

<property name="User Search Base"></property>
<property name="User Object Class">person</property>
<property name="User Search Scope">ONE_LEVEL</property>
<property name="User Search Filter"></property>
<property name="User Identity Attribute"></property>
<property name="User Group Name Attribute"></property>
<property name="User Group Name Attribute - Referenced Group Attribute"></property>

<property name="Group Search Base"></property>
<property name="Group Object Class">group</property>
<property name="Group Search Scope">ONE_LEVEL</property>
<property name="Group Search Filter"></property>
<property name="Group Name Attribute"></property>
<property name="Group Member Attribute"></property>
<property name="Group Member Attribute - Referenced User Attribute"></property>
</userGroupProvider>
To enable the ldap-user-group-provider remove 2 lines. This is 2 of 2. -->

<!--
The ShellUserGroupProvider provides support for retrieving users and groups by way of shell commands
on systems that support `sh`. Implementations available for Linux and Mac OS, and are selected by the
provider based on the system property `os.name`.

-->
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">CN=PAVANBL</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">CN=NiFiTest01, OU=NIFI</property>
<property name="Node Identity 2">CN=NiFiTest02, OU=NIFI</property>
<property name="Node Group"></property>
</accessPolicyProvider>

<!--
The StandardManagedAuthorizer. This authorizer implementation must be configured with the
Access Policy Provider which it will use to access and manage users, groups, and policies.
These users, groups, and policies will be used to make all access decisions during authorization
requests.

- Access Policy Provider - The identifier for an Access Policy Provider defined above.
-->
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>

<!--

***************************************************************************************************

nifi.web.https.host has been modified as per your instructed., provided the private ip address of the vm

nifi.web.https.host=53.13.138.69 #private ip of azure vm
#nifi.web.https.port=9443
nifi.web.https.port=9443
nifi.web.https.network.interface.default=
nifi.web.https.application.protocols=http/1.1
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=20.61.182.212 # load balancer public ip of Azure

nifi.cluster.is.node=true
nifi.cluster.node.address=53.13.138.69
nifi.cluster.node.protocol.host=53.13.138.69
nifi.cluster.node.protocol.port=11443
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=1 mins
nifi.cluster.flow.election.max.candidates=2

# cluster load balancing properties #
nifi.cluster.load.balance.host=53.13.138.69
nifi.cluster.load.balance.port=6342
nifi.cluster.load.balance.connections.per.node=1
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.comms.timeout=30 sec

# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=NiFiTest01:2181,NiFiTest02:2181
nifi.zookeeper.connect.timeout=10 secs
nifi.zookeeper.session.timeout=10 secs
nifi.zookeeper.root.node=/nifi
nifi.zookeeper.client.secure=false
nifi.zookeeper.security.keystore=
nifi.zookeeper.security.keystoreType=
nifi.zookeeper.security.keystorePasswd=
nifi.zookeeper.security.truststore=
nifi.zookeeper.security.truststoreType=
nifi.zookeeper.security.truststorePasswd=
nifi.zookeeper.jute.maxbuffer=

nifi.web.max.content.size=
nifi.web.max.requests.per.second=30000
nifi.web.max.access.token.requests.per.second=25
nifi.web.request.timeout=60 secs
nifi.web.request.ip.whitelist=
nifi.web.should.send.server.version=true
nifi.web.request.log.format=%{client}a - %u %t "%r" %s %O "%{Referer}i" "%{User-Agent}i"

************************************************************************************************

NiFi keystores are generated myself by using the below steps

Create the NiFi Client Certificates. When using the standalone mode of the TLS Toolkit, it is important that all certificates are generated from the same instance, using the same generated NiFi CA certificate to sign each. The certificates can be generated by a single command, or individually. By default, the Distinguished Name (DN) will be CN=<provided_hostname>, OU=NIFI. For more information on toolkit flag options, see NiFi Toolkit Guide: TLS Toolkit Usage.

Create a directory for the cluster configuration and navigate to it.
- mkdir /opt/nifi_cluster_conf && cd /opt/nifi_cluster_conf

Generate the certificates. Running these commands first generates the NiFi CA public certificate and private key if not present, then generates the server certificates, followed by the client certificate necessary for the Initial Admin Identity. An alternative command performing all the steps sequentially is also provided.

/opt/nifi-toolkit-1.22.0/bin/tls-toolkit.sh standalone -n 'node1.nifi' -c 'ca.nifi' — Generates the NiFi CA (ca.nifi) certificate and key if not present and generates and signs node1 certificate, placing the keystore.jks, truststore.jks, and populated nifi.properties in a subdirectory called node1.nifi

Note the creation of the CA certificate and key

% /opt/nifi-toolkit-1.22.0/bin/tls-toolkit.sh standalone -n 'node1.nifi' -c 'ca.nifi'
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandaloneCommandLine - No nifiPropertiesFile specified, using embedded one.
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - Running standalone certificate generation with output directory ../nifi_cluster_conf
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - Generated new CA certificate ../nifi_cluster_conf/nifi-cert.pem and key ../nifi_cluster_conf/nifi-key.key
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - Writing new ssl configuration to ../nifi_cluster_conf/node1.nifi
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - Successfully generated TLS configuration for node1.nifi 1 in ../nifi_cluster_conf/node1.nifi
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - No clientCertDn specified, not generating any client certificates.
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - tls-toolkit standalone completed successfully
% ls -alGh node1.nifi
total 56
drwx------@ 5 johndoe  wheel   160B Jun 26 17:33 .
drwxr-xr-x@ 5 johndoe  wheel   160B Jun 26 17:33 ..
-rw-------@ 1 johndoe  wheel   3.0K Jun 26 17:33 keystore.jks
-rw-------@ 1 johndoe  wheel    17K Jun 26 17:33 nifi.properties
-rw-------@ 1 johndoe  wheel   929B Jun 26 17:33 truststore.jks

**********************************************************************************************************

Created the Zookeeper in two nodes because these is only for testing purpose if everything goes good, we will be creating odd number vms with the zookepper.

*************************************************************************************************

Also created load balancer for these vms using the public ip facing address attaching these two vms to it.

health check probe given /nifi-app/access/status

******************************************************************************************

post configuring all these if we try to access the nifi from the application load balancer ip address we are getting the below errors, kindly let me know, anything i am missing.

MattWho · ‎10-18-2024

@vg27

Sounds like you may have gotten past your original issue after following the guidance I provided already? You new questions don't seem directly related to the original query.

-------
The invalid host header can be resolved by modifying the nifi.web.proxy.host property in the nifi.properties file:

nifi.web.proxy.host=20.61.182.212,20.61.182.212:9443

Or by including additional entry(s) in your certificates SAN list.

---------
Questions:
1. How re you authenticating this user identity?

CN=PAVANBL

2. Have you modified your nif.properties file so that the managed-authorizer is being used?

nifi.security.user.authorizer=managed-authorizer

3. What method(s) or user authentication have you decided to use?
Did you just create an clientAuth certificate that you loaded into your browser to authenticate your user or did you setup some other auth method like kerberos or ldap?

nifi.security.user.login.identity.provider=

4. When it comes to using an load balancer, if you are using any type of login provider to authenticate to your NiFi, you need to configure the load-balancer to use session affinity (a.k.a - Sticky sessions). When you login into a NiFi node the user is issued a client side token and a corresponding server side token is stored on the node. That client side user token is send in every request made after login. The issued client token is ONLY valid for use with the NiFi node that issued it. So session affinity setup in the load balancer is required to make sure that all subsequent request within that same session go to the same NiFi node.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

vg27 · ‎10-18-2024

HI Matt,

Good Day,

Now i can able to connect the web UI with the Load balancer IP address

</html

But unable to authenticate with the user id & password getting the below error.

*************************************************************************************************************

Regarding your questions please find the details.

1. How re you authenticating this user identity?

PAVANBL:

these user has been authenticated with the password which was generated from the certificate key store and downloaded locally these file to view the password generated

*******************************************************************************************************

2 Have you modified your nif.properties file so that the managed-authorizer is being used?

yes i have modified like below

nifi.security.user.authorizer=managed-authorizer

*********************************************************************************************************

.3. What method(s) or user authentication have you decided to use?
Did you just create an clientAuth certificate that you loaded into your browser to authenticate your user or did you setup some other auth method like kerberos or ldap?

nifi.security.user.login.identity.provider=singleUser

Also i uploaded the user certificate to my browser

************************************************************************************************

4.Sticky sessions we will be configured once we can able to test the connection successfully by login.

MattWho · ‎10-21-2024

@vg27

1. So i understand that you have created client certificates for your user. What authority was used to sign these user certificates? Was this authority added to the NiFi configured truststore?

When you open a browser to NiFi's url, NiFi will respond with a WANT for a clientAuth certificate along with a list of trusted authorities from its truststore. If your certificate loaded in your browser is not signed by one of those authorities it will not be presented to NiFi.
If no clientAuth certificate is presented, NiFi will move on to another configured method of user authentication. The fact that you are seeing the NiFi login UI, tells me the TLS exchange did not result in a clientAuth certificate being presented by yoru browser.
With certificate based mutual Auth there is no login required.

3. "nifi.security.user.login.identity.provider=singleUser" is not a valid configuration. I assume you meant "nifi.security.user.login.identity.provider=single-user-provider.

With "Single-user-provider" configured, the only username and password accepted would be for the single user credentials Nifi auto-generated and output to the logs the first time NiFi was started with that provider configured.
If you have no intention of using the single-user-provider, just leave "nifi.security.user.login.identity.provider=" unset.

4. you don't need to worry about sticky sessions if you are only using certificate based authentication, since your client certificate would be passed in every request and their are no tokens involved like in login based providers.

If you did decide to use a login-provider like LDAP or Kerberos later, sticky sessions would need to be setup first or you may never be able to access the UI. Once you enter the username and password, the next request goes is to access UI using that token and if the load balancer were to redirect that to a different node, the UI would not load but instead throw and exception about the unknown user.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Cloudera Community

Support Questions

NIFI cluster Application setup in Azure environment