Member since
11-14-2019
51
Posts
6
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1178 | 05-26-2023 08:40 AM | |
1297 | 05-22-2023 02:38 AM | |
1198 | 05-15-2023 11:25 AM |
08-31-2023
07:12 AM
I had a flow which needs to be replicated for 100 jobs. one of my jolt transform looks like this. [{
"operation": "default",
"spec": {
"*": {
"id": "#{customer_id_1}",
"warehouse_loaddate" : "${now():format('yyyy-MM-dd HH:mm:ss', 'CET')}"
}
}
}] The customer_id_1 is a parameter currently, which is different in all 10 different jobs. like there are many different parameters in the context like customer_id_2 customer_id_3 etc.. With this I need to touch my code at 100 different places... But since I know the customer ID is present in my upstream processor.. so I just want to name the parameter customer_id and get the value from upstream dynamically... Is there a way that I can send a attribute to downstream processor and use it in jolt ? Without using parameters from parameter context ? We use #{<id>} and ${<func>} to access parameters and Functions.. but how can i refer an attribute from my upstream processor ? Please help me understand this.
... View more
Labels:
- Labels:
-
Apache NiFi
07-17-2023
08:03 AM
Below is the error in bulletin board: Unable to write flowfile content to content repository container default due to archive file size constraints; waiting for archive cleanup. Total number of files currently archived = 1 My flows are simple and this is just test environment, so not much flow files as well may be some 10-20 only. My content_repository is mounted on a disk. it is just 8MB occupied. Disk has some 100 GB storage. below are properties of content repository: # Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=50 KB
nifi.content.repository.directory.default=./content_repository
nifi.content.repository.archive.max.retention.period=7 days
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=../nifi-content-viewer/ I am confused why there is an issue with file size constraints ? as my disc has enough storage capacity. and my flows are simple and not more than 10-20.. so want to understand what is eating space or any worng configuration. Please help me understand. below is complete nifi.properties file. # Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Core Properties #
nifi.flow.configuration.file=./conf/flow.xml.gz
nifi.flow.configuration.json.file=./conf/flow.json.gz
nifi.flow.configuration.archive.enabled=true
nifi.flow.configuration.archive.dir=./conf/archive/
nifi.flow.configuration.archive.max.time=30 days
nifi.flow.configuration.archive.max.storage=500 MB
nifi.flow.configuration.archive.max.count=
nifi.flowcontroller.autoResumeState=true
nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowservice.writedelay.interval=500 ms
nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=10 millis
nifi.queue.backpressure.count=10000
nifi.queue.backpressure.size=1 GB
nifi.authorizer.configuration.file=./conf/authorizers.xml
nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
nifi.templates.directory=./conf/templates
nifi.ui.banner.text=
nifi.ui.autorefresh.interval=30 sec
nifi.nar.library.directory=./lib
nifi.nar.library.autoload.directory=./extensions
nifi.nar.working.directory=./work/nar/
nifi.documentation.working.directory=./work/docs/components
nifi.nar.unpack.uber.jar=false
####################
# State Management #
####################
nifi.state.management.configuration.file=./conf/state-management.xml
# The ID of the local state provider
nifi.state.management.provider.local=local-provider
# The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster.
nifi.state.management.provider.cluster=zk-provider
# Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server
nifi.state.management.embedded.zookeeper.start=false
# Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
# H2 Settings
nifi.database.directory=./database_repository
nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
# Repository Encryption properties override individual repository implementation properties
nifi.repository.encryption.protocol.version=
nifi.repository.encryption.key.id=
nifi.repository.encryption.key.provider=
nifi.repository.encryption.key.provider.keystore.location=
nifi.repository.encryption.key.provider.keystore.password=
# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=./flowfile_repository
nifi.flowfile.repository.checkpoint.interval=20 secs
nifi.flowfile.repository.always.sync=false
nifi.flowfile.repository.retain.orphaned.flowfiles=true
nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=20000
# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=50 KB
nifi.content.repository.directory.default=./content_repository
nifi.content.repository.archive.max.retention.period=7 days
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=../nifi-content-viewer/
# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=./provenance_repository
nifi.provenance.repository.max.storage.time=30 days
nifi.provenance.repository.max.storage.size=10 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2
# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=100000
# Component and Node Status History Repository
nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
# Volatile Status History Repository Properties
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min
# QuestDB Status History Repository Properties
nifi.status.repository.questdb.persist.node.days=14
nifi.status.repository.questdb.persist.component.days=3
nifi.status.repository.questdb.persist.location=./status_repository
# Site to Site properties
nifi.remote.input.host=dwh-architrave.architrave.prod
nifi.remote.input.secure=false
nifi.remote.input.socket.port=10000
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs
# web properties #
#############################################
# For security, NiFi will present the UI on 127.0.0.1 and only be accessible through this loopback interface.
# Be aware that changing these properties may affect how your instance can be accessed without any restriction.
# We recommend configuring HTTPS instead. The administrators guide provides instructions on how to do this.
nifi.web.http.host=dwh-architrave.architrave.prod
nifi.web.http.port=8080
nifi.web.http.network.interface.default=
#############################################
nifi.web.https.host=
nifi.web.https.port=
nifi.web.https.network.interface.default=
nifi.web.https.application.protocols=http/1.1
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=
nifi.web.max.content.size=
nifi.web.max.requests.per.second=30000
nifi.web.max.access.token.requests.per.second=25
nifi.web.request.timeout=60 secs
nifi.web.request.ip.whitelist=
nifi.web.should.send.server.version=true
nifi.web.request.log.format=%{client}a - %u %t "%r" %s %O "%{Referer}i" "%{User-Agent}i"
# Include or Exclude TLS Cipher Suites for HTTPS
nifi.web.https.ciphersuites.include=
nifi.web.https.ciphersuites.exclude=
# security properties #
nifi.sensitive.props.key='12345678901234567890A'
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
nifi.sensitive.props.additional.keys=
nifi.security.autoreload.enabled=false
nifi.security.autoreload.interval=10 secs
nifi.security.keystore=
nifi.security.keystoreType=
nifi.security.keystorePasswd=
nifi.security.keyPasswd=
nifi.security.truststore=
nifi.security.truststoreType=
nifi.security.truststorePasswd=
nifi.security.user.authorizer=single-user-authorizer
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=
nifi.security.user.jws.key.rotation.period=PT1H
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate=
# OpenId Connect SSO Properties #
nifi.security.user.oidc.discovery.url=
nifi.security.user.oidc.connect.timeout=5 secs
nifi.security.user.oidc.read.timeout=5 secs
nifi.security.user.oidc.client.id=
nifi.security.user.oidc.client.secret=
nifi.security.user.oidc.preferred.jwsalgorithm=
nifi.security.user.oidc.additional.scopes=
nifi.security.user.oidc.claim.identifying.user=
nifi.security.user.oidc.fallback.claims.identifying.user=
nifi.security.user.oidc.truststore.strategy=JDK
# Apache Knox SSO Properties #
nifi.security.user.knox.url=
nifi.security.user.knox.publicKey=
nifi.security.user.knox.cookieName=hadoop-jwt
nifi.security.user.knox.audiences=
# SAML Properties #
nifi.security.user.saml.idp.metadata.url=
nifi.security.user.saml.sp.entity.id=
nifi.security.user.saml.identity.attribute.name=
nifi.security.user.saml.group.attribute.name=
nifi.security.user.saml.request.signing.enabled=false
nifi.security.user.saml.want.assertions.signed=true
nifi.security.user.saml.signature.algorithm=http://www.w3.org/2001/04/xmldsig-more#rsa-sha256
nifi.security.user.saml.authentication.expiration=12 hours
nifi.security.user.saml.single.logout.enabled=false
nifi.security.user.saml.http.client.truststore.strategy=JDK
nifi.security.user.saml.http.client.connect.timeout=30 secs
nifi.security.user.saml.http.client.read.timeout=30 secs
# Identity Mapping Properties #
# These properties allow normalizing user identities such that identities coming from different identity providers
# (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing
# DNs from certificates and principals from Kerberos into a common identity string:
#
# nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$
# nifi.security.identity.mapping.value.dn=$1@$2
# nifi.security.identity.mapping.transform.dn=NONE
# nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
# nifi.security.identity.mapping.value.kerb=$1@$2
# nifi.security.identity.mapping.transform.kerb=UPPER
# Group Mapping Properties #
# These properties allow normalizing group names coming from external sources like LDAP. The following example
# lowercases any group name.
#
# nifi.security.group.mapping.pattern.anygroup=^(.*)$
# nifi.security.group.mapping.value.anygroup=$1
# nifi.security.group.mapping.transform.anygroup=LOWER
# Listener Bootstrap properties #
# This property defines the port used to listen for communications from NiFi Bootstrap. If this property
# is missing, empty, or 0, a random ephemeral port is used.
nifi.listener.bootstrap.port=0
# cluster common properties (all nodes must have same values) #
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.heartbeat.missable.max=8
nifi.cluster.protocol.is.secure=false
# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=false
nifi.cluster.node.address=dwh-architrave.architrave.prod
nifi.cluster.node.protocol.port=8082
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=30 sec
nifi.cluster.flow.election.max.candidates=
# cluster load balancing properties #
nifi.cluster.load.balance.host=
nifi.cluster.load.balance.port=6342
nifi.cluster.load.balance.connections.per.node=1
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.comms.timeout=30 sec
# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=
nifi.zookeeper.connect.timeout=10 secs
nifi.zookeeper.session.timeout=10 secs
nifi.zookeeper.root.node=/nifi
nifi.zookeeper.client.secure=false
nifi.zookeeper.security.keystore=
nifi.zookeeper.security.keystoreType=
nifi.zookeeper.security.keystorePasswd=
nifi.zookeeper.security.truststore=
nifi.zookeeper.security.truststoreType=
nifi.zookeeper.security.truststorePasswd=
nifi.zookeeper.jute.maxbuffer=
# Zookeeper properties for the authentication scheme used when creating acls on znodes used for cluster management
# Values supported for nifi.zookeeper.auth.type are "default", which will apply world/anyone rights on znodes
# and "sasl" which will give rights to the sasl/kerberos identity used to authenticate the nifi node
# The identity is determined using the value in nifi.kerberos.service.principal and the removeHostFromPrincipal
# and removeRealmFromPrincipal values (which should align with the kerberos.removeHostFromPrincipal and kerberos.removeRealmFromPrincipal
# values configured on the zookeeper server).
nifi.zookeeper.auth.type=
nifi.zookeeper.kerberos.removeHostFromPrincipal=
nifi.zookeeper.kerberos.removeRealmFromPrincipal=
# kerberos #
nifi.kerberos.krb5.file=
# kerberos service principal #
nifi.kerberos.service.principal=
nifi.kerberos.service.keytab.location=
# kerberos spnego principal #
nifi.kerberos.spnego.principal=
nifi.kerberos.spnego.keytab.location=
nifi.kerberos.spnego.authentication.expiration=12 hours
# external properties files for variable registry
# supports a comma delimited list of file locations
nifi.variable.registry.properties=
# analytics properties #
nifi.analytics.predict.enabled=false
nifi.analytics.predict.interval=3 mins
nifi.analytics.query.interval=5 mins
nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares
nifi.analytics.connection.model.score.name=rSquared
nifi.analytics.connection.model.score.threshold=.90
# runtime monitoring properties
nifi.monitor.long.running.task.schedule=
nifi.monitor.long.running.task.threshold=
# Create automatic diagnostics when stopping/restarting NiFi.
# Enable automatic diagnostic at shutdown.
nifi.diagnostics.on.shutdown.enabled=false
# Include verbose diagnostic information.
nifi.diagnostics.on.shutdown.verbose=false
# The location of the diagnostics folder.
nifi.diagnostics.on.shutdown.directory=./diagnostics
# The maximum number of files permitted in the directory. If the limit is exceeded, the oldest files are deleted.
nifi.diagnostics.on.shutdown.max.filecount=10
# The diagnostics folder's maximum permitted size in bytes. If the limit is exceeded, the oldest files are deleted.
nifi.diagnostics.on.shutdown.max.directory.size=10 MB
# Performance tracking properties
## Specifies what percentage of the time we should track the amount of time processors are using CPU, reading from/writing to content repo, etc.
## This can be useful to understand which components are the most expensive and to understand where system bottlenecks may be occurring.
## The value must be in the range of 0 (inclusive) to 100 (inclusive). A larger value will produce more accurate results, while a smaller value may be
## less expensive to compute.
## Results can be obtained by running "nifi.sh diagnostics <filename>" and then inspecting the produced file.
nifi.performance.tracking.percentage=0
# NAR Provider Properties #
# These properties allow configuring one or more NAR providers. A NAR provider retrieves NARs from an external source
# and copies them to the directory specified by nifi.nar.library.autoload.directory.
#
# Each NAR provider property follows the format:
# nifi.nar.library.provider.<identifier>.<property-name>
#
# Each NAR provider must have at least one property named "implementation".
#
# Example HDFS NAR Provider:
# nifi.nar.library.provider.hdfs.implementation=org.apache.nifi.flow.resource.hadoop.HDFSExternalResourceProvider
# nifi.nar.library.provider.hdfs.resources=/path/to/core-site.xml,/path/to/hdfs-site.xml
# nifi.nar.library.provider.hdfs.storage.location=hdfs://hdfs-location
# nifi.nar.library.provider.hdfs.source.directory=/nars
# nifi.nar.library.provider.hdfs.kerberos.principal=nifi@NIFI.COM
# nifi.nar.library.provider.hdfs.kerberos.keytab=/path/to/nifi.keytab
# nifi.nar.library.provider.hdfs.kerberos.password=
#
# Example NiFi Registry NAR Provider:
# nifi.nar.library.provider.nifi-registry.implementation=org.apache.nifi.registry.extension.NiFiRegistryNarProvider
# nifi.nar.library.provider.nifi-registry.url=http://localhost:18080
... View more
Labels:
- Labels:
-
Apache NiFi
07-07-2023
03:17 AM
Thanks @MattWho for taking time to answer this. I created certificates using tls toolkit like below: ./bin/tls-toolkit.sh standalone -n localhost -C 'CN=admin,OU=NiFi' --subjectAlternativeNames <IP of Docker Registry> -o ./certs "1. NiFi node certificate must have "clientAuth and serverAuth extendedKeyUsage" and that clientAuth certificate needs to be authorized for Read,Write, Delete on "Can Manage Bucket" and "Can Proxy User Requests". NO other authorizations are needed for the NiFi node certificate." Dos it mean, i need to recreate in different way ? or something wrong in above command..? My rest api call is something i saw in the logs and just typed in browser and it worked and gave me above json as result.
... View more
07-05-2023
03:33 AM
Hello All, I had secure Nifi and Secured Nifi-registry running on Docker. Both share same certificates Able to access both UI's individually. Given static IP to Registry in Docker and subjectalternativenames with this IP while generating certificates. Given this static_IP:<port> in registry client. created localhost user : CN=localhost, OU=Nifi in both systems and given necessary permissions. Created few buckets and given all necessary permissions.. But when I try to do version control, it says no available buckets. As i troubleshoot, I can see the logs that shows successful authorization to registry.. 2023-07-05 10:25:47,312 INFO [NiFi Web Server-468] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 10.5.0.1 [CN=new_admin, OU=Nifi] GET https://localhost:8443/nifi-api/flow/registries/21d17428-0189-1000-4a0e-a91cc8cd2aa7/buckets
2023-07-05 10:25:47,312 INFO [NiFi Web Server-468] o.a.n.w.s.NiFiAuthenticationFilter Authentication Success [CN=new_admin, OU=Nifi] 10.5.0.1 GET https://localhost:8443/nifi-api/flow/registries/21d17428-0189-1000-4a0e-a91cc8cd2aa7/buckets
2023-07-05 10:26:11,541 INFO [NiFi Web Server-457] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 10.5.0.1 [CN=new_admin, OU=Nifi] GET https://localhost:8443/nifi-api/flow/process-groups/219e3cc2-0189-1000-7c6f-90a4496cf140
2023-07-05 10:26:11,541 INFO [NiFi Web Server-457] o.a.n.w.s.NiFiAuthenticationFilter Authentication Success [CN=new_admin, OU=Nifi] 10.5.0.1 GET https://localhost:8443/nifi-api/flow/process-groups/219e3cc2-0189-1000-7c6f-90a4496cf140 when i tried to open the buckets API URL (from above log) in my browser, i could get the buckets list aslo.. highlighted my buckets in Nifi-registry I wonder why UI is not able to show the list, when Nifi's internal API call is able to fetch them correctly. Does this indicate a bug from Nifi ? or I am missing something ? After going through all the hurdles and still not successful with Integration, I am thingking about skipping secure installtions and just proceed with unsecure Implementations... Because this is running in VPN on a server, I need to ssh tunnel it anyway to even access the UI. So I was thinking it is okay for unsecure Nifi UIs (as this is internal). Please share your thoughts as well for my understanding. Please let know if any more information is needed in this question for better understanding. Thank You..
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
07-04-2023
06:53 AM
Thank you @MattWho . Sorry to comment late on this. Just to add more details to the answer and how it worked for me: in my docker compose, earlier i used below environment variable: - INITIAL_ADMIN_IDENTITY='CN=admin, OU=NiFi' but somehow it is not interpreted properly.. and giving extra quotes as it is inside containers.. so even though the below looks weird, it worked. - INITIAL_ADMIN_IDENTITY=CN=admin, OU=NiFi (observe i removed single quotes around the value after 😃
... View more
06-29-2023
01:36 PM
Here is my Authorizers.xml # cat authorizers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!--
This file lists the userGroupProviders, accessPolicyProviders, and authorizers to use when running securely. In order
to use a specific authorizer it must be configured here and its identifier must be specified in the nifi-registry.properties file.
If the authorizer is a managedAuthorizer, it may need to be configured with an accessPolicyProvider and an userGroupProvider.
This file allows for configuration of them, but they must be configured in order:
...
all userGroupProviders
all accessPolicyProviders
all Authorizers
...
-->
<authorizers>
<!--
The FileUserGroupProvider will provide support for managing users and groups which is backed by a file
on the local file system.
- Users File - The file where the FileUserGroupProvider will store users and groups.
- Initial User Identity [unique key] - The identity of a users and systems to seed the Users File. The name of
each property must be unique, for example: "Initial User Identity A", "Initial User Identity B",
"Initial User Identity C" or "Initial User Identity 1", "Initial User Identity 2", "Initial User Identity 3"
NOTE: Any identity mapping rules specified in nifi-registry.properties will also be applied to the user identities,
so the values should be the unmapped identities (i.e. full DN from a certificate).
-->
<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Initial User Identity 1">'CN=admin, OU=NiFi'</property>
</userGroupProvider>
<!--
The DatabaseUserGroupProvider will provide support for managing users and groups in a relational database. The framework
will provide a database connection to this provider using the same database information from nifi-registry.properties.
- Initial User Identity [unique key] - Same as the Initial User Identity in the FileUserGroupProvider
-->
<!-- To enable the database-user-group-provider remove 2 lines. This is 1 of 2.
<userGroupProvider>
<identifier>database-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.database.DatabaseUserGroupProvider</class>
<property name="Initial User Identity 1">'CN=admin, OU=NiFi'</property>
</userGroupProvider>
To enable the database-user-group-provider remove 2 lines. This is 2 of 2. -->
<!--
The LdapUserGroupProvider will retrieve users and groups from an LDAP server. The users and groups
are not configurable.
'Authentication Strategy' - How the connection to the LDAP server is authenticated. Possible
values are ANONYMOUS, SIMPLE, LDAPS, or START_TLS.
'Manager DN' - The DN of the manager that is used to bind to the LDAP server to search for users.
'Manager Password' - The password of the manager that is used to bind to the LDAP server to
search for users.
'TLS - Keystore' - Path to the Keystore that is used when connecting to LDAP using LDAPS or START_TLS.
'TLS - Keystore Password' - Password for the Keystore that is used when connecting to LDAP
using LDAPS or START_TLS.
'TLS - Keystore Type' - Type of the Keystore that is used when connecting to LDAP using
LDAPS or START_TLS (i.e. JKS or PKCS12).
'TLS - Truststore' - Path to the Truststore that is used when connecting to LDAP using LDAPS or START_TLS.
'TLS - Truststore Password' - Password for the Truststore that is used when connecting to
LDAP using LDAPS or START_TLS.
'TLS - Truststore Type' - Type of the Truststore that is used when connecting to LDAP using
LDAPS or START_TLS (i.e. JKS or PKCS12).
'TLS - Client Auth' - Client authentication policy when connecting to LDAP using LDAPS or START_TLS.
Possible values are REQUIRED, WANT, NONE.
'TLS - Protocol' - Protocol to use when connecting to LDAP using LDAPS or START_TLS. (i.e. TLS,
TLSv1.1, TLSv1.2, etc).
'TLS - Shutdown Gracefully' - Specifies whether the TLS should be shut down gracefully
before the target context is closed. Defaults to false.
'Referral Strategy' - Strategy for handling referrals. Possible values are FOLLOW, IGNORE, THROW.
'Connect Timeout' - Duration of connect timeout. (i.e. 10 secs).
'Read Timeout' - Duration of read timeout. (i.e. 10 secs).
'Url' - Space-separated list of URLs of the LDAP servers (i.e. ldap://<hostname>:<port>).
'Page Size' - Sets the page size when retrieving users and groups. If not specified, no paging is performed.
'Sync Interval' - Duration of time between syncing users and groups. (i.e. 30 mins).
'Group Membership - Enforce Case Sensitivity' - Sets whether group membership decisions are case sensitive. When a user or group
is inferred (by not specifying or user or group search base or user identity attribute or group name attribute) case sensitivity
is enforced since the value to use for the user identity or group name would be ambiguous. Defaults to false.
'User Search Base' - Base DN for searching for users (i.e. ou=users,o=nifi). Required to search users.
'User Object Class' - Object class for identifying users (i.e. person). Required if searching users.
'User Search Scope' - Search scope for searching users (ONE_LEVEL, OBJECT, or SUBTREE). Required if searching users.
'User Search Filter' - Filter for searching for users against the 'User Search Base' (i.e. (memberof=cn=team1,ou=groups,o=nifi) ). Optional.
'User Identity Attribute' - Attribute to use to extract user identity (i.e. cn). Optional. If not set, the entire DN is used.
'User Group Name Attribute' - Attribute to use to define group membership (i.e. memberof). Optional. If not set
group membership will not be calculated through the users. Will rely on group membership being defined
through 'Group Member Attribute' if set. The value of this property is the name of the attribute in the user ldap entry that
associates them with a group. The value of that user attribute could be a dn or group name for instance. What value is expected
is configured in the 'User Group Name Attribute - Referenced Group Attribute'.
'User Group Name Attribute - Referenced Group Attribute' - If blank, the value of the attribute defined in 'User Group Name Attribute'
is expected to be the full dn of the group. If not blank, this property will define the attribute of the group ldap entry that
the value of the attribute defined in 'User Group Name Attribute' is referencing (i.e. name). Use of this property requires that
'Group Search Base' is also configured.
'Group Search Base' - Base DN for searching for groups (i.e. ou=groups,o=nifi). Required to search groups.
'Group Object Class' - Object class for identifying groups (i.e. groupOfNames). Required if searching groups.
'Group Search Scope' - Search scope for searching groups (ONE_LEVEL, OBJECT, or SUBTREE). Required if searching groups.
'Group Search Filter' - Filter for searching for groups against the 'Group Search Base'. Optional.
'Group Name Attribute' - Attribute to use to extract group name (i.e. cn). Optional. If not set, the entire DN is used.
'Group Member Attribute' - Attribute to use to define group membership (i.e. member). Optional. If not set
group membership will not be calculated through the groups. Will rely on group membership being defined
through 'User Group Name Attribute' if set. The value of this property is the name of the attribute in the group ldap entry that
associates them with a user. The value of that group attribute could be a dn or memberUid for instance. What value is expected
is configured in the 'Group Member Attribute - Referenced User Attribute'. (i.e. member: cn=User 1,ou=users,o=nifi-registry vs. memberUid: user1)
'Group Member Attribute - Referenced User Attribute' - If blank, the value of the attribute defined in 'Group Member Attribute'
is expected to be the full dn of the user. If not blank, this property will define the attribute of the user ldap entry that
the value of the attribute defined in 'Group Member Attribute' is referencing (i.e. uid). Use of this property requires that
'User Search Base' is also configured. (i.e. member: cn=User 1,ou=users,o=nifi-registry vs. memberUid: user1)
NOTE: Any identity mapping rules specified in nifi-registry.properties will also be applied to the user identities.
Group names are not mapped.
-->
<!-- To enable the ldap-user-group-provider remove 2 lines. This is 1 of 2.
<userGroupProvider>
<identifier>ldap-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider</class>
<property name="Authentication Strategy">START_TLS</property>
<property name="Manager DN"></property>
<property name="Manager Password"></property>
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url"></property>
<property name="Page Size"></property>
<property name="Sync Interval">30 mins</property>
<property name="Group Membership - Enforce Case Sensitivity">false</property>
<property name="User Search Base"></property>
<property name="User Object Class">person</property>
<property name="User Search Scope">ONE_LEVEL</property>
<property name="User Search Filter"></property>
<property name="User Identity Attribute"></property>
<property name="User Group Name Attribute"></property>
<property name="User Group Name Attribute - Referenced Group Attribute"></property>
<property name="Group Search Base"></property>
<property name="Group Object Class">group</property>
<property name="Group Search Scope">ONE_LEVEL</property>
<property name="Group Search Filter"></property>
<property name="Group Name Attribute"></property>
<property name="Group Member Attribute"></property>
<property name="Group Member Attribute - Referenced User Attribute"></property>
</userGroupProvider>
To enable the ldap-user-group-provider remove 2 lines. This is 2 of 2. -->
<!--
The ShellUserGroupProvider provides support for retrieving users and groups by way of shell commands
on systems that support `sh`. Implementations available for Linux and Mac OS, and are selected by the
provider based on the system property `os.name`.
'Refresh Delay' - duration to wait between subsequent refreshes. Default is '5 mins'.
'Exclude Groups' - regular expression used to exclude groups. Default is '', which means no groups are excluded.
'Exclude Users' - regular expression used to exclude users. Default is '', which means no users are excluded.
'Command Timeout' - amount of time to wait while executing a command before timing out
-->
<!-- To enable the shell-user-group-provider remove 2 lines. This is 1 of 2.
<userGroupProvider>
<identifier>shell-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.shell.ShellUserGroupProvider</class>
<property name="Refresh Delay">5 mins</property>
<property name="Exclude Groups"></property>
<property name="Exclude Users"></property>
<property name="Command Timeout">60 seconds</property>
</userGroupProvider>
To enable the shell-user-group-provider remove 2 lines. This is 2 of 2. -->
<!--
The CompositeUserGroupProvider will provide support for retrieving users and groups from multiple sources.
- User Group Provider [unique key] - The identifier of user group providers to load from. The name of
each property must be unique, for example: "User Group Provider A", "User Group Provider B",
"User Group Provider C" or "User Group Provider 1", "User Group Provider 2", "User Group Provider 3"
NOTE: Any identity mapping rules specified in nifi-registry.properties are not applied in this implementation. This
behavior would need to be applied by the base implementation.
-->
<!-- To enable the composite-user-group-provider remove 2 lines. This is 1 of 2.
<userGroupProvider>
<identifier>composite-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.CompositeUserGroupProvider</class>
<property name="User Group Provider 1"></property>
</userGroupProvider>
To enable the composite-user-group-provider remove 2 lines. This is 2 of 2. -->
<!--
The CompositeConfigurableUserGroupProvider will provide support for retrieving users and groups from multiple sources.
Additionally, a single configurable user group provider is required. Users from the configurable user group provider
are configurable, however users loaded from one of the User Group Provider [unique key] will not be.
- Configurable User Group Provider - A configurable user group provider.
- User Group Provider [unique key] - The identifier of user group providers to load from. The name of
each property must be unique, for example: "User Group Provider A", "User Group Provider B",
"User Group Provider C" or "User Group Provider 1", "User Group Provider 2", "User Group Provider 3"
NOTE: Any identity mapping rules specified in nifi-registry.properties are not applied in this implementation. This
behavior would need to be applied by the base implementation.
-->
<!-- To enable the composite-configurable-user-group-provider remove 2 lines. This is 1 of 2.
<userGroupProvider>
<identifier>composite-configurable-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.CompositeConfigurableUserGroupProvider</class>
<property name="Configurable User Group Provider">file-user-group-provider</property>
<property name="User Group Provider 1"></property>
</userGroupProvider>
To enable the composite-configurable-user-group-provider remove 2 lines. This is 2 of 2. -->
<!--
The FileAccessPolicyProvider will provide support for managing access policies which is backed by a file
on the local file system.
- User Group Provider - The identifier for an User Group Provider defined above that will be used to access
users and groups for use in the managed access policies.
- Authorizations File - The file where the FileAccessPolicyProvider will store policies.
- Initial Admin Identity - The identity of an initial admin user that will be granted access to the UI and
given the ability to create additional users, groups, and policies. The value of this property could be
a DN when using certificates or LDAP. This property will only be used when there
are no other policies defined.
NOTE: Any identity mapping rules specified in nifi-registry.properties will also be applied to the initial admin identity,
so the value should be the unmapped identity. This identity must be found in the configured User Group Provider.
- NiFi Identity [unique key] - The identity of a NiFi node that will have access to this NiFi Registry and will be able
to act as a proxy on behalf of a NiFi Registry end user. A property should be created for the identity of every NiFi
node that needs to access this NiFi Registry. The name of each property must be unique, for example for three
NiFi clients:
"NiFi Identity A", "NiFi Identity B", "NiFi Identity C" or "NiFi Identity 1", "NiFi Identity 2", "NiFi Identity 3"
NOTE: Any identity mapping rules specified in nifi-registry.properties will also be applied to the nifi identities,
so the values should be the unmapped identities (i.e. full DN from a certificate). This identity must be found
in the configured User Group Provider.
- NiFi Group Name - The name of the group, whose members are NiFi instance/node identities,
that will have access to this NiFi Registry and will be able to act as a proxy on behalf of a NiFi Registry end user.
The members of this group will be granted permission to proxy user requests, as well as read any bucket to perform synchronization checks.
-->
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">'CN=admin, OU=NiFi'</property>
<property name="NiFi Group Name"></property>
<!--<property name="NiFi Identity 1"></property>-->
</accessPolicyProvider>
<!--
The DatabaseAccessPolicyProvider will provide support for managing access policies in a relational database. The
framework will provide a database connection to this provider using the same database information from nifi-registry.properties.
- User Group Provider - Same as User Group Provider in the FileAccessPolicyProvider
- Initial Admin Identity - Same as Initial Admin Identity in the FileAccessPolicyProvider
- NiFi Identity [unique key] - Same as NiFi Identity in the FileAccessPolicyProvider
- NiFi Group Name - Same as NiFi Group Name in the FileAccessPolicyProvider
-->
<!-- To enable the database-access-policy-provider remove 2 lines. This is 1 of 2.
<accessPolicyProvider>
<identifier>database-access-policy-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.database.DatabaseAccessPolicyProvider</class>
<property name="User Group Provider">database-user-group-provider</property>
<property name="Initial Admin Identity">'CN=admin, OU=NiFi'</property>
<property name="NiFi Identity 1"></property>
<property name="NiFi Group Name"></property>
</accessPolicyProvider>
To enable the database-access-policy-provider remove 2 lines. This is 2 of 2. -->
<!--
The StandardManagedAuthorizer. This authorizer implementation must be configured with the
Access Policy Provider which it will use to access and manage users, groups, and policies.
These users, groups, and policies will be used to make all access decisions during authorization
requests.
- Access Policy Provider - The identifier for an Access Policy Provider defined above.
-->
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers> Regarding the quotes... It's coming like that as i deploy from docker compose. From environment in docker compose of Registry... - INITIAL_ADMIN_IDENTITY='CN=admin, OU=NiFi' the value has to be in single quotes as it has '=' inside it. and it is populating as it is in authorisers.xml at 2 places automatically like below: *****Sorry i have to paste it in raw form as it is not supported to paste as code format***** <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Initial User Identity 1">'CN=admin, OU=NiFi'</property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class> <property name="User Group Provider">file-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">'CN=admin, OU=NiFi'</property> <property name="NiFi Group Name"></property> <!--<property name="NiFi Identity 1"></property>--> </accessPolicyProvider> and it is also coming automatically with "'CN=admin, OU=NiFi'" in the users.xml Inside docker i can change the file by bash into it.. but I cannot rerun the service.. May be i need to find a way on how to format the environment variable without '=' ?
... View more
06-29-2023
08:48 AM
I had setup Secure Nifi Registry using Docker compose. Everything worked and I am able to open UI at https://localhost:18443/nifi-registry But when i try to add buckets, I don't even see the settings icon. As I checked the logs i can see below error: nifi_registry_container_persistent | 2023-06-29 15:23:01,075 INFO [NiFi Registry Web Server-23] o.a.n.r.w.m.AccessDeniedExceptionMapper identity[CN=admin, OU=NiFi], groups[] does not have permission to access the requested resource. Unknown user with identity 'CN=admin, OU=NiFi'. Returning Forbidden response.
nifi_registry_container_persistent | 2023-06-29 15:23:01,129 INFO [NiFi Registry Web Server-20] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos service ticket login not supported by this NiFi Registry. Returning Conflict response.
nifi_registry_container_persistent | 2023-06-29 15:23:01,190 INFO [NiFi Registry Web Server-16] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException: OpenId Connect is not configured.. Returning Conflict response.
nifi_registry_container_persistent | 2023-06-29 15:23:01,253 INFO [NiFi Registry Web Server-16] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos service ticket login not supported by this NiFi Registry. Returning Conflict response.
nifi_registry_container_persistent | 2023-06-29 15:23:01,281 INFO [NiFi Registry Web Server-20] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException: OpenId Connect is not configured.. Returning Conflict response. My authentication is based only on certificate. Below is my docker compose: version: "3.7"
services:
# version control for nifi flows
registry:
hostname: DWH_Nifi_registry
container_name: nifi_registry_container_persistent
image: 'apache/nifi-registry:1.22.0' # latest image as of 2023-June.
restart: on-failure
user: root
ports:
- '18443:18443'
environment:
- AUTH=tls
- NIFI_REGISTRY_WEB_HTTPS_PORT=18443
- KEYSTORE_PATH=/opt/certs/keystore.jks
- KEYSTORE_TYPE=JKS
- KEYSTORE_PASSWORD=<pp>
- TRUSTSTORE_PATH=/opt/certs/truststore.jks
- TRUSTSTORE_TYPE=JKS
- TRUSTSTORE_PASSWORD=<PP>
- LOG_LEVEL=INFO
- NIFI_REGISTRY_DB_DIR=/opt/nifi-registry/nifi-registry-current/database
- NIFI_REGISTRY_FLOW_PROVIDER=file
- NIFI_REGISTRY_FLOW_STORAGE_DIR=/opt/nifi-registry/nifi-registry-current/flow_storage
- INITIAL_ADMIN_IDENTITY='CN=admin, OU=NiFi'
volumes:
#- ./nifi_registry/conf:/opt/nifi-registry/nifi-registry-current/conf
- ./nifi-registry/logs:/opt/nifi-registry/nifi-current/logs
- ./nifi-registry/database:/opt/nifi-registry/nifi-registry-current/database
- ./nifi-registry/flow_storage:/opt/nifi-registry/nifi-registry-current/flow_storage
- ./nifi-toolkit-1.22.0/certs/localhost/keystore.jks:/opt/certs/keystore.jks
- ./nifi-toolkit-1.22.0/certs/localhost/truststore.jks:/opt/certs/truststore.jks
networks:
- nifi_persistent_network
networks:
nifi_persistent_network:
driver: bridge I can see users.xml and authorizers.xml getting populated. But still I am not allowed to create buckets in UI. Can someone please help me here understanding what i am doing wrong.. users.xml <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<tenants>
<groups/>
<users>
<user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125" identity="'CN=admin, OU=NiFi'"/>
</users>
</tenants> authorizations.xml <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <authorizations> <policies> <policy identifier="05b96464-9ec8-312a-8459-67812a8b48c1" resource="/buckets" action="D"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="15e4e0bd-cb28-34fd-8587-f8d15162cba5" resource="/tenants" action="W"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="2f470357-e82c-38ee-8062-ab6388d6ec75" resource="/actuator" action="W"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="d59a54f7-6dd6-34ad-a279-a26ffdb9eef8" resource="/proxy" action="R"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="dfbf3c51-fdec-3328-b169-3b54eb033147" resource="/buckets" action="W"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="3ee4703f-94ca-33c2-8060-17f5d313f560" resource="/actuator" action="D"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270" resource="/proxy" action="W"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="ff96062a-fa99-36dc-9942-0f6442ae7212" resource="/policies" action="R"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="01b87cb5-c0b6-342d-b108-d8bc03ab5cde" resource="/policies" action="D"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="ad99ea98-3af6-3561-ae27-5bf09e1d969d" resource="/policies" action="W"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="6dbdbffd-8a7d-32e1-ba3e-f600e6c69791" resource="/proxy" action="D"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="0eaa47b9-e409-304e-8682-30d1b0d86d05" resource="/swagger" action="R"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="9d182b11-ebe3-3a7a-8731-98ce6d6e44fd" resource="/buckets" action="R"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="ac587f43-6e1c-3890-81fd-83b4df2e678e" resource="/swagger" action="D"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="627410be-1717-35b4-a06f-e9362b89e0b7" resource="/tenants" action="R"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="2fd3fcf5-b10f-33fa-8d8e-b262fa34815e" resource="/actuator" action="R"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="2dbc92a2-b091-3616-8e88-5078b9103b04" resource="/tenants" action="D"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> <policy identifier="cf4d8390-5ac7-3ff0-82ce-a274b5f88b21" resource="/swagger" action="W"> <user identifier="ab7ae4a1-fcad-3ed3-92ca-ce75fa140125"/> </policy> </policies> </authorizations> providers.xml <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <providers> <!-- NOTE: The providers in this file must be listed in the order defined in providers.xsd which is the following: 1) Flow Persistence Provider (Must occur once and only once) 2) Event Hook Providers (May occur 0 or more times) 3) Bundle Persistence Provider (Must occur once and only once) --> <flowPersistenceProvider> <class>org.apache.nifi.registry.provider.flow.FileSystemFlowPersistenceProvider</class> <property name="Flow Storage Directory">/opt/nifi-registry/nifi-registry-current/flow_storage</property> </flowPersistenceProvider> <!-- <flowPersistenceProvider> <class>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</class> <property name="Flow Storage Directory">./flow_storage</property> <property name="Remote To Push"></property> <property name="Remote Access User"></property> <property name="Remote Access Password"></property> <property name="Remote Clone Repository"></property> </flowPersistenceProvider> --> <!-- <flowPersistenceProvider> <class>org.apache.nifi.registry.provider.flow.DatabaseFlowPersistenceProvider</class> </flowPersistenceProvider> --> <!-- <eventHookProvider> <class>org.apache.nifi.registry.provider.hook.ScriptEventHookProvider</class> <property name="Script Path"></property> <property name="Working Directory"></property> --> <!-- Optional Whitelist Event types <property name="Whitelisted Event Type 1">CREATE_FLOW</property> <property name="Whitelisted Event Type 2">DELETE_FLOW</property> --> <!-- </eventHookProvider> --> <!-- This will log all events to a separate file specified by the EVENT_APPENDER in logback.xml --> <!-- <eventHookProvider> <class>org.apache.nifi.registry.provider.hook.LoggingEventHookProvider</class> </eventHookProvider> --> <extensionBundlePersistenceProvider> <class>org.apache.nifi.registry.provider.extension.FileSystemBundlePersistenceProvider</class> <property name="Extension Bundle Storage Directory">./extension_bundles</property> </extensionBundlePersistenceProvider> <!-- Example S3 Bundle Persistence Provider - Requires nifi-registry-aws-assembly to be added to the classpath via a custom extension dir in nifi-registry.properties Example: nifi.registry.extension.dir.aws=./ext/aws/lib Where "./ext/aws/lib" contains the extracted contents of nifi-registry-aws-assembly - "Region" - The name of the S3 region where the bucket exists - "Bucket Name" - The name of an existing bucket to store extension bundles - "Key Prefix" - An optional prefix that if specified will be added to the beginning of all S3 keys - "Credentials Provider" - Indicates how credentials will be provided, must be a value of DEFAULT_CHAIN or STATIC - DEFAULT_CHAIN will consider in order: Java system properties, environment variables, credential profiles (~/.aws/credentials) - STATIC requires that "Access Key" and "Secret Access Key" be specified directly in this file - "Access Key" - The access key to use when using STATIC credentials provider - "Secret Access Key" - The secret access key to use when using STATIC credentials provider - "Endpoint URL" - An optional URL that overrides the default AWS S3 endpoint URL. Set this when using an AWS S3 API compatible service hosted at a different URL. --> <!-- <extensionBundlePersistenceProvider> <class>org.apache.nifi.registry.aws.S3BundlePersistenceProvider</class> <property name="Region">us-east-1</property> <property name="Bucket Name">my-bundles</property> <property name="Key Prefix"></property> <property name="Credentials Provider">DEFAULT_CHAIN</property> <property name="Access Key"></property> <property name="Secret Access Key"></property> <property name="Endpoint URL"></property> </extensionBundlePersistenceProvider> --> </providers> Nifi-registry.properties: # web properties # nifi.registry.web.war.directory=./lib nifi.registry.web.http.host= nifi.registry.web.http.port= nifi.registry.web.https.host=DWH_Nifi_registry nifi.registry.web.https.port=18443 nifi.registry.web.https.application.protocols=http/1.1 nifi.registry.web.jetty.working.directory=./work/jetty nifi.registry.web.jetty.threads=200 nifi.registry.web.should.send.server.version=true # security properties # nifi.registry.security.keystore=/opt/certs/keystore.jks nifi.registry.security.keystoreType=JKS nifi.registry.security.keystorePasswd=IN7DYhm8+7E0iV3qxfC+MzdeZXLgm1k6ts5oNhk9hR4 nifi.registry.security.keyPasswd=IN7DYhm8+7E0iV3qxfC+MzdeZXLgm1k6ts5oNhk9hR4 nifi.registry.security.truststore=/opt/certs/truststore.jks nifi.registry.security.truststoreType=JKS nifi.registry.security.truststorePasswd=u9PZiAm3Sd/CFuPAdapGShHYLC0LOtwXV9d6FyrKfk4 nifi.registry.security.needClientAuth= nifi.registry.security.authorizers.configuration.file=./conf/authorizers.xml nifi.registry.security.authorizer=managed-authorizer nifi.registry.security.identity.providers.configuration.file=./conf/identity-providers.xml nifi.registry.security.identity.provider= # sensitive property protection properties # # nifi.registry.sensitive.props.additional.keys= # providers properties # nifi.registry.providers.configuration.file=./conf/providers.xml # registry alias properties # nifi.registry.registry.alias.configuration.file=./conf/registry-aliases.xml # extensions working dir # nifi.registry.extensions.working.directory=./work/extensions # legacy database properties, used to migrate data from original DB to new DB below # NOTE: Users upgrading from 0.1.0 should leave these populated, but new installs after 0.1.0 should leave these empty nifi.registry.db.directory= nifi.registry.db.url.append= # database properties nifi.registry.db.url=jdbc:h2:./database/nifi-registry-primary;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE nifi.registry.db.driver.class=org.h2.Driver nifi.registry.db.driver.directory=/opt/nifi-registry/nifi-registry-current/database nifi.registry.db.username=nifireg nifi.registry.db.password=nifireg nifi.registry.db.maxConnections=5 nifi.registry.db.sql.debug=false # extension directories # # Each property beginning with "nifi.registry.extension.dir." will be treated as location for an extension, # and a class loader will be created for each location, with the system class loader as the parent # #nifi.registry.extension.dir.1=/path/to/extension1 #nifi.registry.extension.dir.2=/path/to/extension2 nifi.registry.extension.dir.aws=./ext/aws/lib # Identity Mapping Properties # # These properties allow normalizing user identities such that identities coming from different identity providers # (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing # DNs from certificates and principals from Kerberos into a common identity string: # # nifi.registry.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$ # nifi.registry.security.identity.mapping.value.dn=$1@$2 # nifi.registry.security.identity.mapping.transform.dn=NONE # nifi.registry.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$ # nifi.registry.security.identity.mapping.value.kerb=$1@$2 # nifi.registry.security.identity.mapping.transform.kerb=UPPER # Group Mapping Properties # # These properties allow normalizing group names coming from external sources like LDAP. The following example # lowercases any group name. # # nifi.registry.security.group.mapping.pattern.anygroup=^(.*)$ # nifi.registry.security.group.mapping.value.anygroup=$1 # nifi.registry.security.group.mapping.transform.anygroup=LOWER # kerberos properties # nifi.registry.kerberos.krb5.file= nifi.registry.kerberos.spnego.principal= nifi.registry.kerberos.spnego.keytab.location= nifi.registry.kerberos.spnego.authentication.expiration=12 hours # OIDC # nifi.registry.security.user.oidc.discovery.url= nifi.registry.security.user.oidc.connect.timeout= nifi.registry.security.user.oidc.read.timeout= nifi.registry.security.user.oidc.client.id= nifi.registry.security.user.oidc.client.secret= nifi.registry.security.user.oidc.preferred.jwsalgorithm= # revision management # # This feature should remain disabled until a future NiFi release that supports the revision API changes
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
06-27-2023
07:17 AM
@alim Can you please suggest..
... View more
06-27-2023
04:56 AM
Hi All, I followed instructions from https://community.cloudera.com/t5/Community-Articles/Setting-Up-a-Secure-Apache-NiFi-Registry/ta-p/247753 to set up secure Nifi Registry on my mac. All steps were successful but when i open https link, it says - ` This site can’t provide a secure connection localhost didn’t accept your login certificate, or one may not have been provided. Try contacting the system admin. ERR_BAD_SSL_CLIENT_AUTH_CERT` So after 2 days spent on it resolving with no luck, I am here to get help from community. Before asking here, I made a test on a virtual windows machine and it works fine.. The issue with mac is that when i am adding the certificate to keychain by double clicking on the p12 file, Mac (Ventura 13.4.1 (22F82)) it says ""cannot add certificate to keychain unknown format" So i tried changing the format using openssl - and tried to convert the certificate to .cer using this below command - "openssl pkcs12 -in CN=admin_OU=NiFi.p12 -clcerts -nokeys -out certificate.cer" it asked for a password and converted to .cer. and then when i doubleclick on it now, it got added and i can see it in keychain of mac. Now, even after it got added to keychain, when i open the https link, it throws same error 'ERR_BAD_SSL_CLIENT_AUTH_CERT' As i tried to replicate the error in Windows, i get the same error if i don't add the certiifcate to keychain. But when i doubleclick and add it to keychain, it gets resolved. and i can access the https:// So I can confirm the culprit is mac adding certificate to keychain.. can someone help me resolve this issue in Mac..
... View more
Labels:
- Labels:
-
NiFi Registry
06-15-2023
02:22 AM
1 Kudo
Thanks @MattWho I created https://issues.apache.org/jira/browse/NIFI-11695 Hoping for the best!
... View more