Member since
11-03-2023
12
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
753 | 01-31-2024 12:21 PM |
04-04-2024
04:23 AM
1 Kudo
Good Morning -- any takers on helping answer this question? I would be super appreciative.
... View more
03-20-2024
06:46 AM
I'm hoping to add some additional clarity to this question in the form of a partial log extract of relevant information, which you will see below, followed by additional context that I will provide: Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.nifi.registry.web.security.authentication.oidc.OidcService]: Constructor threw exception
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:223)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:111)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:315)
... 60 common frames omitted
Caused by: java.lang.RuntimeException: OpenId Connect support cannot be enabled if the Login Identity Provider or Apache Knox SSO is configured.
at org.apache.nifi.registry.web.security.authentication.oidc.StandardOidcIdentityProvider.validateOIDCConfiguration(StandardOidcIdentityProvider.java:200)
at org.apache.nifi.registry.web.security.authentication.oidc.StandardOidcIdentityProvider.initializeProvider(StandardOidcIdentityProvider.java:120)
at org.apache.nifi.registry.web.security.authentication.oidc.OidcService.<init>(OidcService.java:71)
at org.apache.nifi.registry.web.security.authentication.oidc.OidcService.<init>(OidcService.java:54)
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:210)
... 62 common frames omitted
2024-03-20 12:46:17,451 INFO [main] o.e.jetty.server.handler.ContextHandler Started oeje10w.WebAppContext@9742012{nifi-registry-docs,/nifi-registry-docs,b=file:///opt/nifi-registry/nifi-registry-current/work/jetty/nifi-registry-web-docs-2.0.0-M2.war/webapp/,a=AVAILABLE,h=oeje10s.SessionHandler@60773809{STARTED}}{./lib/nifi-registry-web-docs-2.0.0-M2.war} Here is the additional context: 1. Upon initial loading of the Docker Stack, I had the Environment Variable relating to the login identity set as follows: CN=admin 2. I spun up the docker stack. 3. I had two new files which I had set up as I indicated in the original post, above. 4. While the stack was up, I copied in the two new files: `authorizers.xml` and `nifi-registry.properties`. 5. importantly, I had the relevant volume created in a semi-persistent way. At the bottom of the docker file, I had this: volumes:
nifi_registry_conf: I also had the correlative setting up higher in the yaml, like this: volumes:
- nifi_registry_conf:/opt/nifi-registry/nifi-registry-current/conf 6. I took down the docker stack. 7. I spun up the docker stack, again, confirming that the contents of the two files in question were indeed present as anticipated. 8. The above logs occurred, which may have something to do with NiFi Registry failing to redirect a request to Keycloak. IMPORTANTLY: I have tried this various ways -- all of which produced similar messages in the logs: 1. I have spun up the docker stack without an INITIAL_ADMIN_IDENTITY at all, and repeated the aforementioned processes, with the same basic result. 2. I have spun up the docker stack with an INITIAL_ADMIN_IDENTITY, and this time, I copied out four files: `nifi-registry.properties`, `authorizers.xml`, `authorizations.xml`, and `users.xml`. Upon doing this, I changed the values in the `users.xml` from `CN=admin` to `obfuscatedemail@random.com`. Then I brought the docker stack down and removed the semi-persistent volumes. I then commented out the INITIAL_ADMIN_IDENTITY field, and spun up the docker stack again. Now I copied back in these four files, and brought the docker stack down again. Then I spun it back up again (so that it would start with the new values). No matter -- the same basic result with the same basic logs. 3. I did a similar thing to item 2, above, but only with the `nifi-registry.properties` file and the `authorizers.xml` file. Then I exec'd into the docker container and removed the `users.xml` and `authorizations.xml` files completely. Then I took down the stack and spun it back up (which forces it to recreate new `users.xml` and `authorizations.xml` files. No matter -- the same basic result with the same basic logs. OBSERVATIONS: The error message `Caused by: java.lang.RuntimeException: OpenId Connect support cannot be enabled if the Login Identity Provider or Apache Knox SSO is configured.` seems to imply or reference a setting in `nifi.properties` that does NOT exist in `nifi-registry.properties` -- the `nifi.security.user.login.identity.provider` field. A correlative field in `nifi-registry.properties` would be, for example, `nifi.registry.security.user.login.identity.provider`. However, as I said, this field doesn't exist. The error message is generic in the sense that it seems to indicate potential frustration with EITHER Apache Knox SSO, or Login Identity Provider. Since none of my setup involved Apache Knox, the assumption is that the program is upset about something pertaining to the Login Identity Provider. However, I have no idea what it might be upset about or what it is referring to. It is of very little help that while the `Nifi System Administrator's Guide` has an entire section on OpenID Connect: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#openid_connect , the `Nifi Registry System Administrator's Guide` has no correlative section or correlative information.
... View more
03-19-2024
03:10 PM
Good day -- I am having trouble getting NiFi Registry to work with OIDC. I have NiFi itself, configured successfully -- working with Keycloak as the OIDC provider. Everything is containerized -- I have a Docker stack running Traefik, Bind9, and Keycloak. I have another stack running NiFi (X2), Zookeeper, and NiFi Registry. As I stated, I have NiFi set up and working just fine with Keycloak. However, NiFi Registry is not prompting me for any login. It is not routing the request back to Keycloak for login, and when you see the NiFi Registry page, there is no identity listed and no menu options available. It just looks like this: I have my authorizers.xml file set up this way: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Initial User Identity 2">random@random.com</property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">random@random.com</property>
<property name="NiFi Group Name"></property>
<!--<property name="NiFi Identity 1"></property>-->
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers> And I have the relevant parts of my nifi-registry.properties configured this way: # security properties #
nifi.registry.security.keystore=/opt/nifi-registry/nifi-registry-current/certs/p12/random.com.p12
nifi.registry.security.keystoreType=pkcs12
nifi.registry.security.keystorePasswd=obfuscatedPassword
nifi.registry.security.keyPasswd=obfuscatedPassword
nifi.registry.security.truststore=/opt/nifi-registry/nifi-registry-current/certs/jks/truststore.jks
nifi.registry.security.truststoreType=jks
nifi.registry.security.truststorePasswd=obfuscatedPassword
nifi.registry.security.needClientAuth=false
nifi.registry.security.authorizers.configuration.file=./conf/authorizers.xml
nifi.registry.security.authorizer=managed-authorizer
nifi.registry.security.identity.providers.configuration.file=./conf/identity-providers.xml
nifi.registry.security.identity.provider=openid
# OIDC #
nifi.registry.security.user.oidc.discovery.url=https://random.com/realms/random/.well-known/openid-configuration
nifi.registry.security.user.oidc.connect.timeout=5 secs
nifi.registry.security.user.oidc.read.timeout=5 secs
nifi.registry.security.user.oidc.client.id=flows-registry.random.com
nifi.registry.security.user.oidc.client.secret=obfscatedSecret
nifi.registry.security.user.oidc.preferred.jwsalgorithm=
nifi.registry.security.user.oidc.additional.scopes=openid,email,profile,offline_access
nifi.registry.security.user.oidc.claim.identifying.user=email
nifi.registry.security.user.oidc.fallback.claims.identifying.user=email
nifi.registry.security.user.oidc.claim.groups=groups
nifi.registry.security.user.oidc.truststore.strategy=JDK
nifi.registry.security.user.oidc.token.refresh.window=60 secs notably, i ensured that I set this line ` nifi.registry.security.needClientAuth=false` to false. I just don't understand what to do to get NiFi Registry to route the request to my Keycloak.
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
01-31-2024
12:21 PM
1 Kudo
Nevermind!!! Problem solved -- setting up the mTLS stuff on NiFi was extremely hard for me. The hardest thing, I think, is just making sure that all of the certs are properly set up and configured. I finally got that done. Key takeaways for me (hopefully making it more simple than the documentation): 1. Because NiFi 2.0.x is now using Java 21 and Jetty 10, it's super persnickety about security, etc. The key issue here is that if you are having 'Invalid SNI' issues (Server Name Indication), it is more likely than not because you don't have your SAN (Subject Alternate Name) set up correctly in your certificates. Bottom-line here, is that Version 2.0.X doesn't allow for ip addresses anymore if you are going to secure the instance. They have to be hostnames. 2. Since you are required to get the hostname issue correct, it could get very complicated. In my case, I was already running a DNS server, so it made it a lot easier, but if you don't have a DNS server, you either need to mod your /etc/hosts file (if you are running NiFi from a local machine), or introduce a DNS server into the mix, or something similar... 3. If you want to have a secured version of NiFi running, and if you want to use it in SINGLE USER mode, that appears to still work ONLY if you access it from the local machine running NiFi, wherein you can use `https://localhost:portnumber/nifi`. 4. However, if you want to use NiFi in mTLS mode (or LDAP, etc.), then you appear to lose the capability of accessing NiFi via `localhost`. So you have to do something to resolve the DNS/Hostname issues. AND critically, your SAN information in your certificates needs to match the hostname, or it won't work properly. 5. Do not make the mistake of thinking you can generate certs without SAN information. This just doesn't seem to work, at all in version 2.0.x I'm hopeful that this is helpful to someone else. Thanks.
... View more
01-30-2024
04:37 PM
Good evening, I am having a lot of trouble getting an instance of NiFi running with Mutual TLS. I have different errors depending on whether the effort involves version 1.xx or version 2.xx. I'm hoping to get some help. This is all being attempted via Docker. My YAML: ``` version: "3.9" services: nifi: image: apache/nifi:1.24.0 # hostname: nifi container_name: nifi ports: # - 8443:8443 - "9449:9449" environment: - NIFI_WEB_HTTPS_PORT=9449 - NIFI_WEB_HTTPS_HOST=0.0.0.0 - NIFI_WEB_PROXY_HOST=localhost,localhost:9449,192.168.1.56,192.168.1.56:9449 - AUTH=tls - KEYSTORE_PATH=/opt/nifi/nifi-current/certs/jks/nifi-ks.jks - KEYSTORE_TYPE=JKS - KEYSTORE_PASSWORD=randomPassword - TRUSTSTORE_PATH=/opt/nifi/nifi-current/certs/jks/truststore.jks - TRUSTSTORE_PASSWORD=randomPassword - TRUSTSTORE_TYPE=JKS - INITIAL_ADMIN_IDENTITY='CN=admin' - NIFI_SENSITIVE_PROPS_KEY=randomKey volumes: - nifi_flowfile_repo:/opt/nifi/nifi-current/flowfile_repository - nifi_content_repo:/opt/nifi/nifi-current/content_repository - nifi_provenance_repo:/opt/nifi/nifi-current/provenance_repository - nifi_conf:/opt/nifi/nifi-current/conf - nifi_db:/opt/nifi/nifi-current/database_repository - /opt/nifi-custom-setup/version_2-0-0/20240130/certs/ca-openssl:/opt/nifi/nifi-current/certs - /opt/nifi-custom-setup/version_2-0-0/add_nars:/opt/nifi/nifi-current/add_nars - /mnt/scripts:/opt/nifi/nifi-current/custom_scripts - /mnt/datashare_source_data2:/mnt/datashare_source_data2 - /mnt/drivers/postgresql-42.6.0.jar:/opt/nifi/nifi-current/lib/postgresql-42.6.0.jar - /usr/local/bin:/usr/local/bin:ro - /usr/local/lib:/usr/local/lib:ro # entrypoint: ["/bin/bash", "/opt/nifi/nifi-current/custom_scripts/custom_entrypoint.sh"] restart: unless-stopped volumes: nifi_flowfile_repo: nifi_content_repo: nifi_provenance_repo: nifi_conf: nifi_db: ``` relevant initial logs when the container boots up: ``` 2024-01-31 00:20:34,016 INFO [main] org.eclipse.jetty.server.Server Started @31422ms 2024-01-31 00:20:34,031 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs: 2024-01-31 00:20:34,031 INFO [main] org.apache.nifi.web.server.JettyServer https://0.0.0.0:9449/nifi ``` Error from the web browser when I attempt to log in (It does successfully prompt me for my .p12 certificate and that seems to work.) Insufficient Permissions home Unable to view the user interface. Contact the system administrator. Error from within the container related to the above: ``` ifi@8c3733a74f02:/opt/nifi/nifi-current/logs$ cat nifi-user.log 2024-01-31 00:20:41,209 INFO [NiFi Web Server-32] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 192.168.1.1 [CN=admin] GET https://ddla.w.ccu:9449/nifi-api/flow/current-user 2024-01-31 00:20:41,209 INFO [NiFi Web Server-32] o.a.n.w.s.NiFiAuthenticationFilter Authentication Success [CN=admin] 192.168.1.1 GET https://ddla.w.ccu:9449/nifi-api/flow/current-user 2024-01-31 00:20:41,252 INFO [NiFi Web Server-32] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[CN=admin], groups[] does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response. nifi@8c3733a74f02:/opt/nifi/nifi-current/logs$ ``` The relevant portion from the `authorizers.xml` file: ``` <!-- <authorizer> <identifier>file-provider</identifier> <class>org.apache.nifi.authorization.FileAuthorizer</class> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Users File">./conf/users.xml</property> <property name="Initial Admin Identity">'CN=admin'</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1"></property> </authorizer> ``` A printout of the .p12 contents via `keytool`: ``` Keystore type: PKCS12 Keystore provider: SUN Your keystore contains 1 entry Alias name: admin Creation date: Jan 30, 2024 Entry type: PrivateKeyEntry Certificate chain length: 1 Certificate[1]: Owner: CN=admin Issuer: CN=nifi-ca ``` Pivoting slightly, when I attempt to use the same YAML, but use the following image: apache/nifi:2.0.0-M1 I get a completely different error. I get an Invalid SNI error. If there is anything you could do to point me in the right direction, I would greatly appreciate it.
... View more
Labels:
- Labels:
-
Apache NiFi
01-03-2024
05:35 AM
You are very correct, that it works. And your efforts to simplify the process are much appreciated. It is perhaps more difficult when launching it from a Docker Compose file. I wound up making the following modifications: volumes: - /opt/nifi-custom-setup/version_2-0-0/add_nars:/opt/nifi/nifi-current/add_nars # Add additional nars - /mnt/scripts:/opt/nifi/nifi-current/custom_scripts # Add Bind mount for `custom_entrypoint.sh` Then I wrote a custom script to handle things: #!/bin/bash # Copy the custom NARs to the NiFi lib directory cp /opt/nifi/nifi-current/add_nars/*.nar /opt/nifi/nifi-current/lib/ # Call the default NiFi startup script exec /opt/nifi/nifi-current/bin/nifi.sh run In the end, I very much wish the documentation would provide some alerts to the end users on how to proceed. Again, I'm very appreciative of the assist.
... View more
01-02-2024
12:55 PM
I'm very appreciative of the quick reply. I guess what I don't have clarity on is how this affects the ConvertRecord processor. Are you advising that by installing these nars, that will actually cause a change to the available record readers that are present in the ConvertRecord processor? It's unclear that this is the case by following the link you referenced: https://central.sonatype.com/artifact/org.apache.nifi/nifi-parquet-nar/overview I guess I don't see anything in the documentation that references this information, and a review of the hyperlink doesn't actually provide any information about what is made available by installing the nars. I went to the trouble of downloading the nar from this link and expanding it https://repo1.maven.org/maven2/org/apache/nifi/nifi-parquet-nar/2.0.0-M1/nifi-parquet-nar-2.0.0-M1.nar I see the following: f nifi-parquet-processors-2.0.0-M1.jar created: META-INF/ inflated: META-INF/MANIFEST.MF created: META-INF/services/ created: org/ created: org/apache/ created: org/apache/nifi/ created: org/apache/nifi/parquet/ created: org/apache/nifi/parquet/hadoop/ created: org/apache/nifi/parquet/record/ created: org/apache/nifi/parquet/stream/ created: org/apache/nifi/parquet/utils/ created: org/apache/nifi/processors/ created: org/apache/nifi/processors/parquet/ created: META-INF/maven/ created: META-INF/maven/org.apache.nifi/ created: META-INF/maven/org.apache.nifi/nifi-parquet-processors/ inflated: META-INF/DEPENDENCIES inflated: META-INF/LICENSE inflated: META-INF/NOTICE inflated: META-INF/services/org.apache.nifi.controller.ControllerService inflated: META-INF/services/org.apache.nifi.processor.Processor inflated: org/apache/nifi/parquet/ParquetReader.class inflated: org/apache/nifi/parquet/ParquetRecordSetWriter.class inflated: org/apache/nifi/parquet/hadoop/AvroParquetHDFSRecordReader.class inflated: org/apache/nifi/parquet/hadoop/AvroParquetHDFSRecordWriter.class inflated: org/apache/nifi/parquet/record/ParquetRecordReader.class inflated: org/apache/nifi/parquet/record/WriteParquetResult.class inflated: org/apache/nifi/parquet/stream/NifiOutputStream.class inflated: org/apache/nifi/parquet/stream/NifiParquetInputFile.class inflated: org/apache/nifi/parquet/stream/NifiParquetOutputFile.class inflated: org/apache/nifi/parquet/stream/NifiSeekableInputStream.class inflated: org/apache/nifi/parquet/utils/ParquetConfig.class inflated: org/apache/nifi/parquet/utils/ParquetUtils.class inflated: org/apache/nifi/processors/parquet/ConvertAvroToParquet.class inflated: org/apache/nifi/processors/parquet/FetchParquet.class inflated: org/apache/nifi/processors/parquet/PutParquet.class inflated: META-INF/maven/org.apache.nifi/nifi-parquet-processors/pom.xml inflated: META-INF/maven/org.apache.nifi/nifi-parquet-processors/pom.properties It does seem like it is possible that going through all of this would allow for an additional record reader to be made available in the ConvertRecord processor, but to be honest with you, since there is no documentation on this, it still remains unclear to me that creating a custom distribution will solve the issue.
... View more
01-02-2024
10:58 AM
There is a bug with the InvokeHTTP processor in version 2.0. I carried over my flows from version 1.23, which I upgraded to 1.24, and then I exported the flows as processing groups and imported them into version 2.0. These flows worked fine, previously. In one, I have an InvokeHTTP processor that is sending the document to a Tika server for processing. In version 2.0 there is a new property in this processor that was added. I did not put it there. The documentation makes no mention of it. The property was `Proxy Type` and it's value was `http`. I tried to set it as an empty string, but I continued to get a failure. The error I was getting was: 18:17:26 UTC ERROR InvokeHTTP[id=dfae880c-b84a-3f2c-f1d9-941b8fc625eb] Request Processing failed: FlowFile[filename=random_20210909a.docx]: java.lang.IllegalArgumentException: Unexpected char 0x20 at 5 in header name: Proxy Type Even after setting this property to an empty string, the error persisted. I remedied the situation by deleting the property altogether. After deleting the property, the flow continued to work as before.
... View more
Labels:
- Labels:
-
Apache NiFi
01-02-2024
08:20 AM
No, I never received a reply. I was able solve the problem on my own, eventually.
... View more