Member since
09-30-2020
13
Posts
0
Kudos Received
0
Solutions
02-02-2023
07:04 AM
Hello @MattWho , Thanks for the assistance!! Please find the below info regarding the error 1: I have started new with an empty directory for both flowfile and provenance repository.. Please note I have not deleted the content of older provinance repository directory Please find the directory path below For ex: /repository/nifi/nifi-1.16.2/provenanceRepository /repository/nifi/nifi-1.16.2/Flowfilerepository 2: Java version used by nifi: openjdk version "1.8.0_332" OpenJDK Runtime Environment (build 1.8.0_332-b09) OpenJDK 64-Bit Server VM (build 25.332-b09, mixed mode) 4: No other error other then the one which i am sharing in the app log below 3: Please find the app log below: 2023-02-02 08:57:59,845 ERROR [Index Provenance Events-2] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2023-01-27T15:15:54Z, (lock=NativeFSLock(path=/flowfile_repo/nifi/nifi-1.16.2/provenance_repository_clr/lucene-8-index-1674832554761/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2023-01-27T15:15:54.997915Z)) at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191) at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43) at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43) at org.apache.lucene.codecs.lucene80.Lucene80DocValuesConsumer.<init>(Lucene80DocValuesConsumer.java:79) at org.apache.lucene.codecs.lucene80.Lucene80DocValuesFormat.fieldsConsumer(Lucene80DocValuesFormat.java:161) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:227) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:163) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:109) at org.apache.lucene.index.NumericDocValuesWriter.flush(NumericDocValuesWriter.java:108) at org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:345) at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:225) at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350) at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476) at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:656) at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3365) at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3771) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3729) at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.commit(LuceneEventIndexWriter.java:101) at org.apache.nifi.provenance.index.lucene.EventIndexTask.commit(EventIndexTask.java:253) at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:232) ... 6 common frames omitted 2023-02-02 08:57:59,845 ERROR [Index Provenance Events-1] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2023-01-27T15:15:54Z, (lock=NativeFSLock(path=/flowfile_repo/nifi/nifi-1.16.2/provenance_repository_clr/lucene-8-index-1674832554761/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2023-01-27T15:15:54.997915Z)) at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191) at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43) at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43) at org.apache.lucene.codecs.lucene80.Lucene80DocValuesConsumer.<init>(Lucene80DocValuesConsumer.java:79) at org.apache.lucene.codecs.lucene80.Lucene80DocValuesFormat.fieldsConsumer(Lucene80DocValuesFormat.java:161) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:227) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:163) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:109) at org.apache.lucene.index.NumericDocValuesWriter.flush(NumericDocValuesWriter.java:108) at org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:345) at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:225) at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350) at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476) at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:656) at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3365) at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3771) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3729) at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.commit(LuceneEventIndexWriter.java:101) at org.apache.nifi.provenance.index.lucene.EventIndexTask.commit(EventIndexTask.java:253) at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:232) ... 6 common frames omitted 2023-02-02 08:57:59,845 ERROR [Index Provenance Events-2] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2023-01-27T15:15:54Z, (lock=NativeFSLock(path=/flowfile_repo/nifi/nifi-1.16.2/provenance_repository_clr/lucene-8-index-1674832554761/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2023-01-27T15:15:54.997915Z)) at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191) at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43) at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43) at org.apache.lucene.codecs.lucene80.Lucene80DocValuesConsumer.<init>(Lucene80DocValuesConsumer.java:79) at org.apache.lucene.codecs.lucene80.Lucene80DocValuesFormat.fieldsConsumer(Lucene80DocValuesFormat.java:161) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:227) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:163) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:109) at org.apache.lucene.index.NumericDocValuesWriter.flush(NumericDocValuesWriter.java:108) at org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:345) at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:225) at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350) at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476) at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:656) at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3365) at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3771) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3729) at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.commit(LuceneEventIndexWriter.java:101) at org.apache.nifi.provenance.index.lucene.EventIndexTask.commit(EventIndexTask.java:253) at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:232) ... 6 common frames omitted 2023-02-02 08:57:59,845 ERROR [Index Provenance Events-1] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2023-01-27T15:15:54Z, (lock=NativeFSLock(path=/flowfile_repo/nifi/nifi-1.16.2/provenance_repository_clr/lucene-8-index-1674832554761/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2023-01-27T15:15:54.997915Z)) at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191) at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43) at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43) at org.apache.lucene.codecs.lucene80.Lucene80DocValuesConsumer.<init>(Lucene80DocValuesConsumer.java:79) at org.apache.lucene.codecs.lucene80.Lucene80DocValuesFormat.fieldsConsumer(Lucene80DocValuesFormat.java:161) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:227) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:163) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:109) at org.apache.lucene.index.NumericDocValuesWriter.flush(NumericDocValuesWriter.java:108) at org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:345) at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:225) at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350) at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476) at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:656) at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3365) at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3771) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3729) at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.commit(LuceneEventIndexWriter.java:101) at org.apache.nifi.provenance.index.lucene.EventIndexTask.commit(EventIndexTask.java:253) at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:232) ... 6 common frames omitted
... View more
01-30-2023
12:39 AM
Hello @MattWho Thanks to help us, I do create a new folder for provenance and flowfile, We shutdown 1st the nifi services then we create a new folder then we edit the nifi.properties changed both path of provenance and flowfile to the new path. but we still encountered the error. Thanks, AJ
... View more
12-01-2022
02:05 AM
Hello Anyone, We still encountered this error, anyone can help us on this. Thanks
... View more
11-18-2022
03:11 AM
Hi All Good Day! Just want to share our problem after we upgrade our nifi version from 1.9.2 to 1.16.2. all processor running fine but after ingesting a data, We got an error on the nifi bulletin. Upon checking on the logs this the error. Thank you
... View more
Labels:
- Labels:
-
Apache NiFi
09-09-2022
04:17 AM
Hello Nifi Community, We have integrated our Nifi 1.16.2 with LDAP AD server. We have created an Initial Local Admin (nifi_ldap) and used "composite-configurable-user-group-provider" as user group provider. We also restricted to one particular group of LDAP server (namely "EDH_ML"). But none of the users of this group ("EDH_ML") is able to access the Nifi and getting "Insufficient Permission Error". Could someone can help us to resolve this error? -- Sharing nifi screenshot and configuration settings/logs Nifi Users Nifi Login Error Nifi User Policies Authorizer.xml <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Legacy Authorized Users File"></property> <property name="Initial User Identity 1">cn=Service Account\, nifi_ldap,ou=Service Accounts,ou=Xyz Dev,dc=dev,dc=coorporate</property> </userGroupProvider> <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">cn=Service Account\, nifi_ldap,ou=Service Accounts,ou=Xyz Dev,dc=dev,dc=coorporate</property> <property name="Manager Password">pass321</property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://ldap.dev:389</property> <property name="Page Size"></property> <property name="Sync Interval">30 mins</property> <property name="Group Membership - Enforce Case Sensitivity">false</property> <property name="User Search Base">dc=dev,dc=coorporate</property> <property name="User Object Class">user</property> <property name="User Search Scope">SUBTREE</property> <property name="User Search Filter">(|(memberof=cn=EDH_ML,ou=Groups - Applications,ou=Groups,ou=Xyz Dev,dc=dev,dc=coorporate))</property> <property name="User Identity Attribute">cn</property> <property name="User Group Name Attribute">memberOf</property> <property name="User Group Name Attribute - Referenced Group Attribute"></property> <property name="Group Search Base">ou=Groups - Applications,ou=Groups,ou=Xyz Dev,dc=dev,dc=coorporate</property> <property name="Group Object Class">group</property> <property name="Group Search Scope">SUBTREE</property> <property name="Group Search Filter">(|(cn=EDH_ML))</property> <property name="Group Name Attribute">cn</property> <property name="Group Member Attribute">member</property> <property name="Group Member Attribute - Referenced User Attribute"></property> </userGroupProvider> <userGroupProvider> <identifier>composite-configurable-user-group-provider</identifier> <class>org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider</class> <property name="Configurable User Group Provider">file-user-group-provider</property> <property name="User Group Provider 1">ldap-user-group-provider</property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">composite-configurable-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">cn=Service Account\, nifi_ldap,ou=Service Accounts,ou=Xyz Dev,dc=dev,dc=coorporate</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1"></property> <property name="Node Group"></property> </accessPolicyProvider> <authorizer> <identifier>managed-authorizer</identifier> <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class> <property name="Access Policy Provider">file-access-policy-provider</property> </authorizer> <authorizer> <identifier>file-provider</identifier> <class>org.apache.nifi.authorization.FileAuthorizer</class> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Users File">./conf/users.xml</property> <property name="Initial Admin Identity">cn=Service Account\, nifi_ldap,ou=Service Accounts,ou=Xyz Dev,dc=dev,dc=coorporate</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1"></property> </authorizer> <authorizer> <identifier>single-user-authorizer</identifier> <class>org.apache.nifi.authorization.single.user.SingleUserAuthorizer</class> </authorizer> login-identity-providers.xml <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">cn=Service Account\, nifi_ldap,ou=Service Accounts,ou=Xyz Dev,dc=dev,dc=coorporate</property> <property name="Manager Password">pass321</property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://ldap.dev:389</property> <property name="User Search Base">dc=dev,dc=coorporate</property> <property name="User Search Filter">sAMAccountName={0}</property> <property name="Identity Strategy">USE_DN</property> <property name="Authentication Expiration">12 hours</property> </provider> nifi-user.log 2022-09-08 14:17:25,082 INFO [NiFi Web Server-19] org.apache.nifi.web.api.AccessResource Logout Started [cn=User_LN\, User_FN,ou=abcde,ou=Users,ou=coorporate,dc=dev,dc=coorporate] 2022-09-08 14:17:25,102 INFO [NiFi Web Server-186] org.apache.nifi.web.api.AccessResource Logout Request [97418afe-fd34-4cee-b788-0b9ade8a7fb4] Completed [cn=User_LN\, User_FN,ou=abcde,ou=Users,ou=coorporate,dc=dev,dc=coorporate] 2022-09-08 14:17:28,208 INFO [NiFi Web Server-145] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 127.0.0.1 [<anonymous>] GET https://localhost:8080/nifi-api/flow/current-user 2022-09-08 14:17:28,208 WARN [NiFi Web Server-145] o.a.n.w.s.NiFiAuthenticationFilter Authentication Failed 127.0.0.1 GET https://localhost:8080/nifi-api/flow/current-user [Anonymous authentication has not been configured.] 2022-09-08 14:17:37,864 INFO [NiFi Web Server-194] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[cn=User_LN\, User_FN,ou=abcde,ou=Users,ou=coorporate,dc=dev,dc=coorporate], groups[] does not have permission to access the requested resource. Unknown user with identity 'cn=User_LN\, User_FN,ou=abcde,ou=Users,ou=coorporate,dc=dev,dc=coorporate'. Returning Forbidden response. 2022-09-08 14:17:42,240 INFO [NiFi Web Server-145] org.apache.nifi.web.api.AccessResource Logout Started [cn=User_LN\, User_FN,ou=abcde,ou=Users,ou=coorporate,dc=dev,dc=coorporate] 2022-09-08 14:17:42,253 INFO [NiFi Web Server-153] org.apache.nifi.web.api.AccessResource Logout Request [b3ebfab9-4149-4d02-a65d-4b59907a0a67] Completed [cn=User_LN\, User_FN,ou=abcde,ou=Users,ou=coorporate,dc=dev,dc=coorporate] 2022-09-08 14:17:44,325 INFO [NiFi Web Server-194] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 127.0.0.1 [<anonymous>] GET https://localhost:8080/nifi-api/flow/current-user 2022-09-08 14:17:44,325 WARN [NiFi Web Server-194] o.a.n.w.s.NiFiAuthenticationFilter Authentication Failed 127.0.0.1 GET https://localhost:8080/nifi-api/flow/current-user [Anonymous authentication has not been configured.] 2022-09-08 14:18:19,841 INFO [NiFi Web Server-153] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[cn=User_LN\, User_FN,ou=abcde,ou=Users,ou=coorporate,dc=dev,dc=coorporate], groups[] does not have permission to access the requested resource. Unknown user with identity 'cn=User_LN\, User_FN,ou=abcde,ou=Users,ou=coorporate,dc=dev,dc=coorporate'. Returning Forbidden response. Thanks, Alvin
... View more
Labels:
- Labels:
-
Apache NiFi
09-02-2022
09:27 AM
Thanks André. Our Application is running as http not https. See the result below. [nifi1 ~]$ openssl s_client -connect ldap.dev.abcde:389 CONNECTED(00000003) write:errno=104 --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 289 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1662113057 Timeout : 300 (sec) Verify return code: 0 (ok) --- Please advise if there something we need to install/configure in our application Thanks, Alvin
... View more
09-02-2022
03:27 AM
Thanks Andr é . Our Application is running as http not https. See the result below. [nifi1 ~]$ openssl s_client -connect ldap.dev.abcde:389 CONNECTED(00000003) write:errno=104 --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 289 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1662113057 Timeout : 300 (sec) Verify return code: 0 (ok) --- Please advise if there something we need to install/configure in our application Thanks, Alvin
... View more
09-01-2022
04:40 AM
Hi Everyone We trying to integrate LDAP to our existing nifi server single node. After configuring ldap details, Nifi fails with timeout connection error. There is no firewall block and we are able to telnet the ldap server. Please see our setup ldap below. LDAP server : CN=Service Account\, nifi_ldap,OU=Service Accounts,OU=Enterprise Dev,DC=dev,DC=abcde ldap://ldap.dev.abcde:389 password: 5$qwerty! nifi.properties *Previously our application was running as http but we change it to https* nifi.authorizer.configuration.file=./conf/authorizers.xml nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml # web properties # ############################################# # For security, NiFi will present the UI on 127.0.0.1 and only be accessible through this loopback interface. # Be aware that changing these properties may affect how your instance can be accessed without any restriction. # We recommend configuring HTTPS instead. The administrators guide provides instructions on how to do this. nifi.web.war.directory=./lib nifi.web.http.host= nifi.web.http.port= nifi.web.http.network.interface.default= ############################################# nifi.web.https.host=localhost nifi.web.https.port=8080 nifi.web.https.network.interface.default= nifi.web.jetty.working.directory=./work/jetty nifi.web.jetty.threads=200 nifi.web.max.header.size=16 KB nifi.web.proxy.context.path= nifi.web.proxy.host= nifi.web.max.content.size= nifi.web.max.requests.per.second=30000 nifi.web.max.access.token.requests.per.second=25 nifi.web.request.timeout=60 secs nifi.web.request.ip.whitelist= nifi.web.should.send.server.version=true nifi.web.request.log.format=%{client}a - %u %t "%r" %s %O "%{Referer}i" "%{User-Agent}i" # security properties # nifi.sensitive.props.key=1234pwd nifi.sensitive.props.key.protected= nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL nifi.sensitive.props.provider=BC nifi.sensitive.props.additional.keys= nifi.security.autoreload.enabled=false nifi.security.autoreload.interval=10 secs nifi.security.keystore= nifi.security.keystoreType= nifi.security.keystorePasswd= nifi.security.keyPasswd= nifi.security.truststore= nifi.security.truststoreType= nifi.security.truststorePasswd= nifi.security.user.authorizer=managed-authorizer nifi.security.allow.anonymous.authentication=false nifi.security.user.login.identity.provider=ldap-provider nifi.security.user.jws.key.rotation.period=PT1H nifi.security.ocsp.responder.url= nifi.security.ocsp.responder.certificate= login-identity-providers.xml: <loginIdentityProviders> <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">CN=Service Account\, nifi_ldap,OU=Service Accounts,OU=Enterprise Dev,DC=dev,DC=abcde</property> <property name="Manager Password">5$qwerty!</property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">300 secs</property> <property name="Read Timeout">300 secs</property> <property name="Url">ldap://ldap.dev.abcde:389</property> <property name="User Search Base">DC=dev,DC=abcde</property> <property name="User Search Filter"></property> <property name="Identity Strategy">USE_DN</property> <property name="Authentication Expiration">12 hours</property> </provider> </loginIdentityProviders> authorizers.xml: <authorizers> <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">CN=Service Account\, nifi_ldap,OU=Service Accounts,OU=Enterprise Dev,DC=dev,DC=abcde</property> <property name="Manager Password">5$qwerty!</property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">300 secs</property> <property name="Read Timeout">300 secs</property> <property name="Url">ldap://ldap.dev.abcde:389</property> <property name="Page Size"></property> <property name="Sync Interval">30 mins</property> <property name="Group Membership - Enforce Case Sensitivity">false</property> <property name="User Search Base">DC=dev,DC=abcde</property> <property name="User Object Class">person</property> <property name="User Search Scope">ONE_LEVEL</property> <property name="User Search Filter"></property> <property name="User Identity Attribute"></property> <property name="User Group Name Attribute"></property> <property name="User Group Name Attribute - Referenced Group Attribute"></property> <property name="Group Search Base">DC=dev,DC=abcde</property> <property name="Group Object Class">group</property> <property name="Group Search Scope">ONE_LEVEL</property> <property name="Group Search Filter"></property> <property name="Group Name Attribute"></property> <property name="Group Member Attribute">member</property> <property name="Group Member Attribute - Referenced User Attribute"></property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">ldap-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">CN=Service Account\, nifi_ldap,OU=Service Accounts,OU=Enterprise Dev,DC=dev,DC=abcde</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1"></property> <property name="Node Group"></property> </accessPolicyProvider> <authorizer> <identifier>managed-authorizer</identifier> <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class> <property name="Access Policy Provider">file-access-policy-provider</property> </authorizer> </authorizers> error in nifi-app.log: 2022-09-01 06:16:24,995 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=412ms 2022-09-01 06:16:24,997 INFO [main] o.e.j.s.handler.ContextHandler._nifi_api No Spring WebApplicationInitializer types detected on classpath 2022-09-01 06:16:25,060 INFO [main] o.e.j.s.handler.ContextHandler._nifi_api Initializing Spring root WebApplicationContext 2022-09-01 06:16:26,607 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loading Application Properties [/nifi/nifi-1.16.2-ldap/./conf/nifi.properties] 2022-09-01 06:18:34,427 ERROR [main] o.s.web.context.ContextLoader Context initialization failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'niFiWebApiConfiguration': BeanPostProcessor before instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'metaDataSourceAdvisor': Cannot resolve reference to bean 'methodSecurityMetadataSource' while setting constructor argument; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.method.configuration.GlobalMethodSecurityConfiguration': Unsatisfied dependency expressed through method 'setObjectPostProcessor' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.security.configuration.AuthenticationSecurityConfiguration': Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is org.springframework.ldap.CommunicationException: ldap.dev.abcde:389; nested exception is javax.naming.CommunicationException: ldap.dev.abcde:389 [Root exception is java.net.ConnectException: Connection timed out (Connection timed out)] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:537) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:953) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572) at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746) Note: Post nifi initialization fail , autorziation.xml and user.xml are not generated in below conf directory : drwxr-xr-x 2 root root 8192 Sep 1 05:41 archive -rw-r--r-- 1 root root 27294 Sep 1 06:14 authorizers.xml -rw-r--r-- 1 root root 1320 Sep 1 05:23 bootstrap-aws.conf -rw-r--r-- 1 root root 1076 Sep 1 05:23 bootstrap-azure.conf -rw-r--r-- 1 root root 5942 Sep 1 05:23 bootstrap.conf -rw-r--r-- 1 root root 952 Sep 1 05:23 bootstrap-gcp.conf -rw-r--r-- 1 root root 2189 Sep 1 05:23 bootstrap-hashicorp-vault.conf -rw-r--r-- 1 root root 2326 Sep 1 05:23 bootstrap-notification-services.xml -rw-r--r-- 1 root root 5140762 Sep 1 05:41 flow.json.gz -rw-r--r-- 1 root root 8114138 Sep 1 05:41 flow.xml.gz -rw-r--r-- 1 root root 10549 Sep 1 05:34 logback.xml -rw-r--r-- 1 root root 7265 Sep 1 06:13 login-identity-providers.xml -rw-r--r-- 1 root root 16524 Sep 1 05:23 nifi-backup.properties -rw-r--r-- 1 root root 17096 Sep 1 06:01 nifi.properties -rw-r--r-- 1 root root 11402 Sep 1 05:23 nifi-toolkit.properties -rw-r--r-- 1 root root 3696 Sep 1 05:23 stateless-logback.xml -rw-r--r-- 1 root root 1946 Sep 1 05:23 stateless.properties -rw-r--r-- 1 root root 9120 Sep 1 05:23 state-management.xml -rw-r--r-- 1 root root 2573 Sep 1 05:23 zookeeper.properties I checked Integration of NiFi with LDAP – Pierre Villard NiFi System Administrator’s Guide (apache.org) but it didn't worked. Please guide me to integrate LDAP with Nifi. Thanks, Alvin
... View more
Labels:
- Labels:
-
Apache NiFi
09-01-2022
03:22 AM
Thank you Matt sorry for the super delay response but it's very helpful
... View more
06-24-2022
08:26 AM
Hi All, We are planning to upgrade our Nifi application(Linux platform) from 1.9.2 version to 1.16.2.Before proceeding with this major version jump , we wanted to check for any possible issues and backward compatibility . We do not see entry for version jump from 1.9.x to 1.16.x in below : https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance We request your guidance for anything we need to be aware of before we make jump from Nifi 1.9.2 to 1.16.2 Thanks, Alvin
... View more
Labels:
- Labels:
-
Apache NiFi
