Member since
03-22-2019
24
Posts
1
Kudos Received
0
Solutions
06-03-2020
08:58 AM
In order to use an ACL that indicates that only the Creator is allowed to access the data, we need to tell ZooKeeper who the Creator is. There are two mechanisms for accomplishing this. The first mechanism is to provide authentication using Kerberos The second option is to use a user name and password. This is configured by specifying a value for the Username and a value for the Password properties for the ZooKeeperStateProvider For the Second Option, where do I specify the UserName and Password? which properties file
... View more
Labels:
05-18-2020
04:32 PM
How to connect to NIFI GetMongo using x509 connection with certificates ?
... View more
- Tags:
- NiFi
Labels:
03-18-2020
10:54 AM
Hi, I am using Query Solr and my SOLR Location property does not support flow file attribute scope, it only supports Variable Registry, Since my Location URL can be Dynamic based on the locale, how can I resolve this? example : https://localhost:8983/solr/${collectionName} https://localhost:8983/solr/content_en_1_0 https://localhost:8983/solr/content_zh_1_0 https://localhost:8983/solr/content_ja_1_0 Thanks
... View more
- Tags:
- Nifi-1.10.0
Labels:
11-25-2019
10:13 AM
It is working fine now with 3.5.6 version of zookeeper
... View more
11-18-2019
11:32 AM
Thanks @Shelton , will give it a try and update here later.
... View more
11-17-2019
02:48 PM
Upgrading to Java 11 is a mandate across our ORG. I was able to compile ( On MAC with Java 11) and seems to be working fine. however see above issue when installing Nifi-1.10.0 with external zookeeper.
... View more
11-17-2019
11:19 AM
Can anyone please help ?
... View more
11-15-2019
08:48 AM
Hi,
I am having issues with the zookeeper (zookeeper-3.4.13) with latest nifi version of Nifi-1.10.0 , below is the stack trace. I am upgrading from nifi-1.9.0 . let me know if latest version of Nifi needs different version of zookeeper ? or the issue is somewhere else.
Not : With Java 8 and nifi-1.9.0 everything works
Actual Logs using Java 11
2019-11-15 03:40:09,202 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED 2019-11-15 03:40:09,241 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting 2019-11-15 03:40:09,355 INFO [main] o.apache.nifi.controller.FlowController It appears that no Cluster Coordinator has been Elected yet. Registering for Cluster Coordinator Role. 2019-11-15 03:40:09,356 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election. 2019-11-15 03:40:09,360 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2019-11-15 03:40:09,361 INFO [main] org.apache.zookeeper.ClientCnxnSocket jute.maxbuffer value is 4194304 Bytes 2019-11-15 03:40:09,365 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Default schema 2019-11-15 03:40:09,371 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election. 2019-11-15 03:40:09,372 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started 2019-11-15 03:40:09,372 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started 2019-11-15 03:40:09,375 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED 2019-11-15 03:40:09,387 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@43eff6be Connection State changed to CONNECTED 2019-11-15 03:40:09,540 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2019-11-15 03:40:09,540 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@43eff6be Connection State changed to SUSPENDED 2019-11-15 03:40:10,490 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED 2019-11-15 03:40:10,490 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@43eff6be Connection State changed to RECONNECTED 2019-11-15 03:40:10,634 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2019-11-15 03:40:10,636 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@43eff6be Connection State changed to SUSPENDED 2019-11-15 03:40:10,835 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED 2019-11-15 03:40:10,841 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@43eff6be Connection State changed to RECONNECTED 2019-11-15 03:40:10,952 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2019-11-15 03:40:10,959 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@43eff6be Connection State changed to SUSPENDED 2019-11-15 03:40:11,017 ERROR [main-EventThread] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:647) at org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:152) at org.apache.curator.framework.imps.FindAndDeleteProtectedNodeInBackground$2.processResult(FindAndDeleteProtectedNodeInBackground.java:104) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:630) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2019-11-15 03:40:11,450 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED 2019-11-15 03:40:11,455 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@43eff6be Connection State changed to RECONNECTED 2019-11-15 03:40:11,551 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2019-11-15 03:40:11,552 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@43eff6be Connection State changed to SUSPENDED 2019-11-15 03:40:11,552 ERROR [main-EventThread] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
Comparision of Java 8 with Nifi-1.9.0 and Java11 Nifi-1.10.0 logs below
===================== JAVA 8 working ================================================ 2019-11-15 18:07:44,302 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@196ca821 finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates 2019-11-15 18:07:44,303 INFO [main] org.wali.MinimalLockingWriteAheadLog Successfully recovered 1800 records in 18 milliseconds 2019-11-15 18:07:44,330 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@196ca821 checkpointed with 1800 Records and 0 Swap Files in 26 milliseconds (Stop-the-world time = 4 milliseconds, Clear Edit Logs time = 3 millis), max Transaction ID 137772318 2019-11-15 18:07:45,253 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator Resetting cluster node statuses from {} to {rn-ssenpcit-lapp205.rno.apple.com:8086=NodeConnectionStatus[nodeId=rn-ssenpcit-lapp205.rno.apple.com:8086, state=DISCONNECTED, Disconnect Code=Has Not Yet Connected to Cluster, Disconnect Reason=Has Not Yet Connected to Cluster, updateId=1], rn-ssenpcit-lapp206.rno.apple.com:8086=NodeConnectionStatus[nodeId=rn-ssenpcit-lapp206.rno.apple.com:8086, state=DISCONNECTED, Disconnect Code=Has Not Yet Connected to Cluster, Disconnect Reason=Has Not Yet Connected to Cluster, updateId=0]} 2019-11-15 18:07:45,363 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded 97 properties from system properties and environment variables 2019-11-15 18:07:45,364 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded a total of 97 properties. Including precedence overrides effective accessible registry key size is 97 2019-11-15 18:07:45,520 INFO [main] o.a.n.c.r.WriteAheadFlowFileRepository Initialized FlowFile Repository using 256 partitions 2019-11-15 18:07:46,240 INFO [main] o.a.n.p.store.WriteAheadStorePartition After recovering ./provenance_repository, next Event ID to be generated will be 148085136 2019-11-15 18:07:47,399 INFO [pool-23-thread-1] o.a.n.p.index.lucene.LuceneEventIndex Determined that Max Event ID indexed for Partition default is approximately 148084987 based on index ./provenance_repository/index-1573685499416 2019-11-15 18:07:47,403 INFO [pool-23-thread-1] o.a.n.p.store.WriteAheadStorePartition The last Provenance Event indexed for partition default is 148074987, but the last event written to partition has ID 148085135. Re-indexing up to the last 10148 events to ensure that the Event Index is accurate and up-to-date 2019-11-15 18:07:49,963 INFO [pool-23-thread-1] o.a.n.p.store.WriteAheadStorePartition Finished re-indexing 10149 events across 1 files for ./provenance_repository in 2.559 seconds 2019-11-15 18:07:49,973 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 35101377658 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size 2019-11-15 18:07:49,975 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false 2019-11-15 18:07:50,935 INFO [main] o.apache.nifi.controller.FlowController Checking if there is already a Cluster Coordinator Elected... 2019-11-15 18:07:51,015 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2019-11-15 18:07:51,340 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED 2019-11-15 18:07:51,362 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting 2019-11-15 18:07:51,370 INFO [main] o.apache.nifi.controller.FlowController The Election for Cluster Coordinator has already begun (Leader is rn-ssenpcit-lapp206.rno.apple.com:11443). Will not register to be elected for this role until after connecting to the cluster and inheriting the cluster's flow. 2019-11-15 18:07:51,372 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is a silent observer in the election. 2019-11-15 18:07:51,373 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2019-11-15 18:07:51,376 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is a silent observer in the election. 2019-11-15 18:07:51,379 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started 2019-11-15 18:07:51,379 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started 2019-11-15 18:07:51,385 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED 2019-11-15 18:07:55,430 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@6ca9f1d0{nifi-api,/nifi-api,file:///ngs/app/ssewiret/nifi/work/jetty/nifi-web-api-1.9.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.9.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.9.0.war} 2019-11-15 18:07:55,831 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=135ms 2019-11-15 18:07:55,862 INFO [main] o.e.j.s.h.C._nifi_content_viewer No Spring WebApplicationInitializer types detected on classpath 2019-11-15 18:07:56,194 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@4a467f08{nifi-content-viewer,/nifi-content-viewer,file:///ngs/app/ssewiret/nifi/work/jetty/nifi-web-content-viewer-1.9.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.9.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-content-viewer-1.9.0.war} 2019-11-15 18:07:56,218 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=9ms 2019-11-15 18:07:56,219 WARN [main] o.e.j.webapp.StandardDescriptorProcessor Duplicate mapping from / to default ======================== JAVA 11 NOT WORKING ============================================ 2019-11-15 20:02:32,893 ERROR [main] org.apache.nifi.encrypt.StringEncryptor ******************************************************************************** 2019-11-15 20:02:32,996 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@56c856c0 finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates 2019-11-15 20:02:32,996 INFO [main] org.wali.MinimalLockingWriteAheadLog Successfully recovered 1800 records in 31 milliseconds 2019-11-15 20:02:33,015 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@56c856c0 checkpointed with 1800 Records and 0 Swap Files in 19 milliseconds (Stop-the-world time = 3 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID 137772323 2019-11-15 20:02:33,517 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator Resetting cluster node statuses from {} to {rn-ssenpcit-lapp205.rno.apple.com:8086=NodeConnectionStatus[nodeId=rn-ssenpcit-lapp205.rno.apple.com:8086, state=DISCONNECTED, Disconnect Code=Has Not Yet Connected to Cluster, Disconnect Reason=Has Not Yet Connected to Cluster, updateId=1], rn-ssenpcit-lapp206.rno.apple.com:8086=NodeConnectionStatus[nodeId=rn-ssenpcit-lapp206.rno.apple.com:8086, state=DISCONNECTED, Disconnect Code=Has Not Yet Connected to Cluster, Disconnect Reason=Has Not Yet Connected to Cluster, updateId=0]} 2019-11-15 20:02:33,569 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded 95 properties from system properties and environment variables 2019-11-15 20:02:33,569 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded a total of 95 properties. Including precedence overrides effective accessible registry key size is 95 2019-11-15 20:02:33,629 INFO [main] o.a.n.c.r.WriteAheadFlowFileRepository Initialized FlowFile Repository using 256 partitions 2019-11-15 20:02:33,768 INFO [main] o.a.n.p.store.WriteAheadStorePartition After recovering ./provenance_repository, next Event ID to be generated will be 148085136 2019-11-15 20:02:33,835 WARN [main] o.a.n.p.index.lucene.LuceneEventIndex Lucene Index ./provenance_repository/index-1573685499416 could not be opened. Assuming that index is defunct and will re-index events belonging to this index. 2019-11-15 20:02:33,835 INFO [main] o.a.n.p.index.lucene.LuceneEventIndex Determined that Lucene Index Directory ./provenance_repository/index-1573685499416 is defunct. Will destroy and rebuild index 2019-11-15 20:02:33,839 INFO [main] o.a.n.p.index.lucene.LuceneEventIndex Will avoid re-indexing Provenance Events because the newest index is defunct, so it will be re-indexed in the background 2019-11-15 20:02:33,839 INFO [main] o.a.n.p.WriteAheadProvenanceRepository Provenance Event Index indicates that no events should be re-indexed upon startup. Will not wait for re-indexing to occur. 2019-11-15 20:02:33,840 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 35101377658 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size 2019-11-15 20:02:33,842 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false 2019-11-15 20:02:34,199 INFO [main] o.apache.nifi.controller.FlowController Checking if there is already a Cluster Coordinator Elected... 2019-11-15 20:02:34,328 INFO [main] org.apache.curator.utils.Compatibility Using emulated InjectSessionExpiration 2019-11-15 20:02:34,456 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2019-11-15 20:02:34,505 INFO [main] org.apache.zookeeper.common.X509Util Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2019-11-15 20:02:34,547 INFO [main] org.apache.zookeeper.ClientCnxnSocket jute.maxbuffer value is 4194304 Bytes 2019-11-15 20:02:34,743 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Default schema 2019-11-15 20:02:34,983 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED 2019-11-15 20:02:35,083 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting 2019-11-15 20:02:35,219 INFO [main] o.apache.nifi.controller.FlowController It appears that no Cluster Coordinator has been Elected yet. Registering for Cluster Coordinator Role. 2019-11-15 20:02:35,220 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election. 2019-11-15 20:02:35,223 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2019-11-15 20:02:35,236 INFO [main] org.apache.zookeeper.ClientCnxnSocket jute.maxbuffer value is 4194304 Bytes 2019-11-15 20:02:35,250 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Default schema 2019-11-15 20:02:35,257 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election. 2019-11-15 20:02:35,257 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started 2019-11-15 20:02:35,257 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started 2019-11-15 20:02:35,283 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED 2019-11-15 20:02:35,290 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@4d490d2c Connection State changed to CONNECTED 2019-11-15 20:02:35,565 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2019-11-15 20:02:35,568 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@4d490d2c Connection State changed to SUSPENDED 2019-11-15 20:02:36,567 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED 2019-11-15 20:02:36,567 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@4d490d2c Connection State changed to RECONNECTED 2019-11-15 20:02:36,680 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2019-11-15 20:02:36,691 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@4d490d2c Connection State changed to SUSPENDED 2019-11-15 20:02:36,729 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED 2019-11-15 20:02:36,730 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@4d490d2c Connection State changed to RECONNECTED 2019-11-15 20:02:36,859 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
... View more
Labels:
11-14-2019
09:16 PM
1 Kudo
Yes , it was picking up openJDK 9 which was in the classpath, i pointed to right JDK 11 and it is working now.
... View more
11-14-2019
01:38 PM
I have installed latest version of Nifi-1.10.0 , everything works locally , however on my Linux server i am having issues when i start NIFI , below is the stack trace.
2019-11-14 21:22:31,464 INFO [main] org.apache.nifi.NiFi Launching NiFi... 2019-11-14 21:22:31,812 INFO [main] org.apache.nifi.security.kms.CryptoUtils Determined default nifi.properties path to be '/ngs/app/ssewiret/nifi/./conf/nifi.properties' 2019-11-14 21:22:31,818 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 143 properties from /ngs/app/ssewiret/nifi/./conf/nifi.properties 2019-11-14 21:22:31,826 INFO [main] org.apache.nifi.NiFi Loaded 143 properties 2019-11-14 21:22:31,838 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 33317 2019-11-14 21:22:31,862 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap 2019-11-14 21:22:31,909 INFO [main] org.apache.nifi.nar.NarUnpacker Expanding 198 NAR files with all processors... 2019-11-14 21:22:31,933 WARN [main] org.apache.nifi.nar.NarUnpacker Unable to load NAR library bundles due to java.util.zip.ZipException: zip END header not found Will proceed without loading any further Nar bundles 2019-11-14 21:22:31,942 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.lang.IllegalStateException: Unable to locate Jetty bundle. java.lang.IllegalStateException: Unable to locate Jetty bundle. at org.apache.nifi.nar.NarClassLoaders.load(NarClassLoaders.java:211) at org.apache.nifi.nar.NarClassLoaders.init(NarClassLoaders.java:119) at org.apache.nifi.NiFi.<init>(NiFi.java:134) at org.apache.nifi.NiFi.<init>(NiFi.java:72) at org.apache.nifi.NiFi.main(NiFi.java:301) 2019-11-14 21:22:31,944 INFO [Thread-0] org.apache.nifi.NiFi Initiating shutdown of Jetty web server... 2019-11-14 21:22:31,946 INFO [Thread-0] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
Thanks for the help.
... View more
Labels:
11-06-2019
03:54 PM
Hi,
We are moving to Kafka 2.2.1 version ,I wanted to know if PublishKafka2_0 would be compatible with 2.2.1 version of Kafka, if not can i change the version in POM.xml
From < kafka2.0.version > 2.0.0 </ kafka2.0.version > To < kafka2.0.version > 2.2.1 </ kafka2.0.version > And rebuild the source to make it work, or do i need to make any code changes?
... View more
Labels:
07-11-2019
07:48 PM
I am still facing the same issue any help would be appreciated.
... View more
07-01-2019
05:55 PM
@Jay Kumar SenSharma Below are the details : ~ git:(master) ✗ grep 'nifi-web-ui' /Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/logs/nifi-app.log 2019-06-30 21:41:27,762 INFO [main] org.apache.nifi.web.server.JettyServer Loading WAR: /Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/./work/nar/framework/nifi-framework-nar-1.9.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-ui-1.9.0.war with context path set to /nifi 2019-06-30 21:41:42,074 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@509e4bc4{nifi,/nifi,file:///Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/work/jetty/nifi-web-ui-1.9.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.9.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-ui-1.9.0.war} ~ git:(master) ✗ ls -lart /Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/work/nar/framework/nifi-framework-nar-1.9.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.9.0.war -rw-r--r-- 1 samaraarkotti staff 950040 Feb 21 17:48 /Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/work/nar/framework/nifi-framework-nar-1.9.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.9.0.war Attached are the logs: nifi-app.txt nifi-bootstrap.txt nifi-user.txt
... View more
06-29-2019
02:56 AM
can anyone really help ?
... View more
06-28-2019
04:01 PM
@Jim Streuli were you able to resolve this issue? , i have upgraded my Mac OS to Catalina and see the same issue. getting HTTP ERROR 404.
... View more
06-28-2019
05:52 AM
re-posting.. didn't added it as comment. Below are the CURL Commands output. CurlCommands.txt ➜ ~ git:(master) ✗ netstat -an | grep 8080 tcp4 0 0 *.8080 *.* LISTEN ➜ ~ git:(master) ✗ ps -ef | grep nifi 501 19949 1 0 9:57PM ttys001 0:00.00 /bin/sh ./nifi.sh start 501 19951 19949 0 9:57PM ttys001 0:01.29 /Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/bin/java -cp /Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/conf:/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/lib/bootstrap/* -Xms12m -Xmx24m -Dorg.apache.nifi.bootstrap.config.log.dir=/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/logs -Dorg.apache.nifi.bootstrap.config.pid.dir=/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/run -Dorg.apache.nifi.bootstrap.config.file=/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/conf/bootstrap.conf org.apache.nifi.bootstrap.RunNiFi start 501 19952 19951 0 9:57PM ttys001 2:18.73 /Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/bin/java -classpath /Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/./conf:/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/./lib/nifi-properties-1.9.0.jar:/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/./lib/jetty-schemas-3.1.jar:/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/./lib/slf4j-api-1.7.25.jar:/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/./lib/nifi-framework-api-1.9.0.jar:/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/./lib/javax.servlet-api-3.1.0.jar:/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/./lib/jcl-over-slf4j-1.7.25.jar:/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/./lib/logback-classic-1.2.3.jar:/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/./lib/jul-to-slf4j-1.7.25.jar:/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/./lib/nifi-runtime-1.9.0.jar:/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/./lib/log4j-over-slf4j-1.7.25.jar:/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/./lib/logback-core-1.2.3.jar:/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/./lib/nifi-nar-utils-1.9.0.jar:/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/./lib/nifi-api-1.9.0.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx2048M -Xms2048M -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -XX:+UseG1GC -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/./conf/nifi.properties -Dnifi.bootstrap.listen.port=58309 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/Users/samaraarkotti/Documents/example-nifi-1.9.0-deploy/nifi-1.9.0/logs org.apache.nifi.NiFi 501 19975 1382 0 9:57PM ttys001 0:00.03 tail -f ../logs/nifi-app.log 501 20666 14587 0 10:06PM ttys002 0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn nifi
... View more
06-28-2019
04:10 AM
I have downloaded Nifi on my laptop running Mac OS Catalina Public Beta 2 and not able to access the UI. It is up and running and able to access all the REST API's shown below , http://127.0.0.1:8080/nifi-api/flow/processor-types http://127.0.0.1:8080/nifi-api/flow/controller-service-types http://127.0.0.1:8080/nifi-api/flow/reporting-task-types http://127.0.0.1:8080/nifi-api/controller/config http://127.0.0.1:8080/nifi-api/resources http://127.0.0.1:8080/nifi-api/system-diagnostics However UI is not accessible , getting below error. Note this is very specific to Mac OS Catalina public beta. it is working fine in Mojave . Nothing stands out in any of the logs. HTTP ERROR 404 Problem accessing /nifi/. Reason: Not Found Powered by Jetty:// 9.4.11.v20180605 For testing purpose , i have downloaded Nifi-Registry and it is also having the same issue.
... View more
Labels:
04-10-2019
06:41 PM
Thanks @Matt Clarke i will set the configure concurrent task for this processor to be 1 , my rest of the processor's in the flow are set to 2 concurrent task's . which should be fine i guess.
... View more
04-10-2019
05:10 PM
I am seeing below messages and it is messing up the SRC_OFFSET in this case, i have configured UpdateAttribute with 2 concurrent Tasks. mine is a 3 node cluster, Nifi version 1.9.0
... View more
Labels:
04-01-2019
08:43 PM
https://community.hortonworks.com/articles/54275/apache-nifi-100-kerberos-authentication.html I am trying to make kerberos work with Spnego following steps from above link , everything works except the spnego part. the UI alway's presents login page, how can i avoid the login page ( don't want to provide username / password ) , tried this numerous times with firefox and other browsers as well. ( what i have observed is , if you login using username/password for first time , and if you clear browser cache launch nifi it will not show login page until you do kinit again , once you do kinit it will present the login screen again next time you launch nifi) docker-machine create --driver virtualbox boot2docker eval "$(docker-machine env boot2docker)" ./kdc stop ./kdc clean ./kdc build ./kdc start ./kdc test $(./kdc shellinit) kinit samar@NIFI.APACHE.ORG open /Applications/Firefox.app from the terminal my principal in kdc.json : {
"principals": [
{
"id": "samar@NIFI.APACHE.ORG",
"password": "samar"
},
{
"id": "HTTP/localhost@NIFI.APACHE.ORG",
"password": "http"
}
],
"domain": "localhost",
"realm": "NIFI.APACHE.ORG",
"ip": "127.0.0.1",
"port": 48088
} FireFox Negotiate Settings network.negotiate-auth.delegation-uris : https://localhost:9443
network.negotiate-auth.trusted-uris : https://localhost:9443 Nifi.properties changes nifi.remote.input.secure=true
nifi.web.https.host=localhost
nifi.web.https.port=9443
nifi.remote.input.secure=true
nifi.security.user.login.identity.provider=kerberos-provider
nifi.kerberos.krb5.file=/Users/samar/Downloads/docker-kdc/krb5.conf
nifi.kerberos.spnego.principal=HTTP/localhost@NIFI.APACHE.ORG
nifi.kerberos.spnego.keytab.location=/Users/samar/Downloads/docker-kdc/krb5.keytab
nifi.kerberos.spnego.authentication.expiration=12 hours uncomment below properties : nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$
nifi.security.identity.mapping.value.dn=$1@$2
nifi.security.identity.mapping.transform.dn=NONE
nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
nifi.security.identity.mapping.value.kerb=$1@$2 login-identity-providers.xml changes <provider>
<identifier>kerberos-provider</identifier>
<class>org.apache.nifi.kerberos.KerberosProvider</class>
<property name="Default Realm">NIFI.APACHE.ORG</property>
<property name="Authentication Expiration">12 hours</property>
</provider> Authorizers.xml changes <authorizer>
<identifier>file-provider</identifier>
<class>org.apache.nifi.authorization.FileAuthorizer</class>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Users File">./conf/users.xml</property>
<property name="Initial Admin Identity">samar@NIFI.APACHE.ORG</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1"></property>
</authorizer> Am i missing something here ? let me know who to avoid providing username/password
... View more
Labels:
04-01-2019
04:07 PM
Thanks @kdoran currently i am doing POC in development environment itself, and Spnego is not working, tried various methods to make it work. deleting the authorizations.xml , users.xml, flowfiles archive ...etc and restarted everything again. but spnego is not working. when i launch nifi it is alway's showing login page. tried with all browsers. am i missing something here ? posted the same question seperately : https://community.hortonworks.com/questions/243723/kerberos-spnego-authentication.html
... View more
03-30-2019
02:48 AM
My SPNEGO is not working , my settings are exactly like @Scott Aslan what i have observed is , if you login using username/password for first time , and if you clear browser cache launch nifi it will not show login page until you do kinit again , once you do kinit it will present the login screen again next time you launch nifi , I have always launch from terminal after seeing this post @kdoran. how can i avoid login page using spnego ? using firefox, and nifi version 1.9.0
... View more
03-23-2019
12:56 AM
https://community.hortonworks.com/articles/54275/apache-nifi-100-kerberos-authentication.html I have followed instructions shown from above link : I am testing this on my Mac OSX, using Nifi Version 1.9.0 ➜ docker-kdc git:(master) ✗ ./kdc test
Checking boot2docker active... ok
Trying to connect to KDC... at 192.168.99.103:48088... ok klist command is not showing the principal i have configured which is samar@NIFI.APACHE.ORG this is my kdc.json {
"principals": [
{
"id": "samar@NIFI.APACHE.ORG",
"password": "samar"
},
{
"id": "HTTP/nifi.apache.org@NIFI.APACHE.ORG",
"password": "http"
}
],
"domain": "nifi.apache.org",
"realm": "NIFI.APACHE.ORG",
"ip": "127.0.0.1",
"port": 48088
} I have updated /etc/hosts as below #127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost
127.0.0.1 localhost nifi.apache.org when i hit the https://nifi.apache.org:9445/nifi , i see below logs in nifi-app.log 2019-03-22 15:17:45,626 INFO [main] org.apache.nifi.web.server.JettyServer https://localhost:9445/nifi
2019-03-22 15:17:45,627 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap
2019-03-22 15:17:45,628 INFO [main] org.apache.nifi.NiFi Controller initialization took 19795119826 nanoseconds (19 seconds).
2019-03-22 15:19:42,144 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@27a57c66 checkpointed with 4 Records and 0 Swap Files in 13 milliseconds (Stop-the-world time = 5 milliseconds, Clear Edit Logs time = 5 millis), max Transaction ID 11
2019-03-22 15:19:42,391 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2019-03-22 15:19:42,392 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2019-03-22 15:19:55,074 WARN [NiFi Web Server-22] o.a.nifi.web.server.HostHeaderHandler Request host header [nifi.apache.org:9445] different from web hostname [localhost(:9445)]. Overriding to [localhost:9445/nifi]
2019-03-22 15:21:23,079 WARN [NiFi Web Server-89] o.a.nifi.web.server.HostHeaderHandler Request host header [nifi.apache.org:9445] different from web hostname [localhost(:9445)]. Overriding to [localhost:9445/nifi/]
2019-03-22 15:21:23,706 WARN [NiFi Web Server-20] o.a.nifi.web.server.HostHeaderHandler Request host header [nifi.apache.org:9445] different from web hostname [localhost(:9445)]. Overriding to [localhost:9445/favicon.ico]
and in the browse i see below ERROR. System Error The request contained an invalid host header [ nifi.apache.org:9445 ] in the request [ /nifi ]. Check for request manipulation or third-party intercept. Valid host headers are [ empty ] or: 127.0.0.1 127.0.0.1:9445 localhost localhost:9445 [::1] [::1]:9445 samar-5.local samar-5.local:9445 17.114.29.10 17.114.29.10:9445 Thanks Samar
... View more
