Member since
02-09-2020
4
Posts
0
Kudos Received
0
Solutions
02-11-2020
10:19 PM
Hi , Am a beginner in NiFi and just started to know cluster configurations an all.
Am trying to run Nifi as a 3 node cluster on Windows 10, with the configuration settings, set in all 3 conf files as mentioned below. But still its failing to start NiFi as a 3 node cluster. Please help in creating this cluster configuration
Below are the settings done on my files
state-management.xml file
<cluster-provider> <id>zk-provider</id> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class> <property name="Connect String">node1:2181,node2:2181,node3:2181</property> <property name="Root Node">/nifi</property> <property name="Session Timeout">10 seconds</property> <property name="Access Control">Open</property> </cluster-provider>
zookeper properties
dataDir=./state/zookeeper autopurge.snapRetainCount=30 server.1=node1:2888:3888 server.2=node2:2888:3888 server.3=node3:2888:3888
nifi properties:
nifi.web.http.host=node1 nifi.web.http.port=8080 nifi.cluster.is.node=true nifi.cluster.node.address=node1 nifi.cluster.node.protocol.port=11122 nifi.zookeeper.connect.string=node1:2181,node2:2181,node3:2181 nifi.zookeeper.root.node=/nifi
In other 2 nodes the host name are placed respectively as (node2 , node3)
C:\nifi-1.11.0-bin\Node1\bin>run-nifi.bat 2020-02-11 20:29:20,237 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2020-02-11 20:29:20,238 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: C:\NIFI-1~1.0-B\Node1 2020-02-11 20:29:20,239 INFO [main] org.apache.nifi.bootstrap.Command Command: C:\Program Files\Java\jdk1.8.0_241\bin\java.exe -classpath C:\NIFI-1~1.0-B\Node1\.\conf;C:\NIFI-1~1.0-B\Node1\.\lib\javax.servlet-api-3.1.0.jar;C:\NIFI-1~1.0-B\Node1\.\lib\jcl-over-slf4j-1.7.26.jar;C:\NIFI-1~1.0-B\Node1\.\lib\jetty-schemas-3.1.jar;C:\NIFI-1~1.0-B\Node1\.\lib\jul-to-slf4j-1.7.26.jar;C:\NIFI-1~1.0-B\Node1\.\lib\log4j-over-slf4j-1.7.26.jar;C:\NIFI-1~1.0-B\Node1\.\lib\logback-classic-1.2.3.jar;C:\NIFI-1~1.0-B\Node1\.\lib\logback-core-1.2.3.jar;C:\NIFI-1~1.0-B\Node1\.\lib\nifi-api-1.11.0.jar;C:\NIFI-1~1.0-B\Node1\.\lib\nifi-framework-api-1.11.0.jar;C:\NIFI-1~1.0-B\Node1\.\lib\nifi-nar-utils-1.11.0.jar;C:\NIFI-1~1.0-B\Node1\.\lib\nifi-properties-1.11.0.jar;C:\NIFI-1~1.0-B\Node1\.\lib\nifi-runtime-1.11.0.jar;C:\NIFI-1~1.0-B\Node1\.\lib\slf4j-api-1.7.26.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dzookeeper.admin.enableServer=false -Dnifi.properties.file.path=C:\NIFI-1~1.0-B\Node1\.\conf\nifi.properties -Dnifi.bootstrap.listen.port=53535 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=C:\NIFI-1~1.0-B\Node1\bin\..\\logs org.apache.nifi.NiFi 2020-02-11 20:29:20,554 WARN [main] org.apache.nifi.bootstrap.Command Failed to set permissions so that only the owner can read pid file C:\NIFI-1~1.0-B\Node1\bin\..\run\nifi.pid; this may allows others to have access to the key needed to communicate with NiFi. Permissions should be changed so that only the owner can read this file 2020-02-11 20:29:20,561 WARN [main] org.apache.nifi.bootstrap.Command Failed to set permissions so that only the owner can read status file C:\NIFI-1~1.0-B\Node1\bin\..\run\nifi.status; this may allows others to have access to the key needed to communicate with NiFi. Permissions should be changed so that only the owner can read this file 2020-02-11 20:29:20,573 INFO [main] org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 4284
______________________________
C:\nifi-1.11.0-bin\Node1\bin>status-nifi.bat 20:31:57.265 [main] DEBUG org.apache.nifi.bootstrap.NotificationServiceManager - Found 0 service elements 20:31:57.271 [main] INFO org.apache.nifi.bootstrap.NotificationServiceManager - Successfully loaded the following 0 services: [] 20:31:57.272 [main] INFO org.apache.nifi.bootstrap.RunNiFi - Registered no Notification Services for Notification Type NIFI_STARTED 20:31:57.273 [main] INFO org.apache.nifi.bootstrap.RunNiFi - Registered no Notification Services for Notification Type NIFI_STOPPED 20:31:57.274 [main] INFO org.apache.nifi.bootstrap.RunNiFi - Registered no Notification Services for Notification Type NIFI_DIED 20:31:57.277 [main] DEBUG org.apache.nifi.bootstrap.Command - Status File: C:\NIFI-1~1.0-B\Node1\bin\..\run\nifi.status 20:31:57.278 [main] DEBUG org.apache.nifi.bootstrap.Command - Status File: C:\NIFI-1~1.0-B\Node1\bin\..\run\nifi.status 20:31:57.279 [main] DEBUG org.apache.nifi.bootstrap.Command - Properties: {pid=4284, port=53536} 20:31:57.280 [main] DEBUG org.apache.nifi.bootstrap.Command - Pinging 53536 20:31:58.343 [main] DEBUG org.apache.nifi.bootstrap.Command - Process with PID 4284 is not running 20:31:58.345 [main] INFO org.apache.nifi.bootstrap.Command - Apache NiFi is not running
This is how my folder structure is:
below is the image of Node 2 and node 3 cmd prompt.
... View more
Labels:
- Labels:
-
Apache NiFi
02-11-2020
11:15 AM
@MattWho Hi , Am trying to run Nifi as a 3 node cluster on Windows 10, with same properties set in all 3 conf files as mentioned below, but still its failing to start nifi as a 3 node cluster. Please help in creating this cluster configuration Below are the settings done on my files state-management.xml file <cluster-provider> <id>zk-provider</id> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class> <property name="Connect String">node1:2181,node2:2181,node3:2181</property> <property name="Root Node">/nifi</property> <property name="Session Timeout">10 seconds</property> <property name="Access Control">Open</property> </cluster-provider> zookeper properties dataDir=./state/zookeeper autopurge.snapRetainCount=30 server.1=node1:2888:3888 server.2=node2:2888:3888 server.3=node3:2888:3888 nifi properties: nifi.web.http.host=node1 nifi.web.http.port=8080 nifi.cluster.is.node=true nifi.cluster.node.address=node1 nifi.cluster.node.protocol.port=11122 nifi.zookeeper.connect.string=node1:2181,node2:2181,node3:2181 nifi.zookeeper.root.node=/nifi In other 2 nodes the host name are placed respectively as (node2 , node3) C:\nifi-1.11.0-bin\Node1\bin>run-nifi.bat 2020-02-11 20:29:20,237 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2020-02-11 20:29:20,238 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: C:\NIFI-1~1.0-B\Node1 2020-02-11 20:29:20,239 INFO [main] org.apache.nifi.bootstrap.Command Command: C:\Program Files\Java\jdk1.8.0_241\bin\java.exe -classpath C:\NIFI-1~1.0-B\Node1\.\conf;C:\NIFI-1~1.0-B\Node1\.\lib\javax.servlet-api-3.1.0.jar;C:\NIFI-1~1.0-B\Node1\.\lib\jcl-over-slf4j-1.7.26.jar;C:\NIFI-1~1.0-B\Node1\.\lib\jetty-schemas-3.1.jar;C:\NIFI-1~1.0-B\Node1\.\lib\jul-to-slf4j-1.7.26.jar;C:\NIFI-1~1.0-B\Node1\.\lib\log4j-over-slf4j-1.7.26.jar;C:\NIFI-1~1.0-B\Node1\.\lib\logback-classic-1.2.3.jar;C:\NIFI-1~1.0-B\Node1\.\lib\logback-core-1.2.3.jar;C:\NIFI-1~1.0-B\Node1\.\lib\nifi-api-1.11.0.jar;C:\NIFI-1~1.0-B\Node1\.\lib\nifi-framework-api-1.11.0.jar;C:\NIFI-1~1.0-B\Node1\.\lib\nifi-nar-utils-1.11.0.jar;C:\NIFI-1~1.0-B\Node1\.\lib\nifi-properties-1.11.0.jar;C:\NIFI-1~1.0-B\Node1\.\lib\nifi-runtime-1.11.0.jar;C:\NIFI-1~1.0-B\Node1\.\lib\slf4j-api-1.7.26.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dzookeeper.admin.enableServer=false -Dnifi.properties.file.path=C:\NIFI-1~1.0-B\Node1\.\conf\nifi.properties -Dnifi.bootstrap.listen.port=53535 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=C:\NIFI-1~1.0-B\Node1\bin\..\\logs org.apache.nifi.NiFi 2020-02-11 20:29:20,554 WARN [main] org.apache.nifi.bootstrap.Command Failed to set permissions so that only the owner can read pid file C:\NIFI-1~1.0-B\Node1\bin\..\run\nifi.pid; this may allows others to have access to the key needed to communicate with NiFi. Permissions should be changed so that only the owner can read this file 2020-02-11 20:29:20,561 WARN [main] org.apache.nifi.bootstrap.Command Failed to set permissions so that only the owner can read status file C:\NIFI-1~1.0-B\Node1\bin\..\run\nifi.status; this may allows others to have access to the key needed to communicate with NiFi. Permissions should be changed so that only the owner can read this file 2020-02-11 20:29:20,573 INFO [main] org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 4284 ______________________________ C:\nifi-1.11.0-bin\Node1\bin>status-nifi.bat 20:31:57.265 [main] DEBUG org.apache.nifi.bootstrap.NotificationServiceManager - Found 0 service elements 20:31:57.271 [main] INFO org.apache.nifi.bootstrap.NotificationServiceManager - Successfully loaded the following 0 services: [] 20:31:57.272 [main] INFO org.apache.nifi.bootstrap.RunNiFi - Registered no Notification Services for Notification Type NIFI_STARTED 20:31:57.273 [main] INFO org.apache.nifi.bootstrap.RunNiFi - Registered no Notification Services for Notification Type NIFI_STOPPED 20:31:57.274 [main] INFO org.apache.nifi.bootstrap.RunNiFi - Registered no Notification Services for Notification Type NIFI_DIED 20:31:57.277 [main] DEBUG org.apache.nifi.bootstrap.Command - Status File: C:\NIFI-1~1.0-B\Node1\bin\..\run\nifi.status 20:31:57.278 [main] DEBUG org.apache.nifi.bootstrap.Command - Status File: C:\NIFI-1~1.0-B\Node1\bin\..\run\nifi.status 20:31:57.279 [main] DEBUG org.apache.nifi.bootstrap.Command - Properties: {pid=4284, port=53536} 20:31:57.280 [main] DEBUG org.apache.nifi.bootstrap.Command - Pinging 53536 20:31:58.343 [main] DEBUG org.apache.nifi.bootstrap.Command - Process with PID 4284 is not running 20:31:58.345 [main] INFO org.apache.nifi.bootstrap.Command - Apache NiFi is not running This is how my folder structure is: below is the image of Node 2 and node 3 cmd prompt.
... View more
02-09-2020
08:09 AM
@mburgess Am facing the same error. Any help would be appreciated. Am a beginner in Nifi, and trying out some example use-cases available over internet. Trying out "Real-Time SQL On Event Streams" Use case: CSV to Json Conversion CSV is like : purchase_no, customer_id, item_id, item_name, price, quantity 10280, 40070, 1028, Box of pencils, 6.99, 2 10280, 40070, 4402, Stapler, 12.99, 1 12440, 28302, 1029, Box of ink pens, 8.99, 1 28340, 41028, 1028, Box of pencils, 6.99, 18 28340, 41028, 1029, Box of ink pens, 8.99, 18 28340, 41028, 2038, Printer paper, 14.99, 10 28340, 41028, 4018, Clear tape, 2.99, 10 28340, 41028, 3329, Tape dispenser, 14.99, 10 28340, 41028, 5192, Envelopes, 4.99, 45 28340, 41028, 3203, Laptop Computer, 978.88, 2 28340, 41028, 2937, 24\" Monitor, 329.98, 2 49102, 47208, 3204, Powerful Laptop Computer, 1680.99, 1 and my schema looks like: { "name": "helloWorld", "namespace": "org.apache.nifi.blogs", "type": "record", "fields": [ { "name": "purchase_no", "type": "long" }, { "name": "customer_id", "type": "long" }, { "name": "item_id", "type": ["null", "long"] }, { "name": "item_name", "type": ["null", "string"] }, { "name": "price", "type": ["null", "double"] }, { "name": "quantity", "type": ["null", "int"] }, { "name": "total_price", "type": ["null", "double"] } ] } I made sure that all the field type is matched, still facing the below error. ERROR 418b70f8-b8a3-3b60-5042-85ad82a7bf72 QueryRecord[id=418b70f8-b8a3-3b60-5042-85ad82a7bf72] Unable to query StandardFlowFileRecord[uuid=80edfc74-9637-4eaa-abcb-ad2444f1c2da,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1580960212569-1, container=default, section=1], offset=4270, length=599],offset=0,name=80edfc74-9637-4eaa-abcb-ad2444f1c2da,size=599] due to Failed to read next record in stream for StandardFlowFileRecord[uuid=80edfc74-9637-4eaa-abcb-ad2444f1c2da,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1580960212569-1, container=default, section=1], offset=4270, length=599],offset=0,name=80edfc74-9637-4eaa-abcb-ad2444f1c2da,size=599] due to Error while getting next record. Root cause: java.lang.NumberFormatException: For input string: "purchase_no, customer_id, item_id, item_name, price, quantity": org.apache.nifi.processor.exception.ProcessException: Failed to read next record in stream for StandardFlowFileRecord[uuid=80edfc74-9637-4eaa-abcb-ad2444f1c2da,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1580960212569-1, container=default, section=1], offset=4270, length=599],offset=0,name=80edfc74-9637-4eaa-abcb-ad2444f1c2da,size=599] due to Error while getting next record. Root cause: java.lang.NumberFormatException: For input string: "purchase_no, customer_id, item_id, item_name, price, quantity" My Queries are as below: [added below as attributes in QueruRecord Processor to query ] Query 1: Will include the purchase_no and customer_id fields of any purchase that cost more than $1,000.00 and will also include a new field named total_price that is the dollar amount for the entire purchase. Property name : "over.1000" Value : SELECT purchase_no, customer_id, SUM(price * quantity) AS total_priceFROM FLOWFILEGROUP BY purchase_no, customer_idHAVING SUM(price * quantity) > 1000 Query 2: Will contain the purchase_no, customer_id, and total price of the most expensive single purchase (as defined by price times quantity) in the data: Property : "largest.order" Value : SELECT purchase_no, customer_id, SUM(price * quantity) AS total_priceFROM FLOWFILEGROUP BY purchase_no, customer_idORDER BY total_price DESCLIMIT 1
... View more