Member since
10-16-2017
11
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1274 | 10-17-2017 04:49 AM |
07-31-2018
08:04 AM
1 Kudo
Thank you @Jonathan Sneep Yes, after adding following 2 nars to my minify lib it starts working: nifi-ssl-context-service-nar-1.7.1.nar nifi-standard-services-api-nar-1.7.1.nar You saved my time, thank you 🙂
... View more
07-31-2018
06:00 AM
I use RouteonAttribute and route my flowfiles coming from Zmq to Kafka_Producer processor ( nifi-kafka-1-0-nar-1.7.1.nar ) based on stream_name attribute. It's working great on nifi. (screen-shot-2018-07-31-at-113736-am.png(434.9 kB) and screen-shot-2018-07-31-at-113744-am.png(300.5 kB) ) But when I export template and convert to config.yml and used in minifi, I am getting below error: 2018-07-31 06:37:45,050 WARN [main] org.apache.nifi.minifi.FlowEnricher Could not find any eligible bundles for org.apache.nifi.processors.kafka.pubsub.PublishKafka_1_0.Automatic start of the flow cannot be guaranteed. ......... 2018-07-31 05:45:35,971 ERROR [main] o.apache.nifi.controller.FlowController Could not create Processor of type org.apache.nifi.processors.kafka.pubsub.PublishKafka_1_0 for ID 1490dc93-67ec-392f-0000-000000000000; creating "Ghost" implementation org.apache.nifi.controller.exception.ProcessorInstantiationException: Unable to find bundle for coordinate default:unknown:unversioned at org.apache.nifi.controller.FlowController.instantiateProcessor(FlowController.java:1271) at org.apache.nifi.controller.FlowController.createProcessor(FlowController.java:1188) at org.apache.nifi.controller.FlowController.createProcessor(FlowController.java:1157) at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1214) at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:359) at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1697) at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84) at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:723) at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:534) at org.apache.nifi.minifi.MiNiFiServer.start(MiNiFiServer.java:122) at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:148) at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:247) 2018-07-31 05:45:35,994 ERROR [main] o.apache.nifi.controller.FlowController Could not create Processor of type org.apache.nifi.processors.kafka.pubsub.PublishKafka_1_0 for ID 63566e4b-ba19-3f4c-0000-000000000000; creating "Ghost" implementation org.apache.nifi.controller.exception.ProcessorInstantiationException: Unable to find bundle for coordinate default:unknown:unversioned at org.apache.nifi.controller.FlowController.instantiateProcessor(FlowController.java:1271) at org.apache.nifi.controller.FlowController.createProcessor(FlowController.java:1188) at org.apache.nifi.controller.FlowController.createProcessor(FlowController.java:1157) at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1214) at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:359) at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1697) at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84) at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:723) at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:534) at org.apache.nifi.minifi.MiNiFiServer.start(MiNiFiServer.java:122) at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:148) at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:247) 2018-07-31 05:45:36,408 WARN [main] o.a.n.c.StandardFlowSynchronizer Schema validation error parsing Flow Configuration at line 16, col 27: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected. NOTE1: I already copied nifi-kafka-1-0-nar-1.7.1.nar into minifi's lib directory and restart minifi. Is this problem due to minify not supporting nifi-kafka-1-0-nar-1.7.1.nar OR due to using same processor 2 times in flow? NOTE2: I found article "How to configure MiNiFi PublishKafka Processor to inject data to Kafka";
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache MiNiFi
-
Apache NiFi
07-04-2018
04:53 AM
I am more comfortable with Ubuntu and want to install Hortonworks HDP over it. As per my finding Hortonworks HDP does not support Ubunbtu officially. Do anyone installed Hortonworks Data Platform 2.7 over Ubuntu? Is it recommended?
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
12-01-2017
05:14 AM
Thank you @J Koppole. My co-worker do not want to increase ulimit more than 90000, (and unlimited) so I need an alternative. According to my co-worker making ulimit unlimited will make security issue and if any other processes can open unlimited files, it will consume whole resources quickly. Is it still safe to make ulimit unlimited. My actual question was: As I know, nifi will generate flowfile for each event message. Does nifi needs to open file for each generated flowfile? Could you explain me when nifi needs to open files and when nifi closes them; so that I can optimize my implementation?
... View more
12-01-2017
05:09 AM
@Sandeep Kumar Thank you Sandeep for your solution. Actually, it was not my actual question. Previously, my nifi getting problem when getting 1lakh flowfile in queue and I increase ulimit from 1024 to 90000; But still I am getting max open file errors when I am having 10 lakh flowfiles in queue. My co-worker do not want to increase ulimit more than this, so I need alternative. I appreciate your solution but I need to know when nifi needs to open files and how to reduce it. My actual question: As I know, nifi will generate flowfile for each event message. Does nifi needs to open file for each generated flowfile? Could somebody explain me when nifi needs to open files and when nifi closes them so that I can optimize my implementation?
... View more
11-28-2017
04:10 AM
I have developed custom processor which will consume zmq data produced by source. I am using nifi to forward those received zmq data to remote machine using RemoteProcessGroup (site to site setup). When large logs is generated on source, I am getting "Too many open files" error in nifi log. As I know, nifi will generate flowfile for each event message. Does nifi needs to open file for each generated flowfile? Could somebody explain me when nifi needs to open files and when nifi closes them so that I can optimize my implementation?
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache NiFi
10-17-2017
04:49 AM
The problem is due to improper configuration on nifi.properties Please make sure: you have following properties in nifi-1.4.0/conf/nifi.properties: # Site to Site properties nifi.remote.input.host=192.168.2.8 # Previously it was localhost when I was getting error nifi.remote.input.socket.port=10000 # Your destination machine should have allowed this port
... View more
10-17-2017
03:33 AM
I got problem while setting up nifi site to site. Source throws error "Unable to find port with id xyz" ; But in destination port with id xyz exist. Please find below link for screenshots. Any help please. Screenshots: https://drive.google.com/drive/folders/0B63F0W31MWMqRTY1UDFWMGFncTA?usp=sharing
... View more
Labels:
- Labels:
-
Apache NiFi