Member since
07-30-2019
105
Posts
129
Kudos Received
43
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
762 | 02-27-2018 01:55 PM | |
1239 | 02-27-2018 05:01 AM | |
3095 | 02-27-2018 04:43 AM | |
665 | 02-27-2018 04:18 AM | |
1905 | 02-27-2018 03:52 AM |
07-15-2016
06:36 PM
Hello Please take a look in the logs/nifi-app.log. There should be errors. Sounds like it might not be able to delete the files (perms on the directory itself perhaps). If nothing interesting in the logs try updating your conf/logback.xml by adding this line in with other similar looking lines <logger name="org.apache.nifi.processors.standard.GetFile" level="DEBUG"/> Thanks
Joe
... View more
07-04-2016
09:15 PM
2 Kudos
There are generally two things to consider. First, the configuration of a given dataflow. Second, the code required for a given dataflow. If your deployed artifacts in dev and prod are aligned then your focus is on configuration. For this NiFi supports templates which you can generate in dev and import into production. Templates are a great start for this but they do have the current downside in that they won't copy sensitive properties and they can be too coupled to environmental items like database URLs or web URLs. There are efforts underway in the Apache NiFi community to make them more portable via environment variable mappings which tie to a given environment and then the templates will tie to the mappings. In the case where you also need to get new deployment artifacts into production we benefit from Apache NiFi's support for easily deployed NiFi Archives (NARs) which nicely contain the code and dependencies so it is generally as easy as moving in a new NAR bundle into the lib. Typically people will have an 'extensions' folder and place their items in there. On restart NiFi will read its configuration classpath location(s) and make that new code available. In a cluster people typically add these items to all nodes then do rolling restart of the nodes in the cluster to avoid any downtime for the flow.
... View more
06-24-2016
01:37 PM
Thanks for putting this out Simon! We definitely need to make this something people don't need to think about. Ideas welcome.
... View more
06-23-2016
09:40 PM
Have you had a chance to enable the site to site port on the receiving system (well both systems really)? http://docs.hortonworks.com/HDPDocuments/HDF1/HDF-1.2/bk_AdminGuide/content/system_properties.html Take a look at the "Site to Site Properties" section.
... View more
06-23-2016
03:29 AM
@Thierry Vernhet Is the intent that it could use patterns to find a file to tail or is it that it will find the file(s) to tail? If it is meant to be about finding a single file then this seems doable. If the idea is to match on multiple files at once can you share how you expect it to behave?
... View more
06-21-2016
01:04 AM
Can you describe what version of Kafka you're running. Have you been able to publish or consume messages from this topic?
... View more
06-01-2016
03:19 PM
1 Kudo
What looks likely to be happening is that your nar is bundling things you did not intend to include or should not include. Specifically you will not want to include the actual implementations of services you're depending on. You can have a nar depend on another nar and NiFi will automatically not actually pull that dependent nar into your nar but rather will wire it up at startup as part of its classpath handling. However, if you include the implementation of other services they could conflict as I suspect is the case here. Could you share/show details of how your project it structure or provide a pom and we could probably give you more specific pointers. There are several examples of nars in the apache nifi source too that show how to depend on services without pulling in their implementations. This is a good example https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-nar/pom.xml It pulls in the nar for the service definitions so that this nar can depend on those things but not actually include them in it's own package or create conflicts.
... View more
04-26-2016
08:40 PM
5 Kudos
Hello Unfortunately this means the process NiFi was told to execute has not returned. In such a case there is an outstanding thread and we intentionally prevent additional instances from being started until this one is dealt with. From the stacktrace you provided we see "Timer-Driven Process Thread-9" Id=69 RUNNABLE (in native code)
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:272)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- waiting on java.lang.UNIXProcess$ProcessPipeInputStream@e6f0a2f
This tells us we're sitting and waiting for that command to do something (finish, respond with data). It appears to be in a hung state. You'll need to restart NiFi and try to assess why that command isn't doing anything. What command are you trying to run? It might be that for your case the 'ExecuteProcess' processor is a better fit. Thanks Joe
... View more
04-21-2016
09:25 PM
1 Kudo
Hello, Keep in mind each FlowFile is comprised of attributes and content. RouteOnAttribute is built specifically to look at FlowFile attributes. RouteOnContent looks at FlowFile content. In the case of JSON you can, for example, use EvaluateJSONPath to extract fields from the JSON to become FlowFile attributes. You can use SplitJSON to split the JSON bundles into invidual flowfiles then extract values and then route on attributes. The various processors offer you a variety of ways to route, transform, and deliver the data. Thanks
Joe
... View more
04-12-2016
02:02 PM
2 Kudos
Using UnpackContent processor you can take the items out of tar or zip archives as individual flow files. Metadata about those objects will be retained on each flow file. You can then operate on those individual unpacked items to do what you need then you could if needed recombine them back into a zip or tar using the merge strategy of 'defragment'.
... View more
04-08-2016
03:33 PM
This is due to this issue https://issues.apache.org/jira/browse/NIFI-990 which fixed the fact that the failure relationship was mistakenly not provided before. When importing templates made against the previous version they will not have that relationship checked since it wasn't there. So it will show as invalid until you check it as auto-terminate or use it. You may wish to recreate the template.
... View more
03-16-2016
02:08 PM
5 Kudos
The HDF release does support interacting with Kerberized Kafka instances as found within the HDP stack. This is because HDP added support for Kerberized Kafka in advance of the community supporting it (Kafka 0.8.x). In Apache Kafka world now (0.9x) there is Kerberos support. So, - Apache NiFi supports non Kerberized Kafka clusters today. - HDF releases of NiFi have patched support for Kerberized Kafka clusters in HDP - Upcoming Apache NiFi releases will add support for the 0.9x Apache Kafka kerberos model Thanks Joe
... View more
02-26-2016
03:11 AM
2 Kudos
I haven't tested this myself but NAS/SAN arrangements have worked quite well in the past. Needs testing to understand latencies/tradeoffs but frankly I suspect it will work just fine.
... View more
02-12-2016
02:03 AM
1 Kudo
In the Get/PutKafka processors you should be able to add a dynamic property called 'fetch.message.max.bytes' and set the value you need. The processor should allow you to add dynamic properties which map to Kafka consumer properties and it will pass them to the consumer/producer config as needed.
... View more
01-28-2016
02:04 AM
2 Kudos
NiFi does not at present offer any generic SOAP interface support. You would need to built a custom processor to do that. NiFi you can think of as a great host for the Java process suggested in this thread. Once the data is pulled via the SOAP API you can then use NiFi to do any number of things such as delivery to Kafka all within a managed process. Then you get the benefits NiFi offers and address your core use case.
... View more
01-23-2016
01:33 AM
Vance currently in NiFi any user with the DFM permission can create as many flows as necessary. It is not uncommon for a single instance of NiFi to be handling hundreds or more processors representing what can be dozens or hundreds of distinct dataflows. It is also quite common for people to be surprised by that however a lot of effort has gone into the design of the repositories, threading model, and user interface to allow it to support a wide variety of functions and flows. It is certainly a solid compliment to the powerful analysis and processing platforms that systems like Storm and Spark provide or the storage/access systems that Kafka and HDFS provide.
... View more
01-22-2016
12:48 AM
4 Kudos
Vance, We completely agree with you. NiFi already supports some powerful security and multi-role authorization capabilities. But as you mention we should support multiple different groups with different levels of access to various parts of the flow. That is an important roadmap item and work is underway. You can see a bit about the nifi community thinking on this wiki page https://cwiki.apache.org/confluence/display/NIFI/Multi-Tentant+Dataflow and there are related threads such as https://cwiki.apache.org/confluence/display/NIFI/Redesign+User+Interface and https://cwiki.apache.org/confluence/display/NIFI/Support+Authorizer+API If you need help setting up secure NiFi you can read more here https://community.hortonworks.com/articles/886/securing-nifi-step-by-step.html and in the administration guide https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#security-configuration Thanks Joe
... View more
01-21-2016
07:06 PM
2 Kudos
You can pull data from JMS using either queues or topics by using these processors: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GetJMSTopic/index.html https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GetJMSQueue/index.html Those processors presently support ActiveMQ out of the box. It is pretty easy to add other vendors as well but we're going to make it even easier in an upcoming release. Once you pull data from JMS then you connect that processor's output to PutHDFS as explained here: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hadoop.PutHDFS/index.html You might find these templates useful to help get you started: https://github.com/hortonworks-gallery/nifi-templates/tree/master/templates Thanks Joe
... View more
01-21-2016
05:15 AM
Hello You've run into a bug. https://issues.apache.org/jira/browse/NIFI-1396, https://issues.apache.org/jira/browse/NIFI-21 Has been around for quite a while but someone else ran into recently. Should be sorted soon but if you're interested in contributing a fix please let us know. Thanks Joe
... View more
01-21-2016
05:08 AM
1 Kudo
Thanks for reporting it and for providing the stack traces. Very helpful. I've filed an Apache NiFi JIRA for it https://issues.apache.org/jira/browse/NIFI-1417
... View more
01-11-2016
01:55 PM
Actually Davide I just checked the current GeoEnrichIP processor and it does indeed include lat/long/city/country/postalcode. It is driven by a given IP address and you control the input dataset. Maxmind, which is the dataset that processor is built around also offers a pay version for higher accuracy. Are you looking for geo enrichment based on an IP address or another type of address? Thanks Joe
... View more
01-11-2016
01:49 PM
1 Kudo
Hello Davide, We do not have an out of the box processor to do this at this time but it would be a fairly straightforward custom processor to build. Of course querying an external service for this information offers different tradeoffs than having a local cache of data so keep that in mind. I will take a look at our existing geo enrichment processor to see what is involved in getting lat/long data as well. Thanks Joe
... View more
01-07-2016
03:29 AM
1 Kudo
Great. Thanks for providing that follow-up!
... View more
01-07-2016
03:26 AM
1 Kudo
Hello. In looking at the API/products facebook offers that URL api.facebook.com/restserver.php was not something I could find. Can you point to the Facebook API documentation of the particular endpoint you wish to access? Also, the programmableweb entry you provided shows that authentication in the form of oauth, apikey, username/password is required. But, it sounds like those weren't set. Might be better to just implement a process to listen to/interact with the Facebook API. Thanks Joe
... View more
01-06-2016
02:35 AM
Hello @Sunile Manjee. As Andrew mentions the http://nifi.apache.org/quickstart.html does outline how to alter the settings to ensure the NiFi process is able to have sufficient open files. One common gotcha is that the setting is tied to a different user than your NiFi process is executed as and in making sure that the session NiFi starts with does indeed have that new open files setting reflected. Are you running 'bin/nifi.sh start' in the same terminal that you run 'ulimit -a' to see if the setting has taken effect? Another good technique you can use is to run 'lsof -p 12345' assuming the pid of NiFi is 12345 and it will show you all the open file handles that NiFi process has. Thanks Joe
... View more
12-23-2015
12:34 PM
2 Kudos
Absolutely. Don't have a good timeline to offer at the moment but it is on the roadmap. Happy to talk it over further with you. Thanks!
... View more
12-12-2015
06:08 PM
1 Kudo
Hello What the NiFi server exposes/provides is a RESTFul API. If you attach your web browser to it you'll get a visually friendly UI. You can lock down access via 2-Way SSL in HDF 1.0 and in the upcoming HDF 1.1 you can also do username/password with active directory or directory server via LDAP. Thanks Joe
... View more
12-09-2015
06:33 PM
3 Kudos
Have run on both BeagleBone Black and the latest Rapsberry PI (not zero). Works quite well but Pi is considerably faster. It is slow to start as we load a lot of classes in the beginning but on a Pi type system you can certainly remove extraneous Nars. I think you can expect around 30MB/s of throughput and about 10K events/s with full features activated.
... View more
12-09-2015
04:11 AM
This gem of a link is a great example of why this knowledge base is super valuable. Thanks
... View more
- « Previous
- Next »