Member since
07-30-2019
105
Posts
129
Kudos Received
43
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1343 | 02-27-2018 01:55 PM | |
1747 | 02-27-2018 05:01 AM | |
4786 | 02-27-2018 04:43 AM | |
1339 | 02-27-2018 04:18 AM | |
4254 | 02-27-2018 03:52 AM |
07-16-2016
02:27 PM
4 Kudos
GetFile when told to keep source files where it finds them will capture them even if it doesn't have write permissions to the directory they are contained in. However, when told to remove source files once pulled it requires write permissions to the directory it is pulling from and when listing it will skip those which it doesn't have permissions for. Given that we know there are files there and it isn't pulling them in this case and specifically yielding, which only happens when the listing attempt provides no valid results, then I strongly believe the parent directory permissions are not sufficient. Please verify.
... View more
07-16-2016
02:11 PM
Not sure just yet. Will take a look. The only time GetFile would yield, as is the case in the log output you show for keepFile=true, is when it finds nothing in the listing.
... View more
07-16-2016
02:01 PM
2 Kudos
I don't have any real numbers to share though I found this article on the topic interesting https://www.maxcdn.com/blog/ssl-performance-myth/ and found that it aligns to what I've observed as well. There are relatively few cases where unsecured site-to-site is appropriate in comparison to secured site-to-site.
... View more
07-15-2016
06:36 PM
Hello Please take a look in the logs/nifi-app.log. There should be errors. Sounds like it might not be able to delete the files (perms on the directory itself perhaps). If nothing interesting in the logs try updating your conf/logback.xml by adding this line in with other similar looking lines <logger name="org.apache.nifi.processors.standard.GetFile" level="DEBUG"/> Thanks
Joe
... View more
07-04-2016
09:15 PM
2 Kudos
There are generally two things to consider. First, the configuration of a given dataflow. Second, the code required for a given dataflow. If your deployed artifacts in dev and prod are aligned then your focus is on configuration. For this NiFi supports templates which you can generate in dev and import into production. Templates are a great start for this but they do have the current downside in that they won't copy sensitive properties and they can be too coupled to environmental items like database URLs or web URLs. There are efforts underway in the Apache NiFi community to make them more portable via environment variable mappings which tie to a given environment and then the templates will tie to the mappings. In the case where you also need to get new deployment artifacts into production we benefit from Apache NiFi's support for easily deployed NiFi Archives (NARs) which nicely contain the code and dependencies so it is generally as easy as moving in a new NAR bundle into the lib. Typically people will have an 'extensions' folder and place their items in there. On restart NiFi will read its configuration classpath location(s) and make that new code available. In a cluster people typically add these items to all nodes then do rolling restart of the nodes in the cluster to avoid any downtime for the flow.
... View more
06-24-2016
01:37 PM
Thanks for putting this out Simon! We definitely need to make this something people don't need to think about. Ideas welcome.
... View more
06-23-2016
09:40 PM
Have you had a chance to enable the site to site port on the receiving system (well both systems really)? http://docs.hortonworks.com/HDPDocuments/HDF1/HDF-1.2/bk_AdminGuide/content/system_properties.html Take a look at the "Site to Site Properties" section.
... View more
06-23-2016
03:29 AM
@Thierry Vernhet Is the intent that it could use patterns to find a file to tail or is it that it will find the file(s) to tail? If it is meant to be about finding a single file then this seems doable. If the idea is to match on multiple files at once can you share how you expect it to behave?
... View more
06-21-2016
01:04 AM
Can you describe what version of Kafka you're running. Have you been able to publish or consume messages from this topic?
... View more
06-01-2016
03:19 PM
1 Kudo
What looks likely to be happening is that your nar is bundling things you did not intend to include or should not include. Specifically you will not want to include the actual implementations of services you're depending on. You can have a nar depend on another nar and NiFi will automatically not actually pull that dependent nar into your nar but rather will wire it up at startup as part of its classpath handling. However, if you include the implementation of other services they could conflict as I suspect is the case here. Could you share/show details of how your project it structure or provide a pom and we could probably give you more specific pointers. There are several examples of nars in the apache nifi source too that show how to depend on services without pulling in their implementations. This is a good example https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-nar/pom.xml It pulls in the nar for the service definitions so that this nar can depend on those things but not actually include them in it's own package or create conflicts.
... View more