stgreszc
Explorer

How to persist dataflows in nifi 1.9.2 (docker)?

by Explorer in Support Questions
‎06-04-2019 02:56 PM
‎06-04-2019 02:56 PM
Hi there, What are the essential volumes that need to be mapped for nifi 1.9.2 to be able to be able to load and persist dataflows, flowfile metadata, etc. I'm using Docker Compose with the vanilla docker container, so would prefer to use volumes or environment variables if possible. Thanks! ... View more

Re: Debugging NiFi Flows in Docker - problem with ...

by Explorer in Support Questions
‎05-28-2019 11:05 PM
‎05-28-2019 11:05 PM
I can find and read the files from inside the nifi docker container. Not sure why ListFile cannot: nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r' cdr_AP9-SYD-UCCE-UCM_08_201905281913_183183 cdr_AP4-SYD-CCP-UCM_05_201905281913_53152 cdr_AP2-HKG-CCP-UCM_13_201905281912_89066 cdr_AP9-SYD-UCCE-UCM_07_201905281911_196358 cdr_AP8-HKG-UCCE-UCM_02_201905281910_243229 nifi@4f614c6c67c4:/data/sftp/cdr$ tail -n 5 cdr_AP9-SYD-UCCE-UCM_08_201905281913_183183 1,8,973062,138959080,1559070286,8,0,0,"","",0,0,4,0,0,0,0,0,0,0,0,0,0,"0","0",138959081,3,0,1054035722,"b003864986001","b003864986001","",0,0,4,0,0,0,0,0,0, ... View more

Re: Debugging NiFi Flows in Docker - problem with ...

by Explorer in Support Questions
‎05-28-2019 11:05 PM
‎05-28-2019 11:05 PM
I can see and read the files from inside the nifi container. Not sure why the nifi ListFile process isn't picking them up. nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r' cdr_AP9-SYD-UCCE-UCM_08_201905281913_183183 cdr_AP4-SYD-CCP-UCM_05_201905281913_53152 cdr_AP2-HKG-CCP-UCM_13_201905281912_89066 cdr_AP9-SYD-UCCE-UCM_07_201905281911_196358 cdr_AP8-HKG-UCCE-UCM_02_201905281910_243229 nifi@4f614c6c67c4:/data/sftp/cdr$ tail -n 5 cdr_AP9-SYD-UCCE-UCM_08_201905281913_183183 1,8,973062,138959080,1559070286,8,0,0,"","",0,0,4,0,0,0,0,0,0,0,0,0,0,"0","0",138959081,3,0,1054035722,"b003864986001","b003864986001","",0,0,4,0,0,0,0,0,0,0, ... View more

Debugging NiFi Flows in Docker - problem with List...

by Explorer in Support Questions
‎05-28-2019 03:48 PM
‎05-28-2019 03:48 PM
Hi there, I'm running NiFi in docker. I'm trying to enable flow debugs and also to determine if NiFi can actually see in my data directory path. I'm trying to use a simple recursive list file processor to find the files based on a regular expression "^c.r". I can docker exec into the container and can see that the nifi container can "see" into my filepath OK: nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r' cdr_AP2-HKG-CCP-UCM_12_201905281539_35251 cdr_AP2-HKG-CCP-UCM_06_201905281539_297892 cmr_AP2-HKG-CCP-UCM_06_201905281539_297892 cdr_AP9-SYD-UCCE-UCM_05_201905281539_200435 cdr_AP2-HKG-CCP-UCM_11_201905281539_98483 However, I can't get the ListFile processor to work and find those files. Any ideas on debugging strategies? ... View more

Re: Can NiFi route raw packets - like UDP

by Explorer in Support Questions
‎10-24-2018 02:43 PM
‎10-24-2018 02:43 PM
I'm starting to trial NiFi as a data ingestion engine. I would like to input the following datatypes: 1) collectd (UDP): I don't think that NiFi has a collectd parser, so I will need to direct these raw UDP streams to a locally running Telegraf and Logstash instance for parsing. 2) Syslog (UDP): I would like to experiment with routing raw syslog UDP packets (unprocessed) to destinations as well as filtering/parsing the syslog data using the NiFi syslog modules. 3) Netflow (UDP): I would like to take a heavy raw Netflow stream and test performance to forward only a subset of Netflow data based on a list of protocol types that I'm interested in (mapped against one of the Netflow data key values). What I haven't been able to understand in the documentation is how I can redirect raw UDP packet flows, (listening and then forwarding to two destinations), without having to process the particular data in the UDP packet. ... View more

Re: Can NiFi route raw packets - like UDP

by Explorer in Support Questions
‎10-19-2018 01:53 PM
‎10-19-2018 01:53 PM
Thanks @Matt Clarke. I guess I have a bit of reading to do to find the right processor that just routes raw UDP packets. ... View more

Can NiFi route raw packets - like UDP

by Explorer in Support Questions
‎10-14-2018 12:43 AM
‎10-14-2018 12:43 AM
Hi there, I am planning on using NiFi for various types of data ingestion, but was wondering if I could use NiFi for raw UDP forwarding. For example if my NiFi server is accepting syslog/UDP can I process the syslog and send to say elasticsearch and also forward that raw unprocessed syslog/UDP flow? Also if I wants to forward/route raw collectd/UDP binary protocol flows could I do that with NiFi? Thanks! ... View more
Contact Me
Online
Offline
Last Visited
‎06-04-2019 08:11 PM
Public Statistics
Date Registered ‎10-13-2018 10:40 PM
Last Visited ‎06-04-2019 08:11 PM
Total Messages Posted 7