Member since
10-13-2018
7
Posts
0
Kudos Received
0
Solutions
05-28-2019
11:05 PM
I can find and read the files from inside the nifi docker container. Not sure why ListFile cannot: nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r' cdr_AP9-SYD-UCCE-UCM_08_201905281913_183183 cdr_AP4-SYD-CCP-UCM_05_201905281913_53152 cdr_AP2-HKG-CCP-UCM_13_201905281912_89066 cdr_AP9-SYD-UCCE-UCM_07_201905281911_196358 cdr_AP8-HKG-UCCE-UCM_02_201905281910_243229 nifi@4f614c6c67c4:/data/sftp/cdr$ tail -n 5 cdr_AP9-SYD-UCCE-UCM_08_201905281913_183183 1,8,973062,138959080,1559070286,8,0,0,"","",0,0,4,0,0,0,0,0,0,0,0,0,0,"0","0",138959081,3,0,1054035722,"b003864986001","b003864986001","",0,0,4,0,0,0,0,0,0,
... View more
05-28-2019
11:05 PM
I can see and read the files from inside the nifi container. Not sure why the nifi ListFile process isn't picking them up. nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r' cdr_AP9-SYD-UCCE-UCM_08_201905281913_183183 cdr_AP4-SYD-CCP-UCM_05_201905281913_53152 cdr_AP2-HKG-CCP-UCM_13_201905281912_89066 cdr_AP9-SYD-UCCE-UCM_07_201905281911_196358 cdr_AP8-HKG-UCCE-UCM_02_201905281910_243229 nifi@4f614c6c67c4:/data/sftp/cdr$ tail -n 5 cdr_AP9-SYD-UCCE-UCM_08_201905281913_183183 1,8,973062,138959080,1559070286,8,0,0,"","",0,0,4,0,0,0,0,0,0,0,0,0,0,"0","0",138959081,3,0,1054035722,"b003864986001","b003864986001","",0,0,4,0,0,0,0,0,0,0,
... View more
05-28-2019
03:48 PM
Hi there, I'm running NiFi in docker. I'm trying to enable flow debugs and also to determine if NiFi can actually see in my data directory path. I'm trying to use a simple recursive list file processor to find the files based on a regular expression "^c.r". I can docker exec into the container and can see that the nifi container can "see" into my filepath OK: nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r' cdr_AP2-HKG-CCP-UCM_12_201905281539_35251 cdr_AP2-HKG-CCP-UCM_06_201905281539_297892 cmr_AP2-HKG-CCP-UCM_06_201905281539_297892 cdr_AP9-SYD-UCCE-UCM_05_201905281539_200435 cdr_AP2-HKG-CCP-UCM_11_201905281539_98483 However, I can't get the ListFile processor to work and find those files. Any ideas on debugging strategies?
... View more
Labels:
- Labels:
-
Apache NiFi
-
Docker
10-24-2018
02:43 PM
I'm starting to trial NiFi as a data ingestion engine. I would like to input the following datatypes: 1) collectd (UDP): I don't think that NiFi has a collectd parser, so I will need to direct these raw UDP streams to a locally running Telegraf and Logstash instance for parsing. 2) Syslog (UDP): I would like to experiment with routing raw syslog UDP packets (unprocessed) to destinations as well as filtering/parsing the syslog data using the NiFi syslog modules. 3) Netflow (UDP): I would like to take a heavy raw Netflow stream and test performance to forward only a subset of Netflow data based on a list of protocol types that I'm interested in (mapped against one of the Netflow data key values). What I haven't been able to understand in the documentation is how I can redirect raw UDP packet flows, (listening and then forwarding to two destinations), without having to process the particular data in the UDP packet.
... View more
10-19-2018
01:53 PM
Thanks @Matt Clarke. I guess I have a bit of reading to do to find the right processor that just routes raw UDP packets.
... View more
10-14-2018
12:43 AM
Hi there, I am planning on using NiFi for various types of data ingestion, but was wondering if I could use NiFi for raw UDP forwarding. For example if my NiFi server is accepting syslog/UDP can I process the syslog and send to say elasticsearch and also forward that raw unprocessed syslog/UDP flow? Also if I wants to forward/route raw collectd/UDP binary protocol flows could I do that with NiFi? Thanks!
... View more
Labels:
- Labels:
-
Apache NiFi