question Re: I know this is a basic question, but just started with Nifi, want to save syslog data into HDFS, can anyone help in Archives of Support Questions (Read Only)

I know this is a basic question, but just started with Nifi, want to save syslog data into HDFS, can anyone help

mark_hadoop — Fri, 14 Jul 2017 18:14:26 GMT

it'll be helpfull if what processor to be used in beween listensyslog and puthdfs is suggested.

Re: I know this is a basic question, but just started with Nifi, want to save syslog data into HDFS, can anyone help

msumbul1 — Fri, 14 Jul 2017 20:28:11 GMT

Hi, You just need put a processor "puthdfs", then righ click to open the configuration panel. they you need to provide the configuration file of hdfs (hdfis-site.xml, core-site.xml) and the folder where you want to put the data. This is to put data to hdfs. To read data (for example locally) use the processor tailfile.

Re: I know this is a basic question, but just started with Nifi, want to save syslog data into HDFS, can anyone help

MattWho — Fri, 14 Jul 2017 20:39:49 GMT

@Hadoop User

The "it'll be helpful if what processor to be used in between listenSyslog and putHDFS is suggested" question is a hard one for anyone to answer without understanding the end result you are looking for.

There are The following processors: - parseSyslog (extract bits from syslog content in to FlowFile attributes) You can then use those attributes if you like to make routing decisions (routeOnAttribute), define unique target HDFS directories based on attribute value in PutHDFS - SplitText or SlitContent (Can be used to FlowFiles that contain more then one syslog message each). You get improved performance if listenSyslog ingests in batches.

- UpdateAttribute (Used to add you own custom attributes or manipulate existing attributes on FlowFiles)

Thanks,

Matt