Incrementally Loading flat files into hadoop (comma delimited text file)

r_mageshkumar — Fri, 16 Sep 2022 10:45:59 GMT

Hi All,

Is it possible to load incremental text files (comma delimited text file) into hadoop as the data is increasing every minute. If we have folder containing date wise flat files in the local system, is it possible to load the files automatically into hadoop periodically.

If a particular file is loaded into hdfs, then it should load from the next record onwards.

What are the suitable ecosystems we can use to load the flat files.

Please do the needful.

Re: Incrementally Loading flat files into hadoop (comma delimited text file)

gkeys — Fri, 28 Oct 2016 19:35:56 GMT

@Magesh Kumar

I believe this question is identical to another one you asked: https://community.hortonworks.com/questions/63947/incremental-flat-file-data-loading-into-hadoop.html#answer-63978 If there are differences, please elaborate.

question Incrementally Loading flat files into hadoop (comma delimited text file) in Archives of Support Questions (Read Only)

Incrementally Loading flat files into hadoop (comma delimited text file)

Re: Incrementally Loading flat files into hadoop (comma delimited text file)