question Re: puthdfs is writing slow in Archives of Support Questions (Read Only)

puthdfs is writing slow

mark_hadoop — Tue, 15 Aug 2017 20:25:56 GMT

I am getting data using gethdfs and did some processing and writing back to hdfs.

untill puthdfs data is processing fast, but puthdfs is writing data slow to hdfs.

could you please let me know how to improve the speed?

Re: puthdfs is writing slow

MattWho — Tue, 15 Aug 2017 20:57:35 GMT

@Hadoop User

Please share your PutHDFS processor configuration with us.

How large are the individual files that are being written to HDFS?

Thanks,

Matt

Re: puthdfs is writing slow

mark_hadoop — Sun, 18 Aug 2019 02:31:45 GMT

@Matt Clarke

a file is only 1-2 kb file.

configuration is

concurrent tasks; 1, rest are not changed.

Thank you

Re: puthdfs is writing slow

MattWho — Wed, 16 Aug 2017 01:14:49 GMT

@Hadoop User

It is unlikely you will see the same performance out of Hadoop between reads and writes. The Hadoop Architecture is designed in such a way to favor multiple many readers and few data writers.

Increasing the number of concurrent tasks may help but performance since you will then have multiple files being written concurrently.

1 - 2 KB files are very small and do not make optimal use of your Hadoop architecture. Commonly, NiFi is used to merge bundles of files together to a more optimal size for storage in Hadoop. I believe 64 KB is the default optimal size.

You can remove some of the overhead of each connection by mergeing files together in to larger files using the MergeContent processor before writing to Hadoop.

Thanks,

Matt

Re: puthdfs is writing slow

mark_hadoop — Fri, 13 Oct 2017 17:34:11 GMT

@Matt Clarke

Thank you.

I have merged files depending on the frequency of writes to 64 KB.

Sorry for turning late.

Re: puthdfs is writing slow

MattWho — Fri, 13 Oct 2017 23:01:16 GMT

@Hadoop User

If merging FlowFiles and adding more concurrent tasks to your putHDFS processor help with your performance issue here, please take a moment to click "accept" on the above answer to close out this thread.

Thank you,

Matt