question Can HdfsBolt append data to existing file? in Support Questions

Can HdfsBolt append data to existing file?

jason_huewk — Sat, 23 Jul 2016 23:04:49 GMT

I want to write data to new file if file not exist and append data to existing file using storm hdfs connector HdfsBolt. May I know how to do this ? Appreciate for any suggestions.

Re: Can HdfsBolt append data to existing file?

sunile_manjee — Sat, 23 Jul 2016 23:14:41 GMT

File on HDFS are immutable. Hdfs bolt allows for example "After every 1,000 tuples it will sync filesystem, making that data visible to other HDFS clients. It will rotate files when they reach 5 megabytes in size."

So you can buffer up events until specified interval. Take a look at my github storm code. You will see how that is performed

https://github.com/sunileman/storm-twitter-sentiment

Re: Can HdfsBolt append data to existing file?

jason_huewk — Sun, 24 Jul 2016 06:37:22 GMT

Appreciate for the advice between the file name will be named something like ddmmyyyy-hh. I want to group the log by hourly and the event per second can be changed, so the number of tuples and file side cannot be determined. In this case how to do it?

Re: Can HdfsBolt append data to existing file?

sunile_manjee — Sun, 24 Jul 2016 06:49:36 GMT

So your looking for windowing on storm.ie do somethikg based on a specificed time period. Until recently you had to build your own windowing logic in storm by keep track of time and do some disk cache to hold events until window tome has completed. Now the functionality comes out of the box. Take a look at an excellent article written on how the new functionality works in storm here. https://community.hortonworks.com/articles/14171/windowing-and-state-checkpointing-in-apache-storm.html

Re: Can HdfsBolt append data to existing file?

jason_huewk — Sun, 24 Jul 2016 09:05:35 GMT

Thank you for the advice,the new feature of windowing seem able to solve my problem but only concern is the in memory capability to hold 1 hour data , may i know any example for how to configure /do the disk cache ?

Between i found out some example of doing append in hdfs

http://stackoverflow.com/questions/32339602/append-to-file-in-hdfs-cdh-5-4-5

the CDH platform can do the append ?

Re: Can HdfsBolt append data to existing file?

sunile_manjee — Sun, 24 Jul 2016 12:29:08 GMT

If you are concerned about memory you can persist the data to hdfs and once the window period is over recombine all persisted data and push to your hour location.

Re: Can HdfsBolt append data to existing file?

jason_huewk — Sun, 24 Jul 2016 13:58:14 GMT

Thank you for the answer