@rajdip chaudhuri As mentioned by @Abdelkrim Hadjidj, NiFi is a great candidate to solve these kinds of issues! He talked about processors like list & fetch files. As the name would have suggested, they list and fetch the data without you having to write any code and also gives you the properties attached to those files. For example the directory structure, a very important need you have mentioned in your use case.
A bunch of advantages that you can have using NiFi for this use case.
- No need to write any code
- Advanced functionality that will help you "maintain the state". That is the files which are only new will be listed. This will help you run your "file server" functionality in the real time as well if needed.
- Very high throughput with low latency.
- Rapid data acquisition pipeline development without writing a lot of code.
- Provides a highly concurrent model without a developer having to worry about the typical complexities of concurrency.
- Is inherently asynchronous which allows for very high throughput and natural buffering even as processing and flow rates fluctuate
- The resource-constrained connections make critical functions such as back-pressure and pressure release very natural and intuitive.
- The points at which data enters and exits the system as well as how it flows through are well understood and easily tracked
- And biggest of all, OPEN SOURCE.
Let know if you need some any other help!
