@BigDataRocks @mclark You will need to plan your nifi production cluster based on your volume requirements. - If you are just looking to transfer huge volume of data from a source to sink, you need to ensure you have enough space available for content repository. Also, ensure that your content repository is setup on a separate disk from flowfile and provenance repository. - Also from productionizing perspective, it is important to have error handling built in your flows, so your teams can get notified in case of errors and any errors are logged to logs. - It will probably be better to run multiple instances of nifi for data from multiple sources, because currently nifi doesn't offer security based on flows. In current security model, flow administrator will have access to all flows running on one instance. By running multiple instances of nifi, you can control security for each flow. - If you decide to use one instance of nifi, you can use ProcessGroups to organize your dataflows. - You should also think about setting up MonitorTasks for disk usage and memory that will give you warnings at appropriate thresholds. - For dataflows, with significant processing requirements, you will need a cluster setup to distribute load across different nodes. You can also increase number of concurrent tasks for any processor that requires more processing power.
... View more
