Hi @Avijeet Dash Generally speaking NiFi will handle that absolutely fine, I've seen it used to move very large video files with no issue, you'll need to ensure that the nodes have relevant disk, cpu and memory to support the file sizes you're interested, but otherwise no major concerns! Hope that helps! ... View more

Hi @Uday Vakalapudi Typically you will always be better off with multiple machines (scale out) rather than a smaller number of large machines (scale up). If you consider the way that Hadoop works, jobs are effectively distributed across the whole cluster and all the resources can be utilised simultaneously. This is the opposite of what virtualisation typically handles, which is multiple machines with different workloads and different workload profiles (I/O, cpu, memory). My short suggestion would be if you're just looking at a test/dev/pilot system, then multiple VM's is fine. But for production, consider scale out on bare metal. Hope that helps. , Typically you will always be better off with multiple machines (scale out) rather than a smaller number of large machines (scale up). If you consider the way that Hadoop works, jobs are effectively distributed across the whole cluster and all the resources can be utilised simultaneously. This is the opposite of what virtualisation typically handles, which is multiple machines with different workloads and different workload profiles (I/O, cpu, memory). My short suggestion would be if you're just looking at a test/dev/pilot system, then multiple VM's is fine. But for production, consider scale out on bare metal. ... View more

Hi @Kaliyug Antagonist You can use whatever filesystem you like for the O/S filesystems etc, our recommendations are primarily targeted at the HDFS data drives. I wouldn't use ext3 for anything any more, ext4 and xfs have moved forward as being the primary default options now. So to try and address your options one by one: 1) No, don't do this. 2) Perfectly acceptable, take care that data drives are mounted with recommended mount options (noatime etc) 3) Also perfectly acceptable, I see more people using XFS everywhere now, ext4 less so, but the deltas are relatively small, I'd go with whichever option you're more comfortable with as an organisation. 4) I wouldn't recommend that, if you're happy using XFS, use it everywhere, it just makes things easier but see point 2) about mount options for data drives 5) You can absolutely use LVM for your O/S partitions, just ideally don't use it for datanode and log directories. Hope that helps! ... View more

All great points, I've always wished that our tutorials came with a short "release notes" link for each one too. ... View more

Hi @Sree Venkata. To ad to Neeraj's already excellent answer and to follow your comment, NiFi now *does* support kerberised clusters. Also there is now an RDBMS connector, although I'd still say, use SQOOP if you're transferring very large chunks of RDBMS data and you want it parellised across the whole hadoop cluster, use NiFi if youve got smaller chunks to transfer that can be parallelised over a smaller NiFi cluster. Hope that (in combination with Neeraj's answer) fulfills your requirements. ... View more

Are all services started? Have you tried restarting the ones that have issues? ... View more