Hi @gkeys I think Nifi and Sqoop are two different tools serving two different use cases and cannot directly be compared, at least not yet. Sqoop is bundled with bulk loading adapters developed by database vendors and/or Hortonworks together. The purpose of Sqoop is bulk loading of data to and from the RDBMS. It uses fast connectors designed for bulk loading. Sqoop's performance is measured based on the bulk loading tool it is using. Since these are specialized, bulk loading tools designed for batch jobs, Sqoop really shines with these use cases. Nifi on the other hand is a system designed to move data within the organization as well as bring data in from outside sources and also facilitate data movement between data centers. The data Nifi helps move across is usually live data from applications, logs, devices and other sources producing event data. Given Nifi is so rich in its features, you can also use it to fetch data from lot of other sources including databases and files. For reading data from databases, Nifi uses a JDBC adapter. This would enable you to move x number of records at a time from some database. The bottleneck being the JDBC adapter. When we measure nifi's performance, we are not including the performance of fetching data from the source. What we are measuring is how fast Nifi is able to move data across as soon as it gets it. That performance is documented here and it's about 50 MB/s of read/write on a typical server. Can a JDBC source deliver data at this rate? Honestly, I doubt it but this has nothing to do with Nifi. It's more a function of the driver and the database and a lot of other variables just like in any other jdbc program. ... View more

@Farhad Heybati There are a number of components in Hadoop and its ecosystem. Each of them have their own high availability/failover strategy and different implications in case of a failure. You already mentioned Namenode and YARN Resource Manager. But there are others like HiveServer2, Hive Metastore, HMaster if you are using HBase and other components. Each have it's own documentation available on Hortonworks website. 1. Namenode also known as Master node is the linchpin of Hadoop. If namenode fails, your cluster is officially lost. To avoid this scenario, you must configure standby namenode. Instructions to setup Namenode HA can be found here. 2. YARN Resource Manager. YARN manages your cluster resources. Basically what job/application should get how much memory/cpu resources is allocated using YARN. So that's pretty important. While YARN has concept of Application Master, Node Manager and Container but what is really a single point of failure is YARN resource manager. So you need HA for Resource Manager. Check these two links. link 1 and link 2. 3. Hive Server2. What if you are using Hive (SQL) to query structured data in Hadoop? Assume you have multiple concurrent jobs running or adhoc users running their queries and they connect to Hive using HiveServer2. What if HiveServer2 goes down? Well, you need redundancy for that. Here is how you do it. 4. Are you using HBase? HBase has a component called HMaster. While not quite as crucial as HiveServer2 or Resource Manager but if HMaster goes down, you might see an impact specially if a region server also goes down before you are able to bring HMaster up. So you need to setup HA for HMaster. Check this link. I hope this helps. If you have any followup question, please feel free to ask. ... View more

@Cameron Warren You need to first scp your file to Azure. Once that's done, you can do "copyFromLocal" to copy file to your hdfs. hdfs dfs -copyFromLocal /path/to/file /dest/path/on/hdfs ... View more