Thank you!! ... View more

here is a single link from a repo command after just 3 saves:   <a href="<a href="<a href="http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo" target="_blank">http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo</a>" target="_blank"><a href="http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo</a" target="_blank">http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo</a</a>>" target="_blank"><a href="<a href="http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo</a" target="_blank">http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo</a</a>" target="_blank"><a href="http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo</a</a" target="_blank">http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo</a</a</a>>> && yum --enablerepo=extras install epel-release -y && yum install nano java java-devel ambari-server ambari-agent -y && ambari-server setup -s && ambari-server install-mpack --mpack=<a href="<a href="<a href="<a href="http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz" target="_blank">http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz</a>" target="_blank"><a href="http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz</a" target="_blank">http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz</a</a>>" target="_blank"><a href="<a href="http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz</a" target="_blank">http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz</a</a>" target="_blank"><a href="http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz</a</a" target="_blank">http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz</a</a</a>>>" target="_blank"><a href="<a href="<a href="http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz</a" target="_blank">http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz</a</a>" target="_blank"><a href="http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz</a</a" target="_blank">http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz</a</a</a>>" target="_blank"><a href="<a href="http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz</a</a" target="_blank">http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz</a</a</a>" target="_blank"><a href="http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz</a</a</a" target="_blank">http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz</a</a</a</a>>>> ... View more

Not sure, it is your folder/files.   If you are not sure yourself,  just make another folder inside of that folder and try to use it.      When you setup the data node to an empty folder it should resolve your issue. ... View more

First thing I see is:   Can't format the storage directory because the current/ directory is not empty.   Is the path/folder empty?    look for hidden files too... ... View more

Ok so bump up the min/max ram in the nifi.properties or in the config (if using ambari or other).   Start with 2gb / 4gb, and go up in steps until you get stability you are looking for.  You should not have to go as high as 16gb / 32gb to be stable.   The default is very low.       ... View more

Sizing and spec of the nifi node:   Node Ram and Cores Min/max memory for nifi     Is the node dedicated to NiFi?  If the Node has other duties too, these may prevent NiFi from having enough resources causing it to crash or get locked up during long term testing (days/weeks).  Restarting NiFi resets this condition, and then it works again.     First step is to increase the min/max memory,  but sharing the Nifi Node Environment details will help us provide more specific details. ... View more

  It sounds like your NiFi needs some tweaking, to get things stable.  You should not have to restart NiFi.   If it is stopping/crashing/clogging up after some time online running as expected, this is usually due to instability of NiFi itself, not the processor.     Can you provide some details about the spec/sizing for the nifi?     Also what is the size of data arriving?   ... View more

With that big of a file, you are going to need to adjust settings (memory) and evaluate performance of the file repositories.  I would recommend to start with small files, and work your way up.  Complete a benchmark at each stage, then begin testing config changes.  This should be a lengthy evaluation where in the end, you end up with a configuration capable of consuming 3 gb files. ... View more

Not to take away from the entire conversation above, which in fact was very detailed and specific comparison.  The major take away in your pro/con evaluation needs to be Physical Disk compared to Network or some level of shared Disk.  Also in a big ha system there are usually more than one disk (not to mean more than one partition).  When you go past dev and POC level benchmarking, deep into performance tuning, the Physical Disk, in high availability arrays, with a physical machine will out perform the Cloud or VMs for large volume and large data processes.   To get more specific you have to compare all the nuts and bolts as well as evaluate the Performance Best Practices for each platform, service, component, all the way down to application design.   This is a great debate and one that I have at every customer.   That said I have led prod clusters installs in the cloud: Amazon, Azure, IBM Cloud, Google Cloud, and Private Cloud and VM systems.   ... View more