Hi @SC SC, DS2v2 is too small for data nodes and not properly sized for benchmarking. Please see https://community.hortonworks.com/articles/22376/recommendations-for-microsoft-azure-hdp-deployment-1.html for Azure sizing recommendations and HDP. /Best regards, Mats ... View more

Hi @Greg Lawrance, The only bridge AFAIK is MITI Meta Bridge which supports both PDI and Atlas in Beta. There are a couple of projects on Github which will give an idea on how you can do it (REST API, Java). https://github.com/shivajid/atlas https://github.com/yhemanth/atlas-demo https://github.com/vakshorton/NifiAtlasBridge /Best regards, Mats ... View more

HI @Mehul Shah, You probably missing Java JCE support or UNlimited Strength Chryptographic plocies (USC) or have some mismatch with your keys for OpenSSL. Please visit http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.1.2/bk_dataflow-ambari-installation/content/distribute_and_install_the_jce.html (This is for HDF 2.1 installed with Ambari but same doc is available for other install options). There are also some info around PKCS12/OpenSSL in https://issues.apache.org/jira/browse/NIFI-3062. /Best regards, Mats ... View more

Hi @vamsi valiveti, Oozie is a scheduler and Flume is not working on a schedule basis instead Flume is treating the data when it receives it. So you use teh Flume configuration to tell for example that each time there is a file in a certain directory Flume will put it in hdfs (if you use the spooldir source) and so on. /Best regards, Mats ... View more

Hi @Arsalan Siddiqi, I recommend you to use Ambari as it will give you access to all configuration parameters and also to data files in HDFS by using the Ambari File View. Please be aware that if you are using Ambari, any changes to config needs to be done through the Ambari GUI or API and editing the config files directly will not work (they are just a copy of the Ambari config). /Best regards, Mats ... View more

Hi @anand maurya, Please have a look at the Hortonworks-Securosis White Paper regarding Security in Hadoop. The paper can be found at: http://hortonworks.com/info/securing-hadoop/ /Best regards, Mats ... View more

Hi @subash sharma, Make sure your Java environment is configured properly - for example JAVA_HOME. Please have a look at https://community.hortonworks.com/questions/39839/how-to-import-metadata-from-hive-into-atlas-and-th.html /Best regards, Mats ... View more

Hi @vshukla, How do I enable impersonation for %sh? I can't find it in the Zeppelin GUI or in the docs? /Best regards, Mats ... View more

Hi @sivasaravanakumar k, The rate of replication work is throttled by HDFS to not interfere with cluster traffic when failures happen during regular cluster load. Some properties controlling this are dfs.namenode.replication.work.multiplier.per.iteration , dfs.namenode.replication.max-streams and dfs.namenode.replication.max-streams-hard-limit . The foremost controls the rate of work to be scheduled to a DN at every heartbeat that occurs, and the other two further limit the maximum parallel threaded network transfers done by a DataNode at a time. Some description of this is available at https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml /Best regards, Mats ... View more

Hi @Jitendra, Please have a look at http://hadoop.apache.org/versioning.html - It cover Ambari as well. /Best regards, Mats ... View more

Sorry, too quick on the trigger. Normally SPARK_HOME=/usr/hdp/current/spark-client In my (old) sandbox the start_master.sh script is located at /usr/hdp/2.4.0.0-169/spark/sbin/start-master.sh. ... View more

Hi, Not sure what you mean with Spark on R? If that is RHadoop, MS R Server or DistributedR those are not really comparable just like that with SparkR, ie no drop-in replacement, as the functionality differs. So before doing a performance comparison you have to look into which functionality you would like to compare. /Best regards, Mats ... View more

Hi @Uday Vakalapudi, HDP do not require any special sizing for /dev as long as we follow the OS recommendations. The "baseline" we present includes general Linux requirements for system partitions. So in this case I recommend to follow the requirements for /dev issued by the operating system vendor. ... View more

@Predrag Minovic - Is it possible to remove the auditlog in hdfs as well without affecting lineage etc? ... View more

Looks like hadoop-yarn-common-x.x.x.jar is missing. ... View more