Hi @SC SC, DS2v2 is too small for data nodes and not properly sized for benchmarking. Please see https://community.hortonworks.com/articles/22376/recommendations-for-microsoft-azure-hdp-deployment-1.html for Azure sizing recommendations and HDP. /Best regards, Mats ... View more

Hi @Greg Lawrance, The only bridge AFAIK is MITI Meta Bridge which supports both PDI and Atlas in Beta. There are a couple of projects on Github which will give an idea on how you can do it (REST API, Java). https://github.com/shivajid/atlas https://github.com/yhemanth/atlas-demo https://github.com/vakshorton/NifiAtlasBridge /Best regards, Mats ... View more

Hi @Subramanian Santhanam, Please provide the complete exception as the above error is quite general. /Best regards, Mats ... View more

Hi @Mahesh Mallikarjunappa, The diagram below show the principles of ZK leader election. There is also quite a nice tutorial available at http://www.allprogrammingtutorials.com/tutorials/programming-with-apache-zookeeper.php /Best regards, Mats ... View more

Hi @Manoj Dhake, The simple answer is yes. Two authorization methods are available for Atlas: Simple and Ranger. The simple method is described in section 2.1.3 here: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_data-governance/content/ch_hdp_data_governance_install_atlas_ambari.html The Ranger Authorization requires enabling the Ranger Atlas plug-in in Ambari. Aftre that you ca add Atlas users in Ranger. Also the Sandox contains the holger_gov/holger_gov user for ATlas demo purposes. /Best regards, Mats ... View more

Hi @Daniel Wang, I checked this in HDP 2.5 and bootstrap.py have the following set on approx line 40: HOST_BOOTSTRAP_TIMEOUT = 300 /Best regards, Mats ... View more

Hi @Mayank Pandey, Please check - https://community.hortonworks.com/questions/20719/sqoop-to-sql-server-with-integrated-security.html Also the JIRA SQOOP-795 covers Kerberos authentication for JDBC clients to Sqoop. /Best regards, Mats ... View more

Hi @Jacky Hung, EMC did a presentation at Spark Summit this summer which is available at: http://www.slideshare.net/SparkSummit/which-is-deeper-comparison-of-deep-learning-frameworks-on-spark /Best regards, Mats ... View more

Hi @Ahmad Debbas, Depends on which distro and which version we are using. As long as it's Apache HDFS and the same version it will definitely work. For Apache HDFS but different version we need to upgrade and that procedure might differ depending on versions. /Best regards, Mats ... View more

Hi @vshukla, How do I enable impersonation for %sh? I can't find it in the Zeppelin GUI or in the docs? /Best regards, Mats ... View more

Hi @oula.alshiekh@gmail.com alshiekh, Easiest way is to use sqoop, for more info please visit: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_dataintegration/content/using_sqoop_to_move_data_into_hive.html http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_data-movement-and-integration/content/ch_data_movement_using_sqoop.html You can also load files into Hadoop (you need to extract it to a file first from your SQL Server), see: http://hortonworks.com/hadoop-tutorial/loading-data-into-the-hortonworks-sandbox/ /Best regards, Mats ... View more

Hi @PARIVALLAL R, Looks like there are some mismatch in your Ambari database? For example do you have Atlas installed/re-installed and/or have you done an upgrade? If an upgrade, make sure all post-upgrade tasks have been completed. Also check the Ambari database tables ambari.clusterservices and ambari.hostcomponentstate for Atlas entries. /Best regards, Mats ... View more

Hi @Ashika Umagiliya, You need to start ntpd as well with: systemctl start ntpd /Best regards, Mats ... View more

Hi @Tristan Fily, The handle_other(): error looks like a false positive error in HUE (see https://issues.cloudera.org/browse/HUE-2198) so it might not have anything to do with the problem. Please try to gather more information from more logs, change debug log level etc. /Best regards, Mats ... View more

Hi @Vinay Sharma, They are all internal values and will not probably affect your system IF they have not previously been tuned to solve a problem. I would go for the largest value in your cases and be prepared that *.mb values can affect memory consumption. dfs.namenode.handler.count - The number of server threads for the datanode, 10 default yarn.timeline-service.ttl-ms - Time to live for timeline store data in milliseconds. Defaults to 604800000 (7 days). mapreduce.task.io.sort.mb - The total amount of buffer memory to use while sorting files, in megabytes. Default 100, gives each merge stream 1MB, which should minimise seeks. yarn.app.mapreduce.am.resource.mb - The amount of memory the MR AppMaster needs, default 1536. /Best Regards, Mats ... View more

Hi @sivasaravanakumar k, The rate of replication work is throttled by HDFS to not interfere with cluster traffic when failures happen during regular cluster load. Some properties controlling this are dfs.namenode.replication.work.multiplier.per.iteration , dfs.namenode.replication.max-streams and dfs.namenode.replication.max-streams-hard-limit . The foremost controls the rate of work to be scheduled to a DN at every heartbeat that occurs, and the other two further limit the maximum parallel threaded network transfers done by a DataNode at a time. Some description of this is available at https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml /Best regards, Mats ... View more