@Matthias Tewordt If you have only 3 nodes in your cluster then we could try the manual registration. But before we go that direction did you follow the documented step of preparing the environment Set Up Password-less SSH Enable NTP on the Cluster and on the Browser Host Check DNS and NSCD Configuring iptables Disable SELinux and PackageKit and check the umask Value Enable NTP on Your Cluster Disable IPTables THP Just to be sure..... If you accomplished the above then, adapt the below for your OS specific. I assume your repos were correctly set and are accessible, you can validate with # yum repolist You should be able to see HDP,HDP-UTILS and Ambari repos Install ambari-agent on all nodes include the Ambari node # yum install -y ambari-agent Edit the ambari-agent.ini see example located at /etc/ambari-agent/conf on all the hosts and [server] hostname={Ambari_FQDN} url_port=8440 secured_url_port=8441 connect_retry_delay=10 max_reconnect_retry_delay=30 This should hostname should be the Ambari server FQDN Start the agent # ambari-agent start The above should be done on all the 3 nodes and ensure they started successfully Log on the Ambari UI and follow the previous steps, now in the host registration process chose manual after, you won't need the ssh generated key The process should complete with GREEN and there you can proceed with your deployment Please revert ... View more

@Matthias Tewordt Could you take a backup of cert-verification.cfg cp /etc/python/cert-verification.cfg /etc/python/cert-verification.cfg.bak Then update /etc/python/cert-verification.cfg to have verify=disable or please create this file sed -i 's/verify=platform_default/verify=disable/' /etc/python/cert-verification.cfg Retry and let me know ... View more

@Vaughn Shideler As hdfs user could you try this $ hdfs dfs -chown -R falcon /apps/falcon/SourceCluster/staging Check the permissions $ hdfs dfs -ls /apps/falcon/SourceCluster You should have falcon:hdfs Then retry that should work ... View more

@Michael Bronson The manual delete should take care of broker that's down that because you as reiterated you have to shut down the cluster (broker down) Now about your last question with worries about Kafka02, if you know how Kafka stores partitions then you will realize the for example you have 3 brokers and created a topic with 6 partitions with replication factor 1. Each cluster will be responsible for 2 partitions. The replication-factor has been set to 1, which means data is not being replicated and the data for a particular partition will only be stored on one server, so here the key is in the replication factor! Whats the retention policy? That also plays a role as to whether you consume restart consuming from the beginning! You could add new Kafka Brokers to a cluster and move existing topics to new Brokers with all the topics intact? see this HCC Kafka document You have to weigh your options to avoid data loss, the reason a kafka deployment should be well thought through DR strategies ... View more

@Michael Bronson I highly doubt whether you can delete or purge the topics when the broker is down PURGE topic You can drain the topic by expiring the messages ./kafka-topics --zookeeper {ZKADR} --alter --topic topic_name --config retention.ms=1000 ./kafka-topics --zookeeper {ZKADR} --alter --topic topic_name --delete-config retention.ms How to delete data from topic To delete manually: Shutdown the cluster Clean kafka log dir (specified by the log.dir attribute in kafka config file ) as well the zookeeper data Restart the cluster For any given topic what you can do is Stop kafka Clean kafka log specific to partition, kafka stores its log file in a format of "logDir/topic-partition" so for a topic named "MyTopic" the log for partition id 0 will be stored in /tmp/kafka-logs/MyTopic-0 where /tmp/kafka-logs is specified by the log.dir attribute Restart kafka Hope that helps ... View more

@Mokkan Mok The HDP and Ambari upgrades will only impact the related binaries, but you should also test their compatibility against bespoke/third party tools that are plug to the hadoop cluster e.g Presto,Juypter ,tableau ,etc ... View more

@Shailna Patidar Any feedback? To check the new replication factor use the below command in this example my_secret has a replication factor of 4 $ hdfs dfs -ls Found 4 items drwx------ - hive hdfs 0 2014-01-29 06:14 .staging -rw-r--r-- 4 hive hdfs 1943 2014-01-24 01:01 my_secret drwxr-xr-x - hive hdfs 0 2014-04-22 12:45 test drwxr-xr-x - hive hdfs 0 2014-04-22 12:45 payroll.csv Hope that helped ... View more

@Şükrü ERGÜNTOP Can you include the sqoop command? What is the size of the data that you are transferring from Oracle database? What value did you give --m option ? Java was not able to allocate enough memory, i.e. it's not Java's heap limit that's in the way but rather no more memory available to be given to Java by OS. Check that the machine is not running out of memory. And first clean ram or increase ram then check if again there is an out of memory error then increase heap size: -Xms128m min(heap size) -Xmx512m max(heap size) -XX:MaxPermSize max(perm size) Hope that helps ... View more

@Matthias Tewordt Can you create this directory and ensure the permissions are correct mkdir -p /var/lib/ambari-agent/data Then re-run the cluster setup ... View more

@Bhushan Kandalkar I think you need to check this documentation it will help you configure connectivity between Hue and HS2. Please revert ... View more