@Rajesh Reddy - No, wget only tests HTTP/S connections. Not plain TCP that Kafka and Zookeeper are using ... View more

You will need to query the Hive metastore to 1. Filter on External tables 2. JOIN all tables (TBLS table) with all databases (DBS table) 3. Select the path See here, for example - https://stackoverflow.com/questions/44151670/search-a-table-in-all-databases-in-hive Or if you cannot connect to the metastore, you will need to scan over Hive tables - https://stackoverflow.com/questions/35004455/how-to-get-all-table-definitions-in-a-database-in-hive ... View more

I would suggest you use HDFS connect rather than Spark Streaming as it is more fault tolerant. Kafka Connect is built into the base Kafka libraries, but you need to compile and add HDFS Connect separately to the classpath of Connect. Build from here: https://github.com/confluentinc/kafka-connect-hdfs and use a tagged branch rather than master as the releases are publicly available libraries, not SNAPSHOT builds that require you to compile Kafka from source. ... View more

Ambari doesn't control any data residing on the datanodes, so you should be safe there. What I would do is let all the Hadoop components remain running "in the dark", by stopping all ambari-agents in the cluster, maybe even uninstalling it. Then, install and setup a new Ambari server, add a cluster, but register no hosts. Configure each of stopped ambari agents to point at the new Ambari server address, and start them. Add the hosts in the Ambari server UI, selecting "manual registration" option at the bottom of the dialog. Hopefully all the hosts register successfully. After which, you are given the option of installing clients and servers. Now, you could try to "reinstall" what is already there, but you might want to deselect all the servers on the datanode column. In theory, it will try to perform OS package installation, and say that the service already exists, and doesn't error out. If it does error, then restart the install process, but deselect everything -- At which point, it should continue and now you have Ambari back up and running with all the hosts monitored, just with no processes to configure. To add the services back, you would need to use the Ambari REST API to add back in the respective Services, Components, and Host Components that you have running on the cluster. If you can't remember what those are from all the things you have the options of installing in HDP, then go to each host and do a process check to see what's running. ... View more

@Rakesh AN I have worked for at least three companies trying to follow Agile/Scrum, and their development cycles of code does follow it. It's hard to upgrade hundreds of Hadoop nodes and software versions, make sure they all work with other components of the cluster, all without breaking other pieces in two-week sprints, though. Stand up meetings are all about perception management between team members and management. It again, has no special relationship or difference whether it is Hadoop development, web or mobile development, etc. ... View more

It might be beneficial to stuff this in a Docker container and run make prod tarball Then, you can run the Docker container for the respective environment, and simply copy the tarball to external clusters. ... View more

I assume this returns a limited result set, though, for large tables? ... View more