I have around ten years of experience in working on data management and Business Intelligence tools and technologies. Would like to explore Hadoop extensively. Interested to learn and work on new age Hadoop services  to master the data science domain. Tushar. ... View more

Can you more elaborate on "Hostname tied to the actual IP address?" and " Use "ifconfig -a" to see a listing of your network interfaces and choose one that has an actual IP address. "   How do I know which hostname and ip address to use. Because while installing single node cluster of cloudera how would you know which hosts to specify?   Thank you! ... View more

Very well explained, thanks for the post~ ... View more

thank you so much Gerd, well, this is one of 6 master nodes, this is not NN or SNN. so what is your suggestion? ... View more

try add allow port on firewalld on host, this solved my prob $ firewall-cmd --zone=public --add-port=9000/tcp (Centos7) or $ service firewalld disable Hope this may help ... View more

in /etc/hosts for all nodes: put ip_adress   FQDN   DN  10.10.1.230 name.domain.com name   the FQDN must be before the name ... View more

@Robin Dong, https://rklicksolutions.wordpress.com/2017/04/04/read-data-from-kafka-stream-and-store-it-in-to-mongodb/ https://github.com/alonsoir/hello-kafka-twitter-scala These links may help you. ... View more

Thank you so much. ... View more

yes, thank you, all working. I missed ';' at end of update statement. thanks again. ... View more

thank you for your time, this is the /etc/hosts file, I have 6 of them b1 ec2-54-216-249-113.us-west-1.compute.amazonaws.com 54.216.249.113 ip-172-31-4-232.us-west-1.compute.internal b2 ec2-54-215-167-71.us-west-1.compute.amazonaws.com 54.216.167.71 ip-172-31-14-204.us-west-1.compute.internal b3 ec2-54-192-74-19.us-west-1.compute.amazonaws.com 54.192.74.19 ip-172-31-0-58.us-west-1.compute.internal b4 ec2-54-183-108-236.us-west-1.compute.amazonaws.com 54.183.108.236 ip-172-31-12-149.us-west-1.compute.internal netstat -tulpn | grep -i 8083 output is nothing ... View more

Hi, Can u reset the ambari-agent in all servers and try to register. amabri-agent reset <Ambari_host_name> ... View more

@Robin Dong You can check the "/etc/ambari-server/conf/ambari.properties" file to know exactly which DataBase your ambari is using and then accordingly you can login to the DB host and run/check the requested commands or the DB admin can assist with the queries. Example: (In my Case it is Postgres) # grep 'jdbc' /etc/ambari-server/conf/ambari.properties custom.postgres.jdbc.name=postgresql-9.3-1101-jdbc4.jar previous.custom.postgres.jdbc.name=postgresql-9.3-1101-jdbc4.jar server.jdbc.connection-pool=internal server.jdbc.database=postgres server.jdbc.database_name=ambari server.jdbc.postgres.schema=ambari server.jdbc.user.name=ambari server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat . ... View more

I know this is an old post, but maybe it can still be usefull to someone. To start the standalone you need to pass both the worker config (connect-standalone.properties) and the connectors configurations (mongo_connector_config.properties in Robin's case). For someone with the same problem, bare in mind that for starting the standalone connector you need to pass ".properties" files. To update or add new connectors you would need to pass ".json" files. You can see in the confluent documentation page here. bin/connect-standalone worker.properties connector1.properties [connector2.properties connector3.properties ...] Vicente ... View more

That's correct https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-troubleshooting/content/resolving_cluster_install_and_configuration_problems.html ... View more

yes, this link provide exact the same commands. however, do you think Hortonworks better have these commands in the Ambari installation. instead gave a warning and having installer to check all over the places to find out? ... View more

thank you so much. even I need more time to digest it, I can tell this is what I need to know. thank you for spending time to help. Robin ... View more

Thank you Cdraper, Finally I use this combination, RHEL 7.3 (m4.large) , ambari 2.4.10 and HDP 2.5 install 7 nodes on AWS. the only issues I deal with is change the namenode java heap size to 4gb in my case. then everything worked. your info is still very helpful. I am going to setup kafka 3 nodes and mongodb from here. will keep you posted. thanks again. Robin ... View more

both of #openssl s_client -connect ip-172-31-13-143.us-west-2.compute.internal:8440 and 8441 have no output. ... View more

sorry, I terminated all nodes, however, I know this alert info: connection failed to http://ip-172-31-15-246.us-west-2.compute.internal on every node. some ports just dont open and directories not exist. my passwdless setup correctly for sure. however, connection refused, it is IP problem? if need static IP, limitation is 5. I have more than 10 nodes. Can you help? ... View more

oh, my ambari service is external mode, use http://ec2....amazon.com:8080, if this is so called external mode. after ambari-server reset, I got the same error. the repo is: http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.0.1/ambari.repo rpm -qa|grep ambari-metrics -- show all packages are installed. so the hbase configure info is the problem. it refuse the connection. I dont need to edit hbase configure with ambari 2.4 in the past at all. any changes on HDP? thanks, Robin ... View more

you are right, Since my setting all working before I never use VCP and just thought about this is may be a cause. I will work on it and get back to you. thanks again. ... View more

still like to know the best combination of RHEL version and ambari.repo version. document says it will fit all, but different error comeup. ... View more

thanks, I got your another post and tried ambari 2.2 so dont need to deal with smartsense. ... View more

According to the mongodb doc here, https://www.mongodb.com/blog/post/in-the-loop-with-hadoop-take-advantage-of-data-locality-with-mongo-hadoop This means that we can accomplish local reads by putting a DataNode process (for Hadoop) or a Spark executor on the same node with a MongoDB shard and a mongos. In other words, we have an equal number of MongoDB shards and Hadoop or Spark worker nodes. We provide the addresses to all these mongos instances to the connector, so that the connector will automatically tag each InputSplit with the address of the mongos that lives together with the shard that holds the data for the split. the above mentioned that we have an equal number of MongoDB shards and Hadoop or Spark worker nodes. I think it means number of mongodb shard = data nodes with spark. so my next step is to ensure sparks installed on all data nodes and then mongodb shards on these worker nodes. Let me know if you think differently. thanks Robin ... View more

I will close this case I got my answer here. ... View more

Thanks for the link @Eyad Garelnabi.I have HDP 2.4 installed on ubuntu,followed the same steps provided but getting the following errors during Install,start and test wizard in Ambari.Any help would be thankful. Fail: Applying File['/etc/yum.repos.d/mongodb.repo'] failed, parent directory /etc/yum.repos.d doesn't exist ... View more

Thank you so much Dave. after some search, we may have to use kafka producer API for this complex data streaming. However, your answer give me a new thought on nifi with kafka. Thank you very much for your time to help. Robin ... View more

How have you installed HDP/HDF? When you do a standard Ambari install of HDP it will ask you where you want to put each service, and you can choose to put Kafka on separate nodes from the other services. You can also move Services using the Ambari interface, generally by putting them in maintenance mode on a particular node, stopping them, then selecting to move them to another node. I would suggest that if you need more Kafka brokers in the same environment you should have them all controlled by the same Ambari, yes. ... View more