About robi

robi · ‎11-11-2019

My name is Robin and certified Cloudera admin/developer and hands on Hadoop CDH and HDP as architect, admin and data engineer for last 8-9 years now. Working on scala and python for ETL and ML/DL projects, found so much fun working with scala and python program. Always try to stay on top of the technology.

robi · ‎06-05-2018

thank you so much Gerd, well, this is one of 6 master nodes, this is not NN or SNN. so what is your suggestion?

robi · ‎06-04-2018

the case here is one of the server need to add more NIC card, so it requests a reboot, what is the steps to reboot one server only in the cluster?

robi · ‎10-13-2017

yes, I already been through these doc, they are only in java, I prefer scala, forgot to mention it in my question. thank you very much, I may have to try java then.

robi · ‎10-13-2017

I am struggled to find a connector for kafka stream data and dump to mongodb, never find a completed use case or good example, any one can help. thank you so much.

robi · ‎09-28-2017

Thank you so much.

robi · ‎09-28-2017

yes, you are right HDP is license free. however the cluster install needs master and slave infrastructure, so all the HDFS and data name, name node, yarn alone with some mandatory like Hive, pig, tez need installed and maintained. All of these cost a lots on AWS/EC2. in other hand, without HDP cluster, the monitoring, upgrade, version compatiblility and security on kafka, spark, zookeeper may bring issues in long run. admin have to deal with it. I am looking for use case of zookeeper/kafka cluster and spark to compare to see if it is worth to do so. How do you think?

robi · ‎09-27-2017

Thank you very much Wang for confirm the zk and kf and work alone. yes, with HDP, kafka and zookeeper is better administrated and monitored. I did setup setup kafka cluster and mongodb with HDP, it seemed very easy steps. However, I tried to save some money for our company, so I came up with this question. thank you for confirm it.

robi · ‎09-27-2017

my project need to stream data from kafka to mongodb, so we did setup HDP cluster and kafka multiple nodes within. After all, I feel we may process all data by using kafka/zookeeper alone, no need to have cluster. any one can tell if kafka stream can work itself without in a cluster? if yes, anythine I need to aware?

robi · ‎08-03-2017

yes, thank you, all working. I missed ';' at end of update statement. thanks again.

robi · ‎08-03-2017

It is strange, I add a new user admin1 and setup password for this user. after this, I lost my admin passwd, I dont know how it related. however, I try to reset admin passwd by: ambari-admin-password-reset, I got this, which /usr/sbin/ambari-admin-password-reset /usr/bin/which: no ambari-admin-password-reset in (/usr/sbin) what can I do from this point, thank you for your help.

robi · ‎07-11-2017

in my case here, I ignored the warning and later I found I namenode just could not start, after increase namenode java heap size to 4 gb, its all working. I think there is a calculation behind scenes, if your figure dont meet the number they come up with, you would get a connection refused error. connection refused is a such generic error, can be caused by all kinds of reasons. try to increase your namenode java heap size then save your change and restart namenode again. I have attached snapshot on change namenode java heap size.

robi · ‎07-07-2017

thank you for your time, this is the /etc/hosts file, I have 6 of them b1 ec2-54-216-249-113.us-west-1.compute.amazonaws.com 54.216.249.113 ip-172-31-4-232.us-west-1.compute.internal b2 ec2-54-215-167-71.us-west-1.compute.amazonaws.com 54.216.167.71 ip-172-31-14-204.us-west-1.compute.internal b3 ec2-54-192-74-19.us-west-1.compute.amazonaws.com 54.192.74.19 ip-172-31-0-58.us-west-1.compute.internal b4 ec2-54-183-108-236.us-west-1.compute.amazonaws.com 54.183.108.236 ip-172-31-12-149.us-west-1.compute.internal netstat -tulpn | grep -i 8083 output is nothing

robi · ‎07-07-2017

I got this telnet localhost 8083, connect to address 127.0.0.1: Connection refused please let me know what I can do from here. 8083 is default port for kafka connector, how come it refused connection?

robi · ‎07-06-2017

I have 3 nodes in kafka cluster, and need to create connector to stream data to mongodb. so on kafka side, I need to create this connector, seemed it has some issues. when I run this curl : curl -X POST -H "Content-Type: application/json" --data @/tmp/mongo_connector_configs.json http://localhost:8083/connectors 1. always get connect to localhost:8083 refused, after change to FQDN, still the same error. 2. So cat id_rsa.pub>>authorized_keys, the error is gone, no errors after execute curl command above. 3. when netstats -tunlp|grep 8083, the output is nothing -- this means 8083 is not listen 4. after I started producer, and enter some doc, I just dont see data insert to mongdb at all. Dont let mongdb part freak you out, it is all about kafka connector creation. Can any one help me with this issue? thank you very much. Robin

robi · ‎07-05-2017

1. make sure your ntpd, it need runing on all hosts. ntpd status and ntpd start 2. check your selinux file if SELINUX=disabled 3. your /etc/hosts file include all host privte ip and public ip. i.e. ip-..... ec2... 10.23.24.234 4. disable the iptables, service iptables stop. or systemctl stop firewall 5. reboot machine and make sure ntpd status is active. this worked for me all the time. you need to clikc on the 'check here for warnings' to see exact info.

robi · ‎07-05-2017

1. make sure your ntpd, it need runing on all hosts. ntpd status and ntpd start 2. check your selinux file if SELINUX=disabled 3. your /etc/hosts file include all host privte ip and public ip. i.e. ip-..... ec2... 10.23.24.234 4. disable the iptables, service iptables stop. or systemctl stop firewall 5. reboot machine and make sure ntpd status is active. this worked for me all the time. you need to clikc on the 'check here for warnings' to see exact info.

robi · ‎07-05-2017

Hi, I came back to distributed mode again. After ssh-keygen -t rsa and cat id_rsa.pub >> authorized_keys, this may cause the 'connection refused' error in my case. It was able to create the connector, and create testCollection in mongodb, however, no data in the testCollection. can you give a tip on why I could not pass the data? also, netstat -tnulp|grep 8083 output nothing. so it means the connector is not created, right? Thank you so much. Robin

robi · ‎07-04-2017

Thank you very much. I am trying to connect-standalone, and replace localhost to public IP, I still got the same error: 1) ./bin/connect-standalone.sh /tmp/connect-standalone .properties [2017-07-04 16:20:19,573] INFO Usage: ConnectStandalone worker.properties connec tor1.properties [connector2.properties ...] (org.apache.kafka.connect.cli.Connec tStandalone:61) 2) curl -X POST -H "Content-Type: application/json" --data @/tmp/mongo_connector_configs.json http://ec2-54-111-22-333.us-west-1.compute.amazonaws.com:8083/connectors curl: (7) Failed connect to ec2-54-153-75-126.us-west-1.compute.amazonaws.com:8083; Connection refused Can you help? thanks, Robin

robi · ‎07-04-2017

this 'kafka logs.dir/meta.properties' was the info very much help for my task here. thank you @mthiele.

robi · ‎07-04-2017

yum install mysql-community-client Loaded plugins: amazon-id, rhui-lb, search-disabled-repos No package mysql-community-client available.

robi · ‎07-04-2017

Thank you Jay. 1. I checked all host hdp-select, all versions are the same. 2. even if I find one out of sync, I dont know how to run this SELECT * FROM host_version; I guess I install oracle database for metadata, not mysql, please let me know if I am wrong. I just installed one more zookeeper, it allowed me to add new service. I know it dont make sense, but that is only thing I did. I think you provide all right answers. just let me know how to this query SELECT * FROM host_version; thank you again.

robi · ‎07-04-2017

I saw your ealier post # mysql -u ambari ambari -pbigdata mysql> SELECT host_id, state FROM host_version WHERE state = "OUT_OF_SYNC"; [root@ip-172-31-6-79 bin]# mysql bash: mysql: command not found

robi · ‎07-04-2017

703-version.png please see the 'current 6/7', it means one of my hosts have issues, but I could not find any alert in my cluster and only warning in deploy.

robi · ‎07-04-2017

703-version.png sorry, somehow it not uploaded last time. here it comes. please see current 6/7, one of my host do have issues, but I could not find any alert in any hosts in the cluster. I dont know how to query on ambari database, I did hdp-select, it is hard to compare the versions on each states on all hosts.

robi · ‎07-04-2017

Please refer to this image of the current version 6/7, seemed one of my host has issues, but how to fix it?

robi · ‎07-04-2017

thank you so much for your respond. ambari-server.log showed this error: error=/var/run/ambari-server/stack-recommendations/1/stackadvisor.err [root@ip-172-31-6-79 bin]# free -m total used free shared buff/cache available Mem: 7565 1478 119 48 5967 5714 Swap: 0 0 0

robi · ‎07-04-2017

Hi, I just installed a new HDP 2.5.3.0 cluster with 10 nodes, when I tried to 'add new service' it is in 'Loading....' and get no where. this is first time to my cluster, I have installed so many, never have this problem. I did all ambari-server restart, stop all service and start all service, also, chown root:root /var/run/ambari still not let me add new service. anyone have any ideas to help me here? thank you very much. Robin

robi · ‎07-02-2017

Hi, gurus, I have kafka 3 nodes cluster and try to create a kafka connector to mongodb. I have followed this link, https://github.com/startappdev/kafka-connect-mongodb. Every step is ok, till create a connector by curl, curl -X POST -H "Content-Type: application/json" --data @/tmp/mongo_connector_configs.json http://localhost:8083/connectors an error after this: curl(7) connection refused. my port 8083 is not used by anyone on my localhost, I used 2181 for zeekeeper, 6667 for producer, what this 8083 does here? hope someone setup this before so can share some details with me. thank you very much. Robin

robi · ‎06-16-2017

yes, this link provide exact the same commands. however, do you think Hortonworks better have these commands in the Ambari installation. instead gave a warning and having installer to check all over the places to find out?

Online	Offline
Last Visited	‎11-12-2019 02:14 AM

Date Registered	‎08-28-2015 01:05 AM
Last Visited	‎11-12-2019 02:14 AM
Total Messages Posted	194
Total Kudos Received	44

Re: Ambari install failed

Re: INTRODUCE YOURSELF: Please Reply here to tell ...

Re: what if I only like to reboot one server in th...

what if I only like to reboot one server in the cl...

Re: Need your help with a connector for kafka stre...

Need your help with a connector for kafka stream d...

Re: can kafka multiple nodes work by itself withou...

Re: can kafka multiple nodes work by itself withou...

Re: can kafka multiple nodes work by itself withou...

can kafka multiple nodes work by itself without an...

Re: @Jacqualin jasmin, I am on AWS/EC2/HDP, and I ...

@Jacqualin jasmin, I am on AWS/EC2/HDP, and I lost...

Re: Upgrade from HDP 2.5.3 to 2.6.1 fails: Connect...

Re: Issues when create kafka-connect-mongodb, conn...

Re: Issues when create kafka-connect-mongodb, conn...

Issues when create kafka-connect-mongodb, connecti...

Re: Ambari install failed

Re: Ambari install failed

Re: when create kafka connector, use localhost:808...

Re: when create kafka connector, use localhost:808...

Re: Ambari 2.2 Kafka broker.id missing?

Re: ambari add new service stuck

Re: ambari add new service stuck

Re: ambari add new service stuck

Re: ambari add new service stuck

Re: ambari add new service stuck

Re: ambari add new service stuck

Re: ambari add new service stuck

ambari add new service stuck

when create kafka connector, use localhost:8083, a...

Re: Commands to disable hugepage on linux at host ...