Member since
08-28-2015
194
Posts
45
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2349 | 07-05-2017 11:58 PM |
11-11-2019
10:50 PM
1 Kudo
My name is Robin and certified Cloudera admin/developer and hands on Hadoop CDH and HDP as architect, admin and data engineer for last 8-9 years now. Working on scala and python for ETL and ML/DL projects, found so much fun working with scala and python program. Always try to stay on top of the technology.
... View more
06-05-2018
07:31 PM
thank you so much Gerd, well, this is one of 6 master nodes, this is not NN or SNN. so what is your suggestion?
... View more
06-04-2018
06:00 PM
the case here is one of the server need to add more NIC card, so it requests a reboot, what is the steps to reboot one server only in the cluster?
... View more
10-13-2017
01:48 PM
yes, I already been through these doc, they are only in java, I prefer scala, forgot to mention it in my question. thank you very much, I may have to try java then.
... View more
10-13-2017
02:25 AM
I am struggled to find a connector for kafka stream data and dump to mongodb, never find a completed use case or good example, any one can help. thank you so much.
... View more
Labels:
- Labels:
-
Apache Kafka
09-28-2017
03:02 PM
Thank you so much.
... View more
09-28-2017
11:05 AM
yes, you are right HDP is license free. however the cluster install needs master and slave infrastructure, so all the HDFS and data name, name node, yarn alone with some mandatory like Hive, pig, tez need installed and maintained. All of these cost a lots on AWS/EC2. in other hand, without HDP cluster, the monitoring, upgrade, version compatiblility and security on kafka, spark, zookeeper may bring issues in long run. admin have to deal with it. I am looking for use case of zookeeper/kafka cluster and spark to compare to see if it is worth to do so. How do you think?
... View more
09-27-2017
08:15 PM
Thank you very much Wang for confirm the zk and kf and work alone. yes, with HDP, kafka and zookeeper is better administrated and monitored. I did setup setup kafka cluster and mongodb with HDP, it seemed very easy steps. However, I tried to save some money for our company, so I came up with this question. thank you for confirm it.
... View more
09-27-2017
01:12 AM
my project need to stream data from kafka to mongodb, so we did setup HDP cluster and kafka multiple nodes within. After all, I feel we may process all data by using kafka/zookeeper alone, no need to have cluster. any one can tell if kafka stream can work itself without in a cluster? if yes, anythine I need to aware?
... View more
Labels:
- Labels:
-
Apache Kafka
08-03-2017
02:22 PM
yes, thank you, all working. I missed ';' at end of update statement. thanks again.
... View more
08-03-2017
01:47 PM
It is strange, I add a new user admin1 and setup password for this user. after this, I lost my admin passwd, I dont know how it related. however, I try to reset admin passwd by: ambari-admin-password-reset, I got this, which /usr/sbin/ambari-admin-password-reset /usr/bin/which: no ambari-admin-password-reset in (/usr/sbin) what can I do from this point, thank you for your help.
... View more
Labels:
- Labels:
-
Apache Ambari
07-11-2017
02:24 AM
in my case here, I ignored the warning and later I found I namenode just could not start, after increase namenode java heap size to 4 gb, its all working. I think there is a calculation behind scenes, if your figure dont meet the number they come up with, you would get a connection refused error. connection refused is a such generic error, can be caused by all kinds of reasons. try to increase your namenode java heap size then save your change and restart namenode again. I have attached snapshot on change namenode java heap size.
... View more
07-07-2017
07:07 PM
thank you for your time, this is the /etc/hosts file, I have 6 of them b1 ec2-54-216-249-113.us-west-1.compute.amazonaws.com 54.216.249.113 ip-172-31-4-232.us-west-1.compute.internal
b2 ec2-54-215-167-71.us-west-1.compute.amazonaws.com 54.216.167.71 ip-172-31-14-204.us-west-1.compute.internal
b3 ec2-54-192-74-19.us-west-1.compute.amazonaws.com 54.192.74.19 ip-172-31-0-58.us-west-1.compute.internal b4 ec2-54-183-108-236.us-west-1.compute.amazonaws.com 54.183.108.236 ip-172-31-12-149.us-west-1.compute.internal netstat -tulpn | grep -i 8083
output is nothing
... View more
07-07-2017
03:53 PM
I got this telnet localhost 8083, connect to address 127.0.0.1: Connection refused please let me know what I can do from here. 8083 is default port for kafka connector, how come it refused connection?
... View more
07-06-2017
02:32 AM
I have 3 nodes in kafka cluster, and need to create connector to stream data to mongodb. so on kafka side, I need to create this connector, seemed it has some issues. when I run this curl : curl -X POST -H "Content-Type: application/json" --data @/tmp/mongo_connector_configs.json http://localhost:8083/connectors 1. always get connect to localhost:8083 refused, after change to FQDN, still the same error. 2. So cat id_rsa.pub>>authorized_keys, the error is gone, no errors after execute curl command above. 3. when netstats -tunlp|grep 8083, the output is nothing -- this means 8083 is not listen 4. after I started producer, and enter some doc, I just dont see data insert to mongdb at all. Dont let mongdb part freak you out, it is all about kafka connector creation. Can any one help me with this issue? thank you very much. Robin
... View more
Labels:
- Labels:
-
Apache Kafka
07-05-2017
11:58 PM
1. make sure your ntpd, it need runing on all hosts. ntpd status and ntpd start 2. check your selinux file if SELINUX=disabled 3. your /etc/hosts file include all host privte ip and public ip. i.e. ip-..... ec2... 10.23.24.234 4. disable the iptables, service iptables stop. or systemctl stop firewall 5. reboot machine and make sure ntpd status is active. this worked for me all the time. you need to clikc on the 'check here for warnings' to see exact info.
... View more
07-05-2017
11:58 PM
1 Kudo
1. make sure your ntpd, it need runing on all hosts. ntpd status and ntpd start 2. check your selinux file if SELINUX=disabled 3. your /etc/hosts file include all host privte ip and public ip. i.e. ip-..... ec2... 10.23.24.234 4. disable the iptables, service iptables stop. or systemctl stop firewall 5. reboot machine and make sure ntpd status is active. this worked for me all the time. you need to clikc on the 'check here for warnings' to see exact info.
... View more
07-05-2017
08:32 PM
Hi, I came back to distributed mode again. After ssh-keygen -t rsa and cat id_rsa.pub >> authorized_keys, this may cause the 'connection refused' error in my case. It was able to create the connector, and create testCollection in mongodb, however, no data in the testCollection. can you give a tip on why I could not pass the data? also, netstat -tnulp|grep 8083 output nothing. so it means the connector is not created, right? Thank you so much. Robin
... View more
07-04-2017
08:25 PM
Thank you very much. I am trying to connect-standalone, and replace localhost to public IP, I still got the same error: 1) ./bin/connect-standalone.sh /tmp/connect-standalone .properties
[2017-07-04 16:20:19,573] INFO Usage: ConnectStandalone worker.properties connec tor1.properties [connector2.properties ...] (org.apache.kafka.connect.cli.Connec tStandalone:61) 2) curl -X POST -H "Content-Type: application/json" --data @/tmp/mongo_connector_configs.json http://ec2-54-111-22-333.us-west-1.compute.amazonaws.com:8083/connectors curl: (7) Failed connect to ec2-54-153-75-126.us-west-1.compute.amazonaws.com:8083; Connection refused Can you help? thanks, Robin
... View more
07-04-2017
07:37 PM
1 Kudo
this 'kafka logs.dir/meta.properties' was the info very much help for my task here. thank you @mthiele.
... View more
07-04-2017
03:10 AM
yum install mysql-community-client
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
No package mysql-community-client available.
... View more
07-04-2017
03:07 AM
Thank you Jay. 1. I checked all host hdp-select, all versions are the same. 2. even if I find one out of sync, I dont know how to run this SELECT * FROM host_version; I guess I install oracle database for metadata, not mysql, please let me know if I am wrong. I just installed one more zookeeper, it allowed me to add new service. I know it dont make sense, but that is only thing I did. I think you provide all right answers. just let me know how to this query SELECT * FROM host_version; thank you again.
... View more
07-04-2017
02:29 AM
I saw your ealier post # mysql -u ambari ambari -pbigdata
mysql> SELECT host_id, state FROM host_version WHERE state = "OUT_OF_SYNC";
[root@ip-172-31-6-79 bin]# mysql
bash: mysql: command not found
... View more
07-04-2017
02:22 AM
703-version.png please see the 'current 6/7', it means one of my hosts have issues, but I could not find any alert in my cluster and only warning in deploy.
... View more
07-04-2017
02:20 AM
703-version.png sorry, somehow it not uploaded last time. here it comes. please see current 6/7, one of my host do have issues, but I could not find any alert in any hosts in the cluster. I dont know how to query on ambari database, I did hdp-select, it is hard to compare the versions on each states on all hosts.
... View more
07-04-2017
01:51 AM
Please refer to this image of the current version 6/7, seemed one of my host has issues, but how to fix it?
... View more
07-04-2017
01:28 AM
thank you so much for your respond. ambari-server.log showed this error: error=/var/run/ambari-server/stack-recommendations/1/stackadvisor.err [root@ip-172-31-6-79 bin]# free -m total used free shared buff/cache available Mem: 7565 1478 119 48 5967 5714 Swap: 0 0 0
... View more
07-04-2017
12:47 AM
Hi, I just installed a new HDP 2.5.3.0 cluster with 10 nodes, when I tried to 'add new service' it is in 'Loading....' and get no where. this is first time to my cluster, I have installed so many, never have this problem. I did all ambari-server restart, stop all service and start all service, also, chown root:root /var/run/ambari still not let me add new service. anyone have any ideas to help me here? thank you very much. Robin
... View more
Labels:
- Labels:
-
Apache Ambari
07-02-2017
11:02 PM
Hi, gurus, I have kafka 3 nodes cluster and try to create a kafka connector to mongodb. I have followed this link, https://github.com/startappdev/kafka-connect-mongodb. Every step is ok, till create a connector by curl, curl -X POST -H "Content-Type: application/json" --data @/tmp/mongo_connector_configs.json http://localhost:8083/connectors an error after this: curl(7) connection refused. my port 8083 is not used by anyone on my localhost, I used 2181 for zeekeeper, 6667 for producer, what this 8083 does here? hope someone setup this before so can share some details with me. thank you very much. Robin
... View more
Labels:
- Labels:
-
Apache Kafka
06-16-2017
02:44 PM
yes, this link provide exact the same commands. however, do you think Hortonworks better have these commands in
the Ambari installation. instead gave a warning and having installer to check all over the places to find out?
... View more