About robi

MehulGain · ‎05-07-2020

We are Data transformation company, do everything from Ingestion to Insights. We have 320+ Consultants and spread across India, Middle East, APAC and North America. We have been working with Hortonworks and MapR in past and also looking at services/implementation partnership for Cloudera. Have 180+ customers across Regions. Have a Strategic AL/ML Platform with solution build for Marketing Mix / Test-Go / Demand Planning / Pricing & Packaging etc. Have customers across Domain and have partnership with Qlik, Informatica, Talend, SAS etc. Kindly align to Partner Manager for Implementation Partnership.

Shriniwas · ‎11-15-2019

Can you more elaborate on "Hostname tied to the actual IP address?" and " Use "ifconfig -a" to see a listing of your network interfaces and choose one that has an actual IP address. " How do I know which hostname and ip address to use. Because while installing single node cluster of cloudera how would you know which hosts to specify? Thank you!

LydiaFIL · ‎01-14-2019

Very well explained, thanks for the post~

robi · ‎07-18-2018

I just started machine learning and deep learning, seemed somehow deep learning use total different libs and way to generate results. so I wonder, if all my study on machine learning would need to do the deep learning? your help is greatly appreciated. Robin

robi · ‎06-05-2018

thank you so much Gerd, well, this is one of 6 master nodes, this is not NN or SNN. so what is your suggestion?

robi · ‎02-28-2018

any one can help me with this error. I basically upgrade ambari from 2.4.2.3 to 2.6.1.3 by following steps in HDP docs. do I miss any steps or is any thing I need to do to fix this error Also, only server and agent are 2.6.1.3 all others still stay at 2.4. why? rpm -qa | grep ambari ambari-metrics-monitor-2.4.2.0-136.x86_64 ambari-metrics-hadoop-sink-2.4.2.0-136.x86_64 ambari-infra-solr-client-2.4.2.0-136.x86_64 ambari-metrics-grafana-2.4.2.0-136.x86_64 ambari-agent-2.6.1.3-3.x86_64 ambari-server-2.6.1.3-3.x86_64 ambari-metrics-collector-2.4.2.0-136.x86_64

resolution · ‎02-14-2018

try add allow port on firewalld on host, this solved my prob $ firewall-cmd --zone=public --add-port=9000/tcp (Centos7) or $ service firewalld disable Hope this may help

faycal11 · ‎10-19-2017

in /etc/hosts for all nodes: put ip_adress FQDN DN 10.10.1.230 name.domain.com name the FQDN must be before the name

asirna · ‎10-13-2017

@Robin Dong, https://rklicksolutions.wordpress.com/2017/04/04/read-data-from-kafka-stream-and-store-it-in-to-mongodb/ https://github.com/alonsoir/hello-kafka-twitter-scala These links may help you.

robi · ‎09-28-2017

Thank you so much.

robi · ‎08-08-2017

my project is to stream data from kafka to mongodb, so I have to use mongodbConfig.Host mongodbConfig.Database, mongodbConfig.Collection etc. when sbt compile, I got errors on all these mongodbConfig: not found: value mongodbConfig But I found no libraries in SCALA to import for mongodbConfig. anyone can help? thank you so much.

robi · ‎08-03-2017

yes, thank you, all working. I missed ';' at end of update statement. thanks again.

robi · ‎07-31-2017

My project is to stream data from kafka and pass to mongodb by a spark/scala program. I found this example, this is the first statement in the code-- SparkSession.builder().master("local[*]").config("spark.some.config.option”, “some-value").getOrCreate() I have no idea what to replace "some.config.option" and "some-value". any one can give me an example. I am new to scala. please help. Thank you very much.

robi · ‎07-28-2017

i tried to compile a piece of scala by SBT, I have a build.sbt file like this: name := "kf_mongodb" version := "1.0" scalaVersion := "2.11.1" val sparkVersion = "2.2.0" libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVersion, "org.apache.spark" %% "spark-sql" % sparkVersion, "org.apache.spark" %% "spark-mllib" % sparkVersion, "org.apache.spark" %% "spark-streaming" % sparkVersion, "org.apache.spark" %% "spark-hive" % sparkVersion, "com.stratio.datasource" % "spark-mongodb_2.11" % "0.12.0" however, compile error islnot found: type StringDeserializer [error] "key.deserializer" -> classOf[StringDeserializer], any one have idea what I need to add to my build.sbt or any .jar file need to copy to my project root dir?

robi · ‎07-27-2017

Make long story short, on my AWS EC2 cluster, we need stream data from kafka to mongodb. so sbt, eclipse for scala is required and need integrated. however, ec2 instances dont have x11, no gui interface, it is impossible to use eclipse for scala. and AWS have a tool called toolkit required many steps to install and configure with securities. . so I have local eclipse for scala installed and do some precompile then upload my .scala, pom.xml to SBT on ec2 instance, I know the build.sbt is on the project base directory, but not sure pom.xml is under same dir or not. Please provide some info if know you where pom.xml goes. thank you very much.

robi · ‎07-16-2017

Hi, my system is HDP 2.5 and kafka with 3 brokers. all data stream to Mongodb. so I have to use the kafka-mongodb-sink connector to connect kafka to mongodb. my question here is can I create 3 connectors connect kafka and mongodb on all 3 brokers and stream data to mongodb to the same db/collections? I could not find the info regarding this topic in document, please help. thanks, Robin

robi · ‎07-07-2017

hello, when I create a kafka connector on my multiple kafka brokers on AWS/EC2, I run this command: curl -X POST -H "Content-Type: application/json" --data @/tmp/mongo_connector_configs.json http://localhost:8083/connector and this is the error: curl: (7) Failed connect to localhost:8083; Connection refused my json file it setup to 777, and have all configure info there. so can anyone tell me how to open this 8083 port. your help is greatly appreciated. Robin

robi · ‎07-07-2017

thank you for your time, this is the /etc/hosts file, I have 6 of them b1 ec2-54-216-249-113.us-west-1.compute.amazonaws.com 54.216.249.113 ip-172-31-4-232.us-west-1.compute.internal b2 ec2-54-215-167-71.us-west-1.compute.amazonaws.com 54.216.167.71 ip-172-31-14-204.us-west-1.compute.internal b3 ec2-54-192-74-19.us-west-1.compute.amazonaws.com 54.192.74.19 ip-172-31-0-58.us-west-1.compute.internal b4 ec2-54-183-108-236.us-west-1.compute.amazonaws.com 54.183.108.236 ip-172-31-12-149.us-west-1.compute.internal netstat -tulpn | grep -i 8083 output is nothing

kalaiselvan_moo · ‎07-07-2017

Hi, Can u reset the ambari-agent in all servers and try to register. amabri-agent reset <Ambari_host_name>

jsensharma · ‎07-04-2017

@Robin Dong You can check the "/etc/ambari-server/conf/ambari.properties" file to know exactly which DataBase your ambari is using and then accordingly you can login to the DB host and run/check the requested commands or the DB admin can assist with the queries. Example: (In my Case it is Postgres) # grep 'jdbc' /etc/ambari-server/conf/ambari.properties custom.postgres.jdbc.name=postgresql-9.3-1101-jdbc4.jar previous.custom.postgres.jdbc.name=postgresql-9.3-1101-jdbc4.jar server.jdbc.connection-pool=internal server.jdbc.database=postgres server.jdbc.database_name=ambari server.jdbc.postgres.schema=ambari server.jdbc.user.name=ambari server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat .

vicente_rocha · ‎04-23-2018

I know this is an old post, but maybe it can still be usefull to someone. To start the standalone you need to pass both the worker config (connect-standalone.properties) and the connectors configurations (mongo_connector_config.properties in Robin's case). For someone with the same problem, bare in mind that for starting the standalone connector you need to pass ".properties" files. To update or add new connectors you would need to pass ".json" files. You can see in the confluent documentation page here. bin/connect-standalone worker.properties connector1.properties [connector2.properties connector3.properties ...] Vicente

robi · ‎06-26-2017

We have HDP 2.5 and kafka 0.10.0,this version require no extra package for kafka stream. However, we have Mongodb in the cluster and need data stream, read data from kafka then pass to Mongodb. there is a mongodb connector called mongodb-sink/confluent and configure is not simple. so I wonder, if I use kafka 0.10.0 stream, do I still need mongodb-sink? in other words, can consumer output directly dump data to mongodb? I am new to mongodb, like to get some idea on this data pipe process. Please help. Robin

icocio · ‎07-03-2017

That's correct https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-troubleshooting/content/resolving_cluster_install_and_configuration_problems.html

robi · ‎06-16-2017

yes, this link provide exact the same commands. however, do you think Hortonworks better have these commands in the Ambari installation. instead gave a warning and having installer to check all over the places to find out?

robi · ‎06-11-2017

I just installed a HDP on AWS/EC2 and mongodb. Didn't get all details documented. when configure mongdb, I messed up my cluster. when I try to install new one, it never work. no matter what OS, ambari.repo and HDP version. I tried all combinations. nothing worked so far. Since I have installed so many times, I dont miss anything when setup, I think, so I really think this is version compatibility. my name node never start, hdfs, yarn, etc, etc. Only zookeeper is up running. I understand ambari.repo 2.4.1.0 is a very stable version, so I take it and install with AWS/EC2 REHL 6 and 7 then HDP 2.4 and 2.5.. all dont work. Please advise. thanks, Robin

robi · ‎06-08-2017

thank you, I already sent a mail to amcbarnett who is owner of the first link to ask this question and I already talk to the guy who is owner of the second link. I followed the exactly RHEL 7.1 m4.large, and ambari 2.4.01 on AWS. somehow dont work for me, there are warnings on installation and ambari-metrics-collector never work. Do you have any good combinations you have installed? thank you so much for spending time for my question. Robin

robi · ‎06-09-2017

thank you so much. even I need more time to digest it, I can tell this is what I need to know. thank you for spending time to help. Robin

robi · ‎06-08-2017

0. when try to check on ambari-metics-collector log file, I could not find one. 1. there is no ambari-metrics-collector installed. 2. after manually install ambari-metics-collector, the ambari-metics-collection still could not start and stop quicker:) any one have ideas? thank you very much.

robi · ‎06-14-2017

Thank you Cdraper, Finally I use this combination, RHEL 7.3 (m4.large) , ambari 2.4.10 and HDP 2.5 install 7 nodes on AWS. the only issues I deal with is change the namenode java heap size to 4gb in my case. then everything worked. your info is still very helpful. I am going to setup kafka 3 nodes and mongodb from here. will keep you posted. thanks again. Robin

aanghel · ‎06-12-2017

@Robin Dong In Linux only iptables controls the Kernel based firewall. You might have firewalld in CentOS7 or ufw in Ubuntu but they're just an abstraction layer on top of iptables. So if 'iptables -L' doesn't show anything then it's all good. The Ambari iptables check is rudimentary and it doesn't know if the rules that exist still allow all the traffic. It only checks for 'service iptables status' or 'systemctl status firewalld', which means there are no filter tables. But please be aware of the cloud firewall as well. For example in AWS even instances in the same Security Group are not allowed by default to communicate with each another and this must be enabled explicitly: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/security-group-rules-reference.html#sg-rules-other-instances

Online	Offline
Last Visited	‎11-12-2019 02:14 AM

Date Registered	‎08-28-2015 01:05 AM
Last Visited	‎11-12-2019 02:14 AM
Total Messages Posted	194
Total Kudos Received	45

Re: Ambari install failed

Re: INTRODUCE YOURSELF: Please Reply here to tell ...

Re: Installation failed. Failed to receive heartbe...

Re: How to create an Edge Node when creating a clo...

do I need to get familiar with all ML skill before...

Re: what if I only like to reboot one server in th...

upgrade error ERROR [main] AmbariServer:1074 - Fa...

Re: Cluster installation - The inspector failed to...

Re: Failed to receive heartbeat from agent. (Curre...

Re: Need your help with a connector for kafka stre...

Re: can kafka multiple nodes work by itself withou...

what library to import to use MongodbConfig?

Re: @Jacqualin jasmin, I am on AWS/EC2/HDP, and I ...

any one used this command -- SparkSession.builder...

scala error: not found: type StringDeserializer [e...

anyone can tell me where the pom.xml file should b...

can kafka multiple brokers use multiple connectors...

kafka http://localhost:8083/connectors curl: (7) F...

Re: Issues when create kafka-connect-mongodb, conn...

Re: Ambari install failed

Re: ambari add new service stuck

Re: when create kafka connector, use localhost:808...

Any one worked on kafka stream and mongodb project...

Re: 2 commands to disable hugepage on linux

Re: Commands to disable hugepage on linux at host ...

@Neeraj Sabharwal, can you please share info about...

Re: which version is the best combination of RHEL ...

Re: I have assigned 800 gb on all my instance when...

[Error 111] connection refused to 0.0.0.0:6188 on ...

Re: HDFS NameNode Web UI connection faile to http:...

Re: systemctl stop iptables or systemctl disable i...