About silaphetmounkhaty-3293181

silaphetmounkhaty-3293181 · ‎03-27-2017

Hi Communnity, Our Development VM CDH 5.9 is HDFS HA enabled. Since there are only five nodes so two JournalNodes are running same with NN but another one is on a DataNode. However, last night, we ran into this below issue and took down both NNs. FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: flush failed for required journal (JournalAndStream(mgr=QJM to [xxx.xx.xx.xx:8485, xxx.xx.xx.xx:8485, xxx.xx.xx.xx:8485], stream=QuorumOutputStream starting at txid 9861544)) Sine this is a Dev cluster so no one was using it runing weekend or nights. After extensive search, the potential issue could be NN Garbarge collection pause. Is there a good approach how to debug and tweak the heap setting? Currently, NN heap setting is 4GB on both. Default time out is 20 seconds ( dfs.qjournal.select-input-streams.timeout.ms). Any help really appreciated that. Thanks, Silaphet

silaphetmounkhaty-3293181 · ‎02-28-2017

Hi, Quick update, it works with Firefox. However, with HDFS HA, it does not work well when we rollover active NN to another node. Is there away to configure this with HDFS HA enabled? Thanks, Silaphet

silaphetmounkhaty-3293181 · ‎02-23-2017

Hi, I have been looking to configure the "Google Chrome" on my Mac laptop by the command does not work. Could you please validate the command and I can give it another try? I don't want to turn off the HTTP web-consoles if possible. Thanks, SIlaphet

silaphetmounkhaty-3293181 · ‎02-15-2017

Hi Community, I have a few of CDH clusters with Kerberos enabled but I've been facing issue to not able to access the NN URL. Here is error message: ================= HTTP ERROR 401 Problem accessing /index.html. Reason: Authentication required Powered by Jetty:// ================= However, Yarn NM, Job history and Spark URLs are just working fine. I've checked with my team who are using both Windows and Mac laptops and they are facing same issue. THis issue still happens with latest CDH5.10 Is there a workaround to address this issue for Mac OS? Thanks, Silaphet

silaphetmounkhaty-3293181 · ‎01-19-2017

Hi, my wget command accidently downloaded old manifest.json and that was the reason CM was not able to detect it. Once I got correct manifest.json and everything was working as expected. This has been fixed and thanks for your response. Thanks, Silaphet

silaphetmounkhaty-3293181 · ‎01-18-2017

Hi All, I've successfully upgraded a development cluster to CDH5.9 with Parcels 5.9.0-1.cdh5.9.0.p0.23 activated. Now, I wanted to deploy Spark2.0 running side by side with 1.6 Spark facing issue. Here are steps. 1. Download the Spark2 csd from below. http://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html 2. Move it to default /opt/cloudera/csd and set permissions accordingly. 3. Restart both CM server and management services. 4. Spark2 shows up when try adding services but failed to start. When try deploying Client config and CM complaining additional Parcels need to be activated. When I look at below link latest parcels and only available for 5.7 for latest. http://archive.cloudera.com/spark2/parcels/latest/ What am I missing here? where can I find additional parcels for Spark2.0 to download? Your help or instruction is really appreciated. Thanks, Silaphet

silaphetmounkhaty-3293181 · ‎12-13-2016

Hi, Sorry for the delay response. Currently, kerberos is using CLOUDERA Realm and all serices are using principals from local KDC. The goal is to generate new pricipals from IPA and have all services using new principals. Also, we want to replace current realm with new realm from IPA. What is correct step process of this migration work? Thanks, Silaphet

silaphetmounkhaty-3293181 · ‎11-29-2016

Hi, After correcting a few parameters on configuration files and we were able to address the issue. Another quick question, is there any info avaialble step by step on disabling Kerberos? We wanted to remove MIT kerberos then enable it with Redhat IPA. Thanks, Silaphet

silaphetmounkhaty-3293181 · ‎11-21-2016

Hi Gabor, We made a typo with Custom Script. We were able to get all principals created in the RHIM. Now, new issue is we can't start CDH because we can't locate KDC. Currently, all dns lookups are disabled. Here is example of zookeeper. Nov 21, 1:07:37.468 PMERRORorg.apache.zookeeper.server.quorum.QuorumPeerMain Unexpected exception, exiting abnormally java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Cannot locate KDC at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:135) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79) Do you happen to have any suggestions where to check? Thanks, Silaphet

silaphetmounkhaty-3293181 · ‎11-17-2016

Hello Community, We just migrated hour Hadoop nodes to Redhat Identity Management (RHIM) recently. Now, we want to enable Kerberos on our CDH5.5 with Redhat IPA by using Cloudera custom script to create all pricipals and distribute keytabs accordingly. The script is located on /etc/cloudera-scm-server on CM host and owned by cloudera-scm user. Here is the error log message: ======= 2016-11-17 11:29:51,736 ERROR GenerateCredentials-0:com.cloudera.cmf.security.GenerateCredentialsCommand: unable to create credential for role 68 due to:/etc/cloudera-scm-server/gen_credentials_ipa.sh failed with exit code 1 and output of << SASL Bind failed Can't contact LDAP server (-1) ! ====== What are we missing on OS and CDH configuration point of view? Thanks, Silaphet

silaphetmounkhaty-3293181 · ‎05-25-2016

Hi Pd, Yes, extra space was the root cause. Thanks very much for great trick. Thanks, Silaphet

silaphetmounkhaty-3293181 · ‎05-25-2016

Hi, Sorry for the delay. The "Zookeeper Root" is /kafa. I ran below command still no luck. bin/kafka-console-consumer --zookeeper localhost:2181 /kafka --topic kafkatest --from-beginning I tried FQN instead of localhost still no luck. Tried different Zookeeper node still the same. The Kafka version is KAFKA-2.0.1-1.2.0.1.p0.5 Please let me if there is another option to try. Thanks, Silaphet

silaphetmounkhaty-3293181 · ‎05-17-2016

Hi, I'm testing Kafka sending messages by running below commands from seperate terminals. bin/kafka-console-producer --broker-list localhost:9092 --topic kafkatest This is a message This is another message On another terminal: bin/kafka-console-consumer --zookeeper localhost:2181 --topic kafkatest --from-beginning No brokers found in ZK. Why the above messages don't show up on consumer terminal? This is CDH5.5.2 with Parcel deployment. What am I missing on CDH/CM configurations? Thanks very much, Silaphet

silaphetmounkhaty-3293181 · ‎05-27-2015

Hello, This is first time we are configuring mySQL as external databases for hive metastore and rest of CM components (Service Monitor, activity monitor etc...) We are separating to have one node for CM and anothe node for mySQL. We were able to get the CM management services up running but it takes more than five minutes login to hive CLI. Once in Hive CLI, hive commands just hangs and can't run any hive queries. When I look at the "metastore" database in mysql, I don't see any tables created. Below is the link that I followed. http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_ig_mysql.html What am I missing here? not enough permission or something else? Thanks, Silaphet

silaphetmounkhaty-3293181 · ‎12-08-2014

That did it. Thanks again for your quick response. Thanks, Silaphet

silaphetmounkhaty-3293181 · ‎12-08-2014

Hi Kafka Guru, I really can't figure out why I can't create a Kafka topic for validation testing. Below is my creation command and I ran this commond on the zookeeper node. kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testkafka Here is a very well known error message: Error while executing topic command org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47) at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685) at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:413) at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:409) at kafka.utils.ZkUtils$.getChildren(ZkUtils.scala:480) at kafka.utils.ZkUtils$.getSortedBrokerList(ZkUtils.scala:81) at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:154) at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:86) at kafka.admin.TopicCommand$.main(TopicCommand.scala:50) at kafka.admin.TopicCommand.main(TopicCommand.scala) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1496) at org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:99) at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:416) at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:413) at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) ... 8 more On my Cloudera Manager Enterprise 5.2, below are Broker configuration properties: Service-Wide: Zookeeper Root (zookeeper.chroot) /kakfa Kafka Broker Default Group (Broker.id) Default value (empty). I reset it to 1 then restart both Broker and zookeeper nodes but still not fixed. Where else should I look into to get this issue resloved? Your help is really appreciated. Thanks, Silaphet

silaphetmounkhaty-3293181 · ‎12-02-2014

Hi, I see they are hidden directories. After deleting both .gnome2 and .mozzila and I was able to get the broker up and running. From CM, I can see /var/lib/kafka is the log directory. I guess /var/log/kafka is server log, is that correct? Big thanks to you for a quick response. Thanks, Silaphet

silaphetmounkhaty-3293181 · ‎12-02-2014

Hi, There is no file called .gnome2 under /var/lib/kafka. There is only recovery-point-offset-checkpoint under this directory. However, on a working test cluster, there are two files under this directory. Here they are: recovery-point-offset-checkpoint replication-offset-checkpoint The difference between these two clusters is test cluster has only one zookeeper compared to three zookeeper nodes on production. I'm not convinced that this would be root cause of it. Do you know where else I should look at? Thanks, Silaphet

silaphetmounkhaty-3293181 · ‎12-02-2014

Hi, I successfully installed Kafka on my test cluster but when I followed same steps on production cluster I am running into an issue. I followed Cloudera documentation step by step. I was able to download, distributed and activated the Kafka Parcel. When I tried adding "Kafka-Cloudera-Labs" on any hadoop nodes and I was NOT able to get the broker up and running. Here is the error log that can be available. 2014-12-02 09:00:37,815 INFO org.I0Itec.zkclient.ZkClient: zookeeper state changed (SyncConnected) 2014-12-02 09:00:37,853 INFO kafka.log.LogManager: Loading log '.gnome2' 2014-12-02 09:00:37,855 FATAL kafka.server.KafkaServerStartable: Fatal error during KafkaServerStable startup. Prepare to shutdown java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1911) at kafka.log.Log$.parseTopicPartitionName(Log.scala:740) at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogMa at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogMa at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimize at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:105) at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113) at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimize at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34) at kafka.log.LogManager.loadLogs(LogManager.scala:105) at kafka.log.LogManager.<init>(LogManager.scala:57) at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:293) at kafka.server.KafkaServer.startup(KafkaServer.scala:73) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala: at kafka.Kafka$.main(Kafka.scala:46) at kafka.Kafka.main(Kafka.scala) 2014-12-02 09:00:37,858 INFO kafka.server.KafkaServer: [Kafka Server 159], shutt I did exact same thing on a test cluster and it worked very well. Both production and test clusters are running same CM5.2 and CDH5.2.0 versions. What could be potential issue and how to fix this issue? Please help!!! Thanks, Silaphet

Online	Offline
Last Visited	‎08-19-2019 11:18 PM

Date Registered	‎12-02-2014 11:43 AM
Last Visited	‎08-19-2019 11:18 PM
Total Messages Posted	19
Total Kudos Received	1

Re: No brokers found in ZK

Error: flush failed for required journal

Re: NameNode URL not accesible on Kerberos Cluster

Re: NameNode URL not accesible on Kerberos Cluster

NameNode URL not accesible on Kerberos Cluster

Re: How to deploy Spark2.0 on CDH5.9

How to deploy Spark2.0 on CDH5.9

Re: Unable to create Kerberos Credentials with Clo...

Re: Unable to create Kerberos Credentials with Clo...

Re: Unable to create Kerberos Credentials with Clo...

Unable to create Kerberos Credentials with Clouder...

Re: No brokers found in ZK

Re: No brokers found in ZK

No brokers found in ZK

Configure external mySQL for hive metastore

Re: KeeperErrorCode = NoNode for /brokers/ids

KeeperErrorCode = NoNode for /brokers/ids

Re: Can't start Kafka Service broker

Re: Can't start Kafka Service broker

Can't start Kafka Service broker