Member since
12-02-2014
19
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
31276 | 05-25-2016 02:06 PM |
03-27-2017
02:06 PM
Hi Communnity, Our Development VM CDH 5.9 is HDFS HA enabled. Since there are only five nodes so two JournalNodes are running same with NN but another one is on a DataNode. However, last night, we ran into this below issue and took down both NNs. FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: flush failed for required journal (JournalAndStream(mgr=QJM to [xxx.xx.xx.xx:8485, xxx.xx.xx.xx:8485, xxx.xx.xx.xx:8485], stream=QuorumOutputStream starting at txid 9861544)) Sine this is a Dev cluster so no one was using it runing weekend or nights. After extensive search, the potential issue could be NN Garbarge collection pause. Is there a good approach how to debug and tweak the heap setting? Currently, NN heap setting is 4GB on both. Default time out is 20 seconds ( dfs.qjournal.select-input-streams.timeout.ms). Any help really appreciated that. Thanks, Silaphet
... View more
Labels:
02-28-2017
02:40 PM
Hi, Quick update, it works with Firefox. However, with HDFS HA, it does not work well when we rollover active NN to another node. Is there away to configure this with HDFS HA enabled? Thanks, Silaphet
... View more
02-23-2017
08:31 AM
Hi, I have been looking to configure the "Google Chrome" on my Mac laptop by the command does not work. Could you please validate the command and I can give it another try? I don't want to turn off the HTTP web-consoles if possible. Thanks, SIlaphet
... View more
02-15-2017
09:04 AM
Hi Community, I have a few of CDH clusters with Kerberos enabled but I've been facing issue to not able to access the NN URL. Here is error message: ================= HTTP ERROR 401 Problem accessing /index.html. Reason: Authentication required Powered by Jetty:// ================= However, Yarn NM, Job history and Spark URLs are just working fine. I've checked with my team who are using both Windows and Mac laptops and they are facing same issue. THis issue still happens with latest CDH5.10 Is there a workaround to address this issue for Mac OS? Thanks, Silaphet
... View more
01-19-2017
08:04 AM
Hi, my wget command accidently downloaded old manifest.json and that was the reason CM was not able to detect it. Once I got correct manifest.json and everything was working as expected. This has been fixed and thanks for your response. Thanks, Silaphet
... View more
01-18-2017
02:28 PM
Hi All, I've successfully upgraded a development cluster to CDH5.9 with Parcels 5.9.0-1.cdh5.9.0.p0.23 activated. Now, I wanted to deploy Spark2.0 running side by side with 1.6 Spark facing issue. Here are steps. 1. Download the Spark2 csd from below. http://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html 2. Move it to default /opt/cloudera/csd and set permissions accordingly. 3. Restart both CM server and management services. 4. Spark2 shows up when try adding services but failed to start. When try deploying Client config and CM complaining additional Parcels need to be activated. When I look at below link latest parcels and only available for 5.7 for latest. http://archive.cloudera.com/spark2/parcels/latest/ What am I missing here? where can I find additional parcels for Spark2.0 to download? Your help or instruction is really appreciated. Thanks, Silaphet
... View more
Labels:
12-13-2016
01:47 PM
Hi, Sorry for the delay response. Currently, kerberos is using CLOUDERA Realm and all serices are using principals from local KDC. The goal is to generate new pricipals from IPA and have all services using new principals. Also, we want to replace current realm with new realm from IPA. What is correct step process of this migration work? Thanks, Silaphet
... View more
11-29-2016
10:43 AM
Hi, After correcting a few parameters on configuration files and we were able to address the issue. Another quick question, is there any info avaialble step by step on disabling Kerberos? We wanted to remove MIT kerberos then enable it with Redhat IPA. Thanks, Silaphet
... View more
11-21-2016
12:07 PM
Hi Gabor, We made a typo with Custom Script. We were able to get all principals created in the RHIM. Now, new issue is we can't start CDH because we can't locate KDC. Currently, all dns lookups are disabled. Here is example of zookeeper. Nov 21, 1:07:37.468 PMERRORorg.apache.zookeeper.server.quorum.QuorumPeerMain Unexpected exception, exiting abnormally java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Cannot locate KDC at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:135) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79) Do you happen to have any suggestions where to check? Thanks, Silaphet
... View more
11-17-2016
12:52 PM
Hello Community,
We just migrated hour Hadoop nodes to Redhat Identity Management (RHIM) recently. Now, we want to enable Kerberos on our CDH5.5 with Redhat IPA by using Cloudera custom script to create all pricipals and distribute keytabs accordingly. The script is located on /etc/cloudera-scm-server on CM host and owned by cloudera-scm user. Here is the error log message:
=======
2016-11-17 11:29:51,736 ERROR GenerateCredentials-0:com.cloudera.cmf.security.GenerateCredentialsCommand: unable to create credential for role 68 due to:/etc/cloudera-scm-server/gen_credentials_ipa.sh failed with exit code 1 and output of <<
SASL Bind failed Can't contact LDAP server (-1) !
======
What are we missing on OS and CDH configuration point of view?
Thanks,
Silaphet
... View more
Labels:
05-25-2016
02:06 PM
Hi Pd, Yes, extra space was the root cause. Thanks very much for great trick. Thanks, Silaphet
... View more
05-25-2016
10:52 AM
Hi, Sorry for the delay. The "Zookeeper Root" is /kafa. I ran below command still no luck. bin/kafka-console-consumer --zookeeper localhost:2181 /kafka --topic kafkatest --from-beginning I tried FQN instead of localhost still no luck. Tried different Zookeeper node still the same. The Kafka version is KAFKA-2.0.1-1.2.0.1.p0.5 Please let me if there is another option to try. Thanks, Silaphet
... View more
05-17-2016
12:28 PM
Hi, I'm testing Kafka sending messages by running below commands from seperate terminals. bin/kafka-console-producer --broker-list localhost:9092 --topic kafkatest This is a message This is another message On another terminal: bin/kafka-console-consumer --zookeeper localhost:2181 --topic kafkatest --from-beginning No brokers found in ZK. Why the above messages don't show up on consumer terminal? This is CDH5.5.2 with Parcel deployment. What am I missing on CDH/CM configurations? Thanks very much, Silaphet
... View more
05-27-2015
09:46 AM
Hello, This is first time we are configuring mySQL as external databases for hive metastore and rest of CM components (Service Monitor, activity monitor etc...) We are separating to have one node for CM and anothe node for mySQL. We were able to get the CM management services up running but it takes more than five minutes login to hive CLI. Once in Hive CLI, hive commands just hangs and can't run any hive queries. When I look at the "metastore" database in mysql, I don't see any tables created. Below is the link that I followed. http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_ig_mysql.html What am I missing here? not enough permission or something else? Thanks, Silaphet
... View more
12-08-2014
08:29 AM
That did it. Thanks again for your quick response. Thanks, Silaphet
... View more
12-08-2014
07:54 AM
Hi Kafka Guru, I really can't figure out why I can't create a Kafka topic for validation testing. Below is my creation command and I ran this commond on the zookeeper node. kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testkafka Here is a very well known error message: Error while executing topic command org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47) at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685) at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:413) at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:409) at kafka.utils.ZkUtils$.getChildren(ZkUtils.scala:480) at kafka.utils.ZkUtils$.getSortedBrokerList(ZkUtils.scala:81) at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:154) at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:86) at kafka.admin.TopicCommand$.main(TopicCommand.scala:50) at kafka.admin.TopicCommand.main(TopicCommand.scala) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1496) at org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:99) at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:416) at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:413) at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) ... 8 more On my Cloudera Manager Enterprise 5.2, below are Broker configuration properties: Service-Wide: Zookeeper Root (zookeeper.chroot) /kakfa Kafka Broker Default Group (Broker.id) Default value (empty). I reset it to 1 then restart both Broker and zookeeper nodes but still not fixed. Where else should I look into to get this issue resloved? Your help is really appreciated. Thanks, Silaphet
... View more
12-02-2014
01:55 PM
1 Kudo
Hi, I see they are hidden directories. After deleting both .gnome2 and .mozzila and I was able to get the broker up and running. From CM, I can see /var/lib/kafka is the log directory. I guess /var/log/kafka is server log, is that correct? Big thanks to you for a quick response. Thanks, Silaphet
... View more
12-02-2014
01:08 PM
Hi, There is no file called .gnome2 under /var/lib/kafka. There is only recovery-point-offset-checkpoint under this directory. However, on a working test cluster, there are two files under this directory. Here they are: recovery-point-offset-checkpoint replication-offset-checkpoint The difference between these two clusters is test cluster has only one zookeeper compared to three zookeeper nodes on production. I'm not convinced that this would be root cause of it. Do you know where else I should look at? Thanks, Silaphet
... View more
12-02-2014
11:43 AM
Hi, I successfully installed Kafka on my test cluster but when I followed same steps on production cluster I am running into an issue. I followed Cloudera documentation step by step. I was able to download, distributed and activated the Kafka Parcel. When I tried adding "Kafka-Cloudera-Labs" on any hadoop nodes and I was NOT able to get the broker up and running. Here is the error log that can be available. 2014-12-02 09:00:37,815 INFO org.I0Itec.zkclient.ZkClient: zookeeper state changed (SyncConnected) 2014-12-02 09:00:37,853 INFO kafka.log.LogManager: Loading log '.gnome2' 2014-12-02 09:00:37,855 FATAL kafka.server.KafkaServerStartable: Fatal error during KafkaServerStable startup. Prepare to shutdown java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1911) at kafka.log.Log$.parseTopicPartitionName(Log.scala:740) at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogMa at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogMa at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimize at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:105) at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113) at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimize at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34) at kafka.log.LogManager.loadLogs(LogManager.scala:105) at kafka.log.LogManager.<init>(LogManager.scala:57) at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:293) at kafka.server.KafkaServer.startup(KafkaServer.scala:73) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala: at kafka.Kafka$.main(Kafka.scala:46) at kafka.Kafka.main(Kafka.scala) 2014-12-02 09:00:37,858 INFO kafka.server.KafkaServer: [Kafka Server 159], shutt I did exact same thing on a test cluster and it worked very well. Both production and test clusters are running same CM5.2 and CDH5.2.0 versions. What could be potential issue and how to fix this issue? Please help!!! Thanks, Silaphet
... View more