Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CDH 5.4.4 and MiniHBaseCluster

CDH 5.4.4 and MiniHBaseCluster

Expert Contributor

Hi, bumped my testing stuff to CDH 5.4.4 and got failure while running tests. Here is a log

 

2015-07-19 23:40:18,607 INFO  [SERGEYs-MBP:51977.activeMasterManager] master.TableNamespaceManager (TableNamespaceManager.java:start(85)) - Namespace table not found. Creating...

2015-07-19 23:40:18,620 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(645)) - Got user-level KeeperException when processing sessionid:0x14ea84290170000 type:create cxid:0x1ac zxid:0x35 txntype:-1 reqpath:n/a Error Path:/hbase/table-lock/hbase:namespace Error:KeeperErrorCode = NoNode for /hbase/table-lock/hbase:namespace

2015-07-19 23:40:18,630 INFO  [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] handler.CreateTableHandler (CreateTableHandler.java:process(189)) - Create table hbase:namespace

2015-07-19 23:40:18,644 INFO  [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion (HRegion.java:createHRegion(5598)) - creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = file:/Users/ssa/devel/myown/hadoop/mini-hdfs-cluster-maven-plugin/target/hbase-root/.tmp Table name == hbase:namespace

2015-07-19 23:40:18,655 INFO  [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion (HRegion.java:doClose(1425)) - Closed hbase:namespace,,1437342018607.9c2e23572f970747f86ec499b89c281b.

2015-07-19 23:40:18,702 INFO  [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] hbase.MetaTableAccessor (MetaTableAccessor.java:addRegionsToMeta(1169)) - Added 2

2015-07-19 23:40:18,704 WARN  [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] zookeeper.ZKTableStateManager (ZKTableStateManager.java:setTableState(100)) - Moving table hbase:namespace state from ENABLING to ENABLED

2015-07-19 23:40:18,706 INFO  [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] handler.CreateTableHandler (CreateTableHandler.java:completed(219)) - Table, hbase:namespace, creation successful

Process Thread Dump: Thread dump because: Master not initialized after 200000ms seconds

186 active threads

Thread 255 (MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0):

  State: WAITING

  Blocked count: 11

  Waited count: 11

  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20b5f2ac

  Stack:

    sun.misc.Unsafe.park(Native Method)

    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)

    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)

    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

    java.lang.Thread.run(Thread.java:745)

Thread 249 (CatalogJanitor-SERGEYs-MBP:51977):

 

====

 

2015-07-19 23:43:33,188 ERROR [main] hbase.MiniHBaseCluster (MiniHBaseCluster.java:init(229)) - Error starting cluster

java.lang.RuntimeException: Master not initialized after 200000ms seconds

at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:225)

at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:436)

at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:224)

at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:93)

at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:80)

 

at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:67)

 

fails here: MINI_HBASE_CLUSTER = new MiniHBaseCluster(configuration, 1);

 

What it could be? Have no idea.

5 REPLIES 5

Re: CDH 5.4.4 and MiniHBaseCluster

Expert Contributor
-Djava.net.preferIPv4Stack=true helped
Probably some weird MacOS update...?

Re: CDH 5.4.4 and MiniHBaseCluster

Expert Contributor

Could you provide more detail about what you're trying to do with the minicluster? Is there a pom or simplified test project I could look at?

 

For example, the hbase downstreamer project sets up a minicluster and writes/reads some entries. If I build it for CDH5.4.4

 

$ mvn clean package -Dhbase.version=1.0.0-cdh5.4.4 -Dhadoop.version=2.6.0-cdh5.4.4

 

Then it appears to work normally.

Re: CDH 5.4.4 and MiniHBaseCluster

Expert Contributor

Hi, thanks for your reply.

Absolutely the same situation

 

5/07/22 00:12:44 INFO hbase.MetaTableAccessor: Added 2

15/07/22 00:12:44 WARN zookeeper.ZKTableStateManager: Moving table hbase:namespace state from ENABLING to ENABLED

15/07/22 00:12:44 INFO handler.CreateTableHandler: Table, hbase:namespace, creation successful

15/07/22 00:13:10 INFO blockmanagement.CacheReplicationMonitor: Rescanning after 30004 milliseconds

15/07/22 00:13:10 INFO blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s).

15/07/22 00:13:40 INFO blockmanagement.CacheReplicationMonitor: Rescanning after 30003 milliseconds

15/07/22 00:13:40 INFO blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 2 millisecond(s).

15/07/22 00:14:10 INFO blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds

15/07/22 00:14:10 INFO blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s).

15/07/22 00:14:40 INFO blockmanagement.CacheReplicationMonitor: Rescanning after 30005 milliseconds

15/07/22 00:14:40 INFO blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s).

15/07/22 00:15:10 INFO blockmanagement.CacheReplicationMonitor: Rescanning after 30004 milliseconds

 

15/07/22 00:16:02 ERROR hbase.MiniHBaseCluster: Error starting cluster

java.lang.RuntimeException: Master not initialized after 200000ms seconds

at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:225)

at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:436)

at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:224)

at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:93)

at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:985)

at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:945)

at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:819)

at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:813)

at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:757)

at org.hbase.downstreamer.TestHBaseMiniCluster.testSpinUpMiniHBaseCluster(TestHBaseMiniCluster.java:16)

 

the same line:

15/07/22 00:12:44 INFO handler.CreateTableHandler: Table, hbase:namespace, creation successful

 

and build hungs. Here I just get output from replication monitor.

If I add property -Djava.net.preferIPv4Stack=true

 

mvn clean package -Dhbase.version=1.0.0-cdh5.4.4 -Dhadoop.version=2.6.0-cdh5.4.4 -Djava.net.preferIPv4Stack=true

it works! 

I'm running it on MacOS, I suppose problem in some weird MacOS neworking stuff.

Re: CDH 5.4.4 and MiniHBaseCluster

Expert Contributor

That looks like you don't have the CDH repos set up for maven.

 

ref: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_vd_cdh5_maven_repo....

 

Here's what the entry in my ~/.m2/settings.xml looks like:

 

<settings>
  <profiles>
    <profile>
      <id>cdh</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <repositories>
        <repository>
          <id>apache.releases</id>
          <url>https://repository.apache.org/content/groups/public/</url>
          <name>Apache Releases Repository</name>
          <snapshots>
            <enabled>false</enabled>
          </snapshots>
      </repository>
      <repository>
        <id>cdh.releases.repo</id>
        <url>https://repository.cloudera.com/content/groups/cdh-releases-rcs</url>
        <name>CDH Releases Repository</name>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </repository>
      <repository>
        <id>datanucleus</id>
        <url>http://www.datanucleus.org/downloads/maven2</url>
        <name>Datanucleus</name>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </repository>
      <repository>
        <id>java.net repository</id>
        <url>http://download.java.net/maven/2</url>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </repository>
      <repository>
        <id>cdh.thirdparty.repo</id>
        <url>https://repository.cloudera.com/content/repositories/third-party</url>
        <name>Cloudera Third Party Repository</name>
        <snapshots><enabled>false</enabled></snapshots>
      </repository>
    </repositories>
    </profile>
  </profiles>
</settings>
Highlighted

Re: CDH 5.4.4 and MiniHBaseCluster

Expert Contributor

Hi, I've updated the post wih problem.