Reply
Expert Contributor
Posts: 162
Registered: ‎07-29-2013

CDH 5.4.4 and MiniHBaseCluster

Hi, bumped my testing stuff to CDH 5.4.4 and got failure while running tests. Here is a log

 

2015-07-19 23:40:18,607 INFO  [SERGEYs-MBP:51977.activeMasterManager] master.TableNamespaceManager (TableNamespaceManager.java:start(85)) - Namespace table not found. Creating...

2015-07-19 23:40:18,620 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(645)) - Got user-level KeeperException when processing sessionid:0x14ea84290170000 type:create cxid:0x1ac zxid:0x35 txntype:-1 reqpath:n/a Error Path:/hbase/table-lock/hbase:namespace Error:KeeperErrorCode = NoNode for /hbase/table-lock/hbase:namespace

2015-07-19 23:40:18,630 INFO  [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] handler.CreateTableHandler (CreateTableHandler.java:process(189)) - Create table hbase:namespace

2015-07-19 23:40:18,644 INFO  [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion (HRegion.java:createHRegion(5598)) - creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = file:/Users/ssa/devel/myown/hadoop/mini-hdfs-cluster-maven-plugin/target/hbase-root/.tmp Table name == hbase:namespace

2015-07-19 23:40:18,655 INFO  [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion (HRegion.java:doClose(1425)) - Closed hbase:namespace,,1437342018607.9c2e23572f970747f86ec499b89c281b.

2015-07-19 23:40:18,702 INFO  [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] hbase.MetaTableAccessor (MetaTableAccessor.java:addRegionsToMeta(1169)) - Added 2

2015-07-19 23:40:18,704 WARN  [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] zookeeper.ZKTableStateManager (ZKTableStateManager.java:setTableState(100)) - Moving table hbase:namespace state from ENABLING to ENABLED

2015-07-19 23:40:18,706 INFO  [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] handler.CreateTableHandler (CreateTableHandler.java:completed(219)) - Table, hbase:namespace, creation successful

Process Thread Dump: Thread dump because: Master not initialized after 200000ms seconds

186 active threads

Thread 255 (MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0):

  State: WAITING

  Blocked count: 11

  Waited count: 11

  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20b5f2ac

  Stack:

    sun.misc.Unsafe.park(Native Method)

    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)

    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)

    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

    java.lang.Thread.run(Thread.java:745)

Thread 249 (CatalogJanitor-SERGEYs-MBP:51977):

 

====

 

2015-07-19 23:43:33,188 ERROR [main] hbase.MiniHBaseCluster (MiniHBaseCluster.java:init(229)) - Error starting cluster

java.lang.RuntimeException: Master not initialized after 200000ms seconds

at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:225)

at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:436)

at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:224)

at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:93)

at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:80)

 

at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:67)

 

fails here: MINI_HBASE_CLUSTER = new MiniHBaseCluster(configuration, 1);

 

What it could be? Have no idea.

Expert Contributor
Posts: 162
Registered: ‎07-29-2013

Re: CDH 5.4.4 and MiniHBaseCluster

-Djava.net.preferIPv4Stack=true helped
Probably some weird MacOS update...?
Cloudera Employee
Posts: 88
Registered: ‎01-08-2014

Re: CDH 5.4.4 and MiniHBaseCluster

Could you provide more detail about what you're trying to do with the minicluster? Is there a pom or simplified test project I could look at?

 

For example, the hbase downstreamer project sets up a minicluster and writes/reads some entries. If I build it for CDH5.4.4

 

$ mvn clean package -Dhbase.version=1.0.0-cdh5.4.4 -Dhadoop.version=2.6.0-cdh5.4.4

 

Then it appears to work normally.

Expert Contributor
Posts: 162
Registered: ‎07-29-2013

Re: CDH 5.4.4 and MiniHBaseCluster

[ Edited ]

Hi, thanks for your reply.

Absolutely the same situation

 

5/07/22 00:12:44 INFO hbase.MetaTableAccessor: Added 2

15/07/22 00:12:44 WARN zookeeper.ZKTableStateManager: Moving table hbase:namespace state from ENABLING to ENABLED

15/07/22 00:12:44 INFO handler.CreateTableHandler: Table, hbase:namespace, creation successful

15/07/22 00:13:10 INFO blockmanagement.CacheReplicationMonitor: Rescanning after 30004 milliseconds

15/07/22 00:13:10 INFO blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s).

15/07/22 00:13:40 INFO blockmanagement.CacheReplicationMonitor: Rescanning after 30003 milliseconds

15/07/22 00:13:40 INFO blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 2 millisecond(s).

15/07/22 00:14:10 INFO blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds

15/07/22 00:14:10 INFO blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s).

15/07/22 00:14:40 INFO blockmanagement.CacheReplicationMonitor: Rescanning after 30005 milliseconds

15/07/22 00:14:40 INFO blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s).

15/07/22 00:15:10 INFO blockmanagement.CacheReplicationMonitor: Rescanning after 30004 milliseconds

 

15/07/22 00:16:02 ERROR hbase.MiniHBaseCluster: Error starting cluster

java.lang.RuntimeException: Master not initialized after 200000ms seconds

at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:225)

at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:436)

at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:224)

at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:93)

at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:985)

at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:945)

at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:819)

at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:813)

at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:757)

at org.hbase.downstreamer.TestHBaseMiniCluster.testSpinUpMiniHBaseCluster(TestHBaseMiniCluster.java:16)

 

the same line:

15/07/22 00:12:44 INFO handler.CreateTableHandler: Table, hbase:namespace, creation successful

 

and build hungs. Here I just get output from replication monitor.

If I add property -Djava.net.preferIPv4Stack=true

 

mvn clean package -Dhbase.version=1.0.0-cdh5.4.4 -Dhadoop.version=2.6.0-cdh5.4.4 -Djava.net.preferIPv4Stack=true

it works! 

I'm running it on MacOS, I suppose problem in some weird MacOS neworking stuff.

Cloudera Employee
Posts: 88
Registered: ‎01-08-2014

Re: CDH 5.4.4 and MiniHBaseCluster

That looks like you don't have the CDH repos set up for maven.

 

ref: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_vd_cdh5_maven_repo....

 

Here's what the entry in my ~/.m2/settings.xml looks like:

 

<settings>
  <profiles>
    <profile>
      <id>cdh</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <repositories>
        <repository>
          <id>apache.releases</id>
          <url>https://repository.apache.org/content/groups/public/</url>
          <name>Apache Releases Repository</name>
          <snapshots>
            <enabled>false</enabled>
          </snapshots>
      </repository>
      <repository>
        <id>cdh.releases.repo</id>
        <url>https://repository.cloudera.com/content/groups/cdh-releases-rcs</url>
        <name>CDH Releases Repository</name>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </repository>
      <repository>
        <id>datanucleus</id>
        <url>http://www.datanucleus.org/downloads/maven2</url>
        <name>Datanucleus</name>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </repository>
      <repository>
        <id>java.net repository</id>
        <url>http://download.java.net/maven/2</url>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </repository>
      <repository>
        <id>cdh.thirdparty.repo</id>
        <url>https://repository.cloudera.com/content/repositories/third-party</url>
        <name>Cloudera Third Party Repository</name>
        <snapshots><enabled>false</enabled></snapshots>
      </repository>
    </repositories>
    </profile>
  </profiles>
</settings>
Expert Contributor
Posts: 162
Registered: ‎07-29-2013

Re: CDH 5.4.4 and MiniHBaseCluster

Hi, I've updated the post wih problem.

Announcements