Hi, reading that link: http://www.cloudera.com/content/cloudera/en/documentation/core/v5-3-x/topics/cm_metrics_hbase.html   flush_avg_time_across_regionservers flush_region_avg_time_across_regionservers   these metrics are not CDH5 anymore. Is there any possibility to get timings for these properties? Or they just gone away from HBase?     ... View more

Hi, both these classes do some stuff inside to create HConnection/Connection. I see that it's possible to provide executor service while creating HConnection/Connection.   1. Do I have to do that 2. looks like I have problems with performance because of shared connection. The more threads I use, the bigger response time I get. HBase is cold, it doesn't suffer because of load.     Here are metrics for my dummy perfomance test: 10 threads looks perfect in terms of respose: 99% <= 28.81 milliseconds -- Meters ---------------------------------------------------------------------- .putMeter count = 414914 mean rate = 885.58 events/second 1-minute rate = 911.56 events/second 5-minute rate = 778.16 events/second 15-minute rate = 549.72 events/second -- Timers ---------------------------------------------------------------------- .putTimer count = 414914 mean rate = 884.66 calls/second 1-minute rate = 911.53 calls/second 5-minute rate = 765.60 calls/second 15-minute rate = 515.06 calls/second min = 4.87 milliseconds max = 211.77 milliseconds mean = 10.81 milliseconds stddev = 5.43 milliseconds median = 10.34 milliseconds 75% <= 11.59 milliseconds 95% <= 14.41 milliseconds 98% <= 19.59 milliseconds 99% <= 28.81 milliseconds 99.9% <= 60.67 milliseconds   50 threads give better throughput, but response is awful 99% <= 202.50 milliseconds -- Meters ---------------------------------------------------------------------- .putMeter count = 1348130 mean rate = 2217.72 events/second 1-minute rate = 2114.95 events/second 5-minute rate = 2194.61 events/second 15-minute rate = 1942.24 events/second -- Timers ---------------------------------------------------------------------- .putTimer count = 1348130 mean rate = 2216.04 calls/second 1-minute rate = 2120.26 calls/second 5-minute rate = 2169.94 calls/second 15-minute rate = 1844.06 calls/second min = 8.01 milliseconds max = 348.75 milliseconds mean = 28.26 milliseconds stddev = 31.91 milliseconds median = 18.58 milliseconds 75% <= 27.48 milliseconds 95% <= 76.83 milliseconds 98% <= 100.05 milliseconds 99% <= 202.50 milliseconds 99.9% <= 348.75 milliseconds 100 threads give +10% throughput and the same response time.     What is the rgiht way to fix it? ... View more

Hi, bumped my testing stuff to CDH 5.4.4 and got failure while running tests. Here is a log   2015-07-19 23:40:18,607 INFO  [SERGEYs-MBP:51977.activeMasterManager] master.TableNamespaceManager (TableNamespaceManager.java:start(85)) - Namespace table not found. Creating... 2015-07-19 23:40:18,620 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(645)) - Got user-level KeeperException when processing sessionid:0x14ea84290170000 type:create cxid:0x1ac zxid:0x35 txntype:-1 reqpath:n/a Error Path:/hbase/table-lock/hbase:namespace Error:KeeperErrorCode = NoNode for /hbase/table-lock/hbase:namespace 2015-07-19 23:40:18,630 INFO  [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] handler.CreateTableHandler (CreateTableHandler.java:process(189)) - Create table hbase:namespace 2015-07-19 23:40:18,644 INFO  [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion (HRegion.java:createHRegion(5598)) - creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = file:/Users/ssa/devel/myown/hadoop/mini-hdfs-cluster-maven-plugin/target/hbase-root/.tmp Table name == hbase:namespace 2015-07-19 23:40:18,655 INFO  [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion (HRegion.java:doClose(1425)) - Closed hbase:namespace,,1437342018607.9c2e23572f970747f86ec499b89c281b. 2015-07-19 23:40:18,702 INFO  [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] hbase.MetaTableAccessor (MetaTableAccessor.java:addRegionsToMeta(1169)) - Added 2 2015-07-19 23:40:18,704 WARN  [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] zookeeper.ZKTableStateManager (ZKTableStateManager.java:setTableState(100)) - Moving table hbase:namespace state from ENABLING to ENABLED 2015-07-19 23:40:18,706 INFO  [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] handler.CreateTableHandler (CreateTableHandler.java:completed(219)) - Table, hbase:namespace, creation successful Process Thread Dump: Thread dump because: Master not initialized after 200000ms seconds 186 active threads Thread 255 (MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0):   State: WAITING   Blocked count: 11   Waited count: 11   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20b5f2ac   Stack:     sun.misc.Unsafe.park(Native Method)     java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)     java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)     java.lang.Thread.run(Thread.java:745) Thread 249 (CatalogJanitor-SERGEYs-MBP:51977):   ====   2015-07-19 23:43:33,188 ERROR [main] hbase.MiniHBaseCluster (MiniHBaseCluster.java:init(229)) - Error starting cluster java.lang.RuntimeException: Master not initialized after 200000ms seconds at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:225) at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:436) at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:224) at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:93) at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:80)   at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:67)   fails here: MINI_HBASE_CLUSTER = new MiniHBaseCluster(configuration, 1);   What it could be? Have no idea. ... View more

Hi, did HBase service restart and on of RS on node didn't start. Here are logs from agent:     [24/May/2015 16:12:34 +0000] 7684 MainThread util INFO Extracted 13 files and 0 dirs to /run/cloudera-scm-agent/process/2688-hbase-REGIONSERVER. [24/May/2015 16:12:34 +0000] 7684 MainThread agent INFO Created /run/cloudera-scm-agent/process/2688-hbase-REGIONSERVER/logs [24/May/2015 16:12:34 +0000] 7684 MainThread agent INFO Chowning /run/cloudera-scm-agent/process/2688-hbase-REGIONSERVER/logs to hbase (995) hbase (995) [24/May/2015 16:12:34 +0000] 7684 MainThread agent INFO Chmod'ing /run/cloudera-scm-agent/process/2688-hbase-REGIONSERVER/logs to 0751 [24/May/2015 16:12:34 +0000] 7684 MainThread cgroups INFO Creating cgroup /run/cloudera-scm-agent/cgroups/blkio/2688-hbase-REGIONSERVER [24/May/2015 16:12:34 +0000] 7684 MainThread cgroups INFO Creating cgroup /run/cloudera-scm-agent/cgroups/cpuacct/2688-hbase-REGIONSERVER     Catalog  /run/cloudera-scm-agent/process/2688-hbase-REGIONSERVER is empty for unknown reason   Here are logs from cloudera-manager-server: 2015-05-24 17:12:34,006 INFO 2122353975@scm-web-12491:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing role command Restart BasicCmdArgs{args=[]}. Service: DbService{id=26, name=hbase} Role: DbRole{id=286, name=hbase-REGIONSERVER-f125dcf94ca5890bacabcee567cf9072, hostName=host02.domain.ru} 2015-05-24 17:12:34,073 INFO 2122353975@scm-web-12491:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing role command Start BasicCmdArgs{args=[]}. Service: DbService{id=26, name=hbase} Role: DbRole{id=286, name=hbase-REGIONSERVER-f125dcf94ca5890bacabcee567cf9072, hostName=host02.domain.ru} 2015-05-24 17:12:49,670 INFO 336630595@agentServer-15907:com.cloudera.cmf.command.components.StalenessChecker: No staleness check scheduled, scheduling one in 30 seconds 2015-05-24 17:13:19,671 INFO ScheduledStalenessChecker:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ProcessStalenessCheckCommand BasicCmdArgs{args=[First reason why: Process (id=2688) has a brand new heartbeat]}. 2015-05-24 17:13:26,107 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check done. Duration: PT6.339S 2015-05-24 17:15:06,592 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: Aborting BringUp command (5241) on service DbService{id=26, name=hbase} role DbRole{id=286, name=hbase-REGIONSERVER-f125dcf94ca5890bacabcee567cf9072, hostName=host02.domain.ru}. 2015-05-24 17:15:19,769 WARN 336630595@agentServer-15907:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=2688 name=null host=5fcc9bbd-9c79-4db4-809d-5590b7427799/host02.domain.ru 2015-05-24 17:16:57,792 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands 2015-05-24 17:16:57,794 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2013-05-24T13:16:57.792Z to reap. 2015-05-24 17:16:57,795 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeScannerService: Reaped 0 requests. 2015-05-24 17:16:57,795 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeConfiguratorService: Reaped 0 requests. There are more running roles on that node and all their process folders are empty. What is could be? ... View more

job init failed : org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist:  job.splitmetainfo   Hi, I'm trying to run mapreduce job using console hadoop jar. I get exception: 2015-05-06 17:21:30,305 WARN [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job init failed org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://nameservice1/user/devops/.staging/job_1430916274046_0127/job.splitmetainfo at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1566) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1430) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1388) at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1272) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1045) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407) Caused by: java.io.FileNotFoundException: File does not exist: hdfs://nameservice1/user/devops/.staging/job_1430916274046_0127/job.splitmetainfo at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1093) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085) at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1561)   hdfs://nameservice1/user/devops/.staging/job_1430916274046_0127 contains files: Found 3 items -rw-r--r--   3 devops devops         311 2015-05-06 17:21 /user/devops/.staging/job_1430916274046_0127/job_1430916274046_0127.summary -rw-r--r--   3 devops devops       11529 2015-05-06 17:21 /user/devops/.staging/job_1430916274046_0127/job_1430916274046_0127_1.jhist -rw-r--r--   3 devops devops     224982 2015-05-06 17:21 /user/devops/.staging/job_1430916274046_0127/job_1430916274046_0127_1_conf.xml   What's wrong, what i'm missing? ... View more

Hi, I've installed spark cdh 5.3 from packages and worker on one of the nodes not starting. It says:   Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/impl/StaticLoggerBinder at org.apache.spark.Logging$class.initializeLogging(Logging.scala:116) at org.apache.spark.Logging$class.initializeIfNecessary(Logging.scala:107) at org.apache.spark.Logging$class.log(Logging.scala:51) at org.apache.spark.deploy.worker.Worker$.log(Worker.scala:470) at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:472) at org.apache.spark.deploy.worker.Worker.main(Worker.scala) Caused by: java.lang.ClassNotFoundException: org.slf4j.impl.StaticLoggerBinder at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 6 more   What is missing? Why package-based installation could be broken? Where does spark keep its jars? ... View more

Just installed spark  1.2.0+cdh5.3.3 And can't start spark-history-server:   15/04/22 14:13:57 INFO history.HistoryServer: Registered signal handlers for [TERM, HUP, INT] 15/04/22 14:13:57 INFO spark.SecurityManager: Changing view acls to: spark 15/04/22 14:13:57 INFO spark.SecurityManager: Changing modify acls to: spark 15/04/22 14:13:57 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark) 15/04/22 14:13:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:184) at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala) Caused by: java.io.IOException: Incomplete HDFS URI, no host: hdfs:///user/spark/applicationHistory at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:141)   How can I specify local path for history-server? I don't have any HDFS. I run spark against Datastax DSE? ... View more