My HiveServer2 daemon thread "Get-Input-Paths-" count has been increased Continuously. About a week later the HiveServer2 cpu reached 100%(single core) and all SQL could no be executed.     https://issues.apache.org/jira/browse/HIVE-16949   Environment: CentOS Linux release 7.3.1611 (Core)  3.10.0-514.el7.x86_64 java version "1.8.0_121" CDH & CM version: 5.11.1     thread dump file: how to upload.........   this report is after HiveServer2 restart, but the thread still increase continously    http://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTcvMDcvMTUvLS1oaXZlc2VydmVyLnppcC0tMy0yNi0xOA==   snipt of the thread dump file: "Get-Input-Paths-1" #37702 daemon prio=5 os_prio=0 tid=0x0000000001021000 nid=0x2ca15 waiting on condition [0x00007fb0046f0000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006de406738> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "Get-Input-Paths-0" #37701 daemon prio=5 os_prio=0 tid=0x0000000001bb8800 nid=0x2ca14 waiting on condition [0x00007fafe99d8000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006de406738> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None     oh~ I found the issue on JIRA https://issues.apache.org/jira/browse/HIVE-16949    ... View more

Does Spark 2 supported CDH 5.11.0 ? I can't find the Spark 2 service in "Add Service" page And the Spark2 parcels "2.1.0.cloudera1-1.cdh5.7.0.p0.120904" has been already Distributed and Activated ... View more

I have a cluster with 5 servers (hostname hadoop-215 to hadoop-219);   Today, I changed Hue configuration, but it can't startup. Then I tried out other services(Impala/HBase thrift server),  both of them can't be started, and have no logs in /var/logs/{service_name}.           $ tail /var/log/cloudera-scm-server/cloudera-scm-server.log 2015-10-19 13:46:39,857 INFO StaleEntityEviction:com.cloudera.cmf.model.HeartbeatStore: Reaped 1 process heartbeats 2015-10-19 13:46:39,861 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands 2015-10-19 13:46:39,862 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2013-10-19T05:46:39.861Z to reap. 2015-10-19 13:46:39,863 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeScannerService: Reaped 0 requests. 2015-10-19 13:46:39,863 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeConfiguratorService: Reaped 0 requests. 2015-10-19 13:46:48,220 INFO 1810859969@scm-web-125735:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing service command Start SvcCmdArgs{targetRoles=[DbRole{id=107, name=hue-HUE_SERVER-b672e8f973cd8ddace77def263871fb0, hostName=hadoop-215}], args=[]}. Service: DbService{id=23, name=hue} 2015-10-19 13:46:48,243 INFO 1810859969@scm-web-125735:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing role command Start BasicCmdArgs{args=[]}. Service: DbService{id=23, name=hue} Role: DbRole{id=107, name=hue-HUE_SERVER-b672e8f973cd8ddace77def263871fb0, hostName=hadoop-215} 2015-10-19 13:46:48,307 INFO 1810859969@scm-web-125735:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: Added BringUp command to service DbService{id=23, name=hue}. 2015-10-19 13:46:48,340 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:46:53,345 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:46:58,350 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:47:03,354 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:47:03,434 INFO 1338356818@agentServer-14005:com.cloudera.cmf.command.components.StalenessChecker: No staleness check scheduled, scheduling one in 30 seconds 2015-10-19 13:47:03,456 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:47:08,460 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:47:13,465 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:47:18,471 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:47:23,476 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:47:28,481 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:47:33,434 INFO ScheduledStalenessChecker:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ProcessStalenessCheckCommand BasicCmdArgs{args=[First reason why: Process (id=940) has a brand new heartbeat]}. 2015-10-19 13:47:33,440 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:47:33,949 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.AbstractRoleHandler: (26 skipped) Client configs of DbService{id=22, name=zookeeper} wanted but not available. 2015-10-19 13:47:33,977 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check done. Duration: PT0.535S 2015-10-19 13:47:33,982 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:47:38,989 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:47:43,993 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:47:48,998 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:47:54,003 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:47:59,008 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:48:04,012 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:48:09,017 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:48:14,022 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:48:19,027 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:48:24,032 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:48:29,038 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:48:34,042 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:48:39,054 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:48:44,059 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:48:49,063 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:48:54,068 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:48:59,072 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:49:04,077 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:49:09,081 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:49:14,085 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}. 2015-10-19 13:49:19,090 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: Aborting BringUp command (3097) on service DbService{id=23, name=hue} role DbRole{id=107, name=hue-HUE_SERVER-b672e8f973cd8ddace77def263871fb0, hostName=hadoop-215}. 2015-10-19 13:49:19,098 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) has finished on service DbService{id=23, name=hue}. 2015-10-19 13:49:33,589 WARN 770526468@agentServer-14007:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=940 name=null host=1db4c618-eee6-49cc-9d8f-475b1fe7fd15/hadoop-215 2015-10-19 13:56:39,865 INFO StaleEntityEviction:com.cloudera.cmf.model.HeartbeatStore: Reaped 1 process heartbeats 2015-10-19 13:56:39,872 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands 2015-10-19 13:56:39,874 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2013-10-19T05:56:39.872Z to reap. 2015-10-19 13:56:39,875 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeScannerService: Reaped 0 requests. 2015-10-19 13:56:39,875 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeConfiguratorService: Reaped 0 requests. 2015-10-19 14:00:00,003 INFO com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command GlobalPoolsRefresh BasicCmdArgs{scheduleId=1, scheduledTime=2015-10-19T06:00:00.000Z}. 2015-10-19 14:00:00,140 INFO com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.scheduler.CommandDispatcherJob: Skipping scheduled command 'GlobalPoolsRefresh' since it is a noop. $ tail /var/log/cloudera-scm-agent/cloudera-scm-agent.log [19/Oct/2015 13:46:48 +0000] 16355 CP Server Thread-6 _cplogging INFO 192.168.4.215 - - [19/Oct/2015:13:46:48] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0" [19/Oct/2015 13:46:48 +0000] 16355 MainThread util INFO Using generic audit plugin for process hue-HUE_SERVER [19/Oct/2015 13:46:48 +0000] 16355 MainThread util INFO Creating metadata plugin for process hue-HUE_SERVER [19/Oct/2015 13:46:48 +0000] 16355 MainThread util INFO Using specific metadata plugin for process hue-HUE_SERVER [19/Oct/2015 13:46:48 +0000] 16355 MainThread util INFO Using generic metadata plugin for process hue-HUE_SERVER [19/Oct/2015 13:46:48 +0000] 16355 MainThread __init__ INFO Instantiating generic monitor for service HUE and role HUE_SERVER [19/Oct/2015 13:46:48 +0000] 16355 MainThread agent INFO Activating Process 940-hue-HUE_SERVER [19/Oct/2015 13:46:48 +0000] 16355 MainThread agent INFO Created /var/run/cloudera-scm-agent/process/940-hue-HUE_SERVER [19/Oct/2015 13:46:48 +0000] 16355 MainThread agent INFO Chowning /var/run/cloudera-scm-agent/process/940-hue-HUE_SERVER to hue (480) hue (479) [19/Oct/2015 13:46:48 +0000] 16355 MainThread agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/940-hue-HUE_SERVER to 0751 [19/Oct/2015 13:46:48 +0000] 16355 MainThread parcel INFO prepare_environment begin: {u'GPLEXTRAS': u'5.4.3-1.cdh5.4.3.p0.5', u'CDH': u'5.4.3-1.cdh5.4.3.p0.6', u'KAFKA': u'0.8.2.0-1.kafka1.3.0.p0.29'}, [u'cdh'], [u'cdh-plugin', u'hue-plugin'] [19/Oct/2015 13:46:48 +0000] 16355 MainThread parcel INFO The following requested parcels are not available: {} [19/Oct/2015 13:46:48 +0000] 16355 MainThread parcel INFO Obtained tags ['kafka'] for parcel KAFKA [19/Oct/2015 13:46:48 +0000] 16355 MainThread parcel INFO Obtained tags ['cdh', 'impala', 'sentry', 'solr', 'spark'] for parcel CDH [19/Oct/2015 13:46:48 +0000] 16355 MainThread parcel INFO Obtained tags ['cdh-plugin', 'impala-plugin', 'solr-plugin', 'spark-plugin', 'hadoop_lzo'] for parcel GPLEXTRAS [19/Oct/2015 13:46:48 +0000] 16355 MainThread parcel INFO prepare_environment end: {'CDH': '5.4.3-1.cdh5.4.3.p0.6', 'GPLEXTRAS': '5.4.3-1.cdh5.4.3.p0.5'} [19/Oct/2015 13:46:48 +0000] 16355 MainThread util INFO Extracted 35 files and 0 dirs to /var/run/cloudera-scm-agent/process/940-hue-HUE_SERVER. [19/Oct/2015 13:46:48 +0000] 16355 MainThread agent INFO Created /var/run/cloudera-scm-agent/process/940-hue-HUE_SERVER/logs [19/Oct/2015 13:46:48 +0000] 16355 MainThread agent INFO Chowning /var/run/cloudera-scm-agent/process/940-hue-HUE_SERVER/logs to hue (480) hue (479) [19/Oct/2015 13:46:48 +0000] 16355 MainThread agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/940-hue-HUE_SERVER/logs to 0751 [19/Oct/2015 13:46:48 +0000] 16355 MainThread cgroups INFO Creating cgroup /var/run/cloudera-scm-agent/cgroups/blkio/940-hue-HUE_SERVER [19/Oct/2015 13:46:48 +0000] 16355 MainThread cgroups INFO Creating cgroup /var/run/cloudera-scm-agent/cgroups/cpuacct/940-hue-HUE_SERVER [19/Oct/2015 13:46:48 +0000] 16355 MainThread cgroups INFO Creating cgroup /var/run/cloudera-scm-agent/cgroups/cpu/940-hue-HUE_SERVER [19/Oct/2015 13:46:48 +0000] 16355 MainThread cgroups INFO Reconfiguring cgroup pseudofile /var/run/cloudera-scm-agent/cgroups/cpu/940-hue-HUE_SERVER/cpu.rt_runtime_us with value 1000 [19/Oct/2015 13:46:48 +0000] 16355 MainThread agent INFO reading limits: {u'limit_memlock': None, u'limit_fds': None} [19/Oct/2015 13:46:48 +0000] 16355 MainThread agent INFO Triggering supervisord update. [19/Oct/2015 13:46:48 +0000] 16355 MainThread navigator_plugin INFO Scheduling a refresh for Audit Plugin for hue-HUE_SERVER with pipelines [] [19/Oct/2015 13:46:48 +0000] 16355 MainThread navigator_plugin INFO Scheduling a refresh for Metadata Plugin for hue-HUE_SERVER with pipelines [] [19/Oct/2015 13:46:48 +0000] 16355 MainThread abstract_monitor INFO Refreshing GenericMonitor HUE-HUE_SERVER for None [19/Oct/2015 13:46:48 +0000] 16355 MainThread __init__ INFO New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x416f550>,) [19/Oct/2015 13:46:51 +0000] 16355 Audit-Plugin navigator_plugin INFO Refreshing Audit Plugin for hue-HUE_SERVER with pipelines [] [19/Oct/2015 13:46:51 +0000] 16355 Audit-Plugin navigator_plugin_pipeline INFO Stopping Navigator Plugin Pipeline '' for hue-HUE_SERVER (log dir: None) [19/Oct/2015 13:46:51 +0000] 16355 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for hue-HUE_SERVER with pipelines [] [19/Oct/2015 13:46:51 +0000] 16355 Metadata-Plugin navigator_plugin_pipeline INFO Stopping Navigator Plugin Pipeline '' for hue-HUE_SERVER (log dir: None) [19/Oct/2015 13:47:48 +0000] 16355 MonitorDaemon-Scheduler __init__ INFO Monitor expired: ('GenericMonitor HUE-HUE_SERVER for hue-HUE_SERVER-b672e8f973cd8ddace77def263871fb0',) [19/Oct/2015 13:49:33 +0000] 16355 MainThread agent INFO Deactivating process 940-hue-HUE_SERVER [19/Oct/2015 13:49:33 +0000] 16355 MainThread agent INFO Supervisord doesn't know about process 940-hue-HUE_SERVER. It cannot be deactivated by this Agent, assuming it even exists [19/Oct/2015 13:49:33 +0000] 16355 MainThread agent INFO Deleting process 940-hue-HUE_SERVER [19/Oct/2015 13:49:33 +0000] 16355 MainThread cgroups INFO Destroying cgroup /var/run/cloudera-scm-agent/cgroups/blkio/940-hue-HUE_SERVER [19/Oct/2015 13:49:33 +0000] 16355 MainThread cgroups INFO Destroying cgroup /var/run/cloudera-scm-agent/cgroups/cpuacct/940-hue-HUE_SERVER [19/Oct/2015 13:49:33 +0000] 16355 MainThread cgroups INFO Destroying cgroup /var/run/cloudera-scm-agent/cgroups/cpu/940-hue-HUE_SERVER [19/Oct/2015 13:49:33 +0000] 16355 MainThread agent INFO Triggering supervisord update. [19/Oct/2015 13:49:33 +0000] 16355 MainThread agent INFO Retiring process 940-hue-HUE_SERVER [19/Oct/2015 13:49:36 +0000] 16355 Audit-Plugin navigator_plugin INFO stopping Audit Plugin for hue-HUE_SERVER with pipelines [] [19/Oct/2015 13:49:36 +0000] 16355 Audit-Plugin navigator_plugin_pipeline INFO Stopping Navigator Plugin Pipeline '' for hue-HUE_SERVER (log dir: None) [19/Oct/2015 13:49:36 +0000] 16355 Metadata-Plugin navigator_plugin INFO stopping Metadata Plugin for hue-HUE_SERVER with pipelines [] [19/Oct/2015 13:49:36 +0000] 16355 Metadata-Plugin navigator_plugin_pipeline INFO Stopping Navigator Plugin Pipeline '' for hue-HUE_SERVER (log dir: None)   Now, I moved the Hue server to another server, It's start successed. But the server hadoop-215 is my master server(have namenode, resource manager, hbase master on it), can't be stoped.     Any body can help me ?                          ... View more

I have been solved the problem. Just share the solution to everyone here.   ----------- I have two clusters (prod and dev) all have the same problem env: Centos6.6 minamal jdk1.7 CDH&CM 5.4.3   In the HUE main page It's shown  Potential misconfiguration detected. Fix and restart Hue. Hive Editor    The application won't work without a running HiveServer2.      In HiveEditor page the database list could not be shown(always loading...) I executed a simple sql and got an error "Could not start SASL: Error in sasl_client_start......."   the solution I got some helpful tips in this link: https://groups.google.com/a/cloudera.org/forum/#!topic/hue-user/TVon3U2uSis   the prombem is some cyrus system libs not installed. then i executed "yum –y install cyrus-sasl-plain cyrus-sasl-gssapi" the problem solved         ... View more