Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Solved Go to solution

Unable to connect to server error while logging from Ambari console as admin

avatar
author-rank muthukumar_siva
Rising Star

Created ‎10-06-2016 06:10 AM

Hi im getting the below error while i try to login as admin user id from ambari console. It was working fine till yesterday. Today i tried to login it was not. Restarted ambari server few times and does not help. No firewall or nothing in between. port 8080 was in listening state once i start my ambari server. All iptables / selinux are disabled only. Cleared all cache and history from the browser and tried with no luck (IE and Firefox) Kindly help if anything else can be checked to fix the issue?

" Unable to connect to Ambari Server. Confirm Ambari Server is running and you can reach Ambari Server from this machine. "

Ambari-server logambari-dev-login-error.png

06 Oct 2016 13:10:55,347  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component HIVE_SERVER on ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:10:55,359  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component MYSQL_SERVER on ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:10:55,365  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component HIVE_METASTORE on ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:10:55,371  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component WEBHCAT_SERVER on ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:10:55,383  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component NODEMANAGER on ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:11:04,786  INFO [main] AbstractConnector:338 - Started SelectChannelConnector@0.0.0.0:8080
06 Oct 2016 13:11:04,786  INFO [main] Server:272 - jetty-8.1.17.v20150415
06 Oct 2016 13:11:06,639  INFO [main] SslContextFactory:300 - Enabled Protocols [SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
06 Oct 2016 13:11:06,645  INFO [main] AbstractConnector:338 - Started SslSelectChannelConnector@0.0.0.0:8440
06 Oct 2016 13:11:06,663  INFO [main] SslContextFactory:300 - Enabled Protocols [SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
06 Oct 2016 13:11:06,667  INFO [main] AbstractConnector:338 - Started SslSelectChannelConnector@0.0.0.0:8441
06 Oct 2016 13:11:06,667  INFO [main] AmbariServer:539 - ********* Started Server **********
06 Oct 2016 13:11:06,667  INFO [main] ActionManager:77 - Starting scheduler thread
06 Oct 2016 13:11:06,667  INFO [main] ServerActionExecutor:138 - Starting Server Action Executor thread...
06 Oct 2016 13:11:06,669  INFO [main] ServerActionExecutor:165 - Server Action Executor thread started.
06 Oct 2016 13:11:06,669  INFO [main] AmbariServer:542 - ********* Started ActionManager **********
06 Oct 2016 13:11:06,669  INFO [main] ExecutionScheduleManager:201 - Starting scheduler
06 Oct 2016 13:11:06,715  INFO [main] MLog:80 - MLog clients using log4j logging.
06 Oct 2016 13:11:06,777  INFO [main] C3P0Registry:204 - Initializing c3p0-0.9.1.1 [built 15-March-2007 01:32:31; debug? true; trace: 10]
06 Oct 2016 13:11:06,801  INFO [main] StdSchedulerFactory:1184 - Using default implementation for ThreadExecutor
06 Oct 2016 13:11:06,815  INFO [main] SchedulerSignalerImpl:61 - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
06 Oct 2016 13:11:06,815  INFO [main] QuartzScheduler:240 - Quartz Scheduler v.2.2.1 created.
06 Oct 2016 13:11:06,816  INFO [main] JobStoreTX:670 - Using thread monitor-based data access locking (synchronization).
06 Oct 2016 13:11:06,816  INFO [main] JobStoreTX:59 - JobStoreTX initialized.
06 Oct 2016 13:11:06,817  INFO [main] QuartzScheduler:305 - Scheduler meta-data: Quartz Scheduler (v2.2.1) 'ExecutionScheduler' with instanceId 'NON_CLUSTERED'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 5 threads.
  Using job-store 'org.quartz.impl.jdbcjobstore.JobStoreTX' - which supports persistence. and is not clustered.

06 Oct 2016 13:11:06,817  INFO [main] StdSchedulerFactory:1339 - Quartz scheduler 'ExecutionScheduler' initialized from an externally provided properties instance.
06 Oct 2016 13:11:06,817  INFO [main] StdSchedulerFactory:1343 - Quartz scheduler version: 2.2.1
06 Oct 2016 13:11:06,818  INFO [main] QuartzScheduler:2311 - JobFactory set to: org.apache.ambari.server.state.scheduler.GuiceJobFactory@3d73f2a1
06 Oct 2016 13:11:06,818  INFO [main] AmbariServer:545 - ********* Started Scheduled Request Manager **********
06 Oct 2016 13:11:06,822  INFO [main] AmbariServer:548 - ********* Started Services **********
06 Oct 2016 13:11:06,851  INFO [AmbariServerAlertService] AmbariServerAlertService:257 - Scheduled server alert ambari_server_agent_heartbeat to run every 2 minutes
06 Oct 2016 13:11:52,489  WARN [qtp-ambari-agent-51] SecurityFilter:103 - Request https://ip-172-27-3-42.ap-southeast-1.compute.internal:8440/ca doesn't match any pattern.
06 Oct 2016 13:11:52,489  WARN [qtp-ambari-agent-51] SecurityFilter:62 - This request is not allowed on this port: https://ip-172-27-3-42.ap-southeast-1.compute.internal:8440/ca
06 Oct 2016 13:11:53,101  INFO [qtp-ambari-agent-51] HeartBeatHandler:927 - agentOsType = redhat7
06 Oct 2016 13:11:53,153  INFO [qtp-ambari-agent-51] HostImpl:285 - Received host registration, host=[hostname=ip-172-27-3-43,fqdn=ip-172-27-3-43.ap-southeast-1.compute.internal,domain=ap-southeast-1.compute.internal,architecture=x86_64,processorcount=8,physicalprocessorcount=8,osname=redhat,osversion=7.2,osfamily=redhat,memory=62450636,uptime_hours=3908,mounts=(available=2979420,mountpoint=/,used=17979792,percent=86%,size=20959212,device=/dev/xvda2,type=xfs)(available=31333092,mountpoint=/dev,used=0,percent=0%,size=31333092,device=devtmpfs,type=devtmpfs)(available=31225316,mountpoint=/dev/shm,used=0,percent=0%,size=31225316,device=tmpfs,type=tmpfs)(available=30856168,mountpoint=/run,used=369148,percent=2%,size=31225316,device=tmpfs,type=tmpfs)(available=1824350156,mountpoint=/data0,used=667996,percent=1%,size=1922709696,device=/dev/xvdb,type=ext4)(available=1824365240,mountpoint=/data1,used=652912,percent=1%,size=1922709696,device=/dev/xvdc,type=ext4)(available=1824365112,mountpoint=/data2,used=653040,percent=1%,size=1922709696,device=/dev/xvdd,type=ext4)(available=1820131272,mountpoint=/data3,used=4886880,percent=1%,size=1922709696,device=/dev/xvde,type=ext4)(available=1824365244,mountpoint=/data4,used=652908,percent=1%,size=1922709696,device=/dev/xvdf,type=ext4)(available=1824365232,mountpoint=/data5,used=652920,percent=1%,size=1922709696,device=/dev/xvdg,type=ext4)(available=6245064,mountpoint=/run/user/0,used=0,percent=0%,size=6245064,device=tmpfs,type=tmpfs)(available=6245064,mountpoint=/run/user/1009,used=0,percent=0%,size=6245064,device=tmpfs,type=tmpfs)(available=6245064,mountpoint=/run/user/1002,used=0,percent=0%,size=6245064,device=tmpfs,type=tmpfs)]
, registrationTime=1475730713101, agentVersion=2.2.1.0
06 Oct 2016 13:11:53,183  INFO [qtp-ambari-agent-51] TopologyManager:646 - TopologyManager.replayRequests: Entering
06 Oct 2016 13:11:53,183  INFO [qtp-ambari-agent-51] TopologyManager:316 - TopologyManager.onHostRegistered: Entering
06 Oct 2016 13:11:53,184  INFO [qtp-ambari-agent-51] TopologyManager:318 - TopologyManager.onHostRegistered: host = ip-172-27-3-43.ap-southeast-1.compute.internal is already associated with the cluster or is currently being processed
06 Oct 2016 13:11:53,274  INFO [qtp-ambari-agent-51] HeartBeatHandler:994 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, disabledComponents=, enabledComponents=METRICS_COLLECTOR}
06 Oct 2016 13:11:55,609  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:154 - Heartbeat lost from host ip-172-27-3-45.ap-southeast-1.compute.internal
06 Oct 2016 13:11:55,610  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component METRICS_MONITOR on ip-172-27-3-45.ap-southeast-1.compute.internal
06 Oct 2016 13:11:55,611  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component DATANODE on ip-172-27-3-45.ap-southeast-1.compute.internal
06 Oct 2016 13:11:55,612  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component NODEMANAGER on ip-172-27-3-45.ap-southeast-1.compute.internal
06 Oct 2016 13:11:55,641  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:154 - Heartbeat lost from host ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:11:55,641  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component METRICS_MONITOR on ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:11:55,642  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component DATANODE on ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:11:55,643  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component HIVE_SERVER on ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:11:55,644  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component MYSQL_SERVER on ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:11:55,644  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component HIVE_METASTORE on ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:11:55,645  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component WEBHCAT_SERVER on ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:11:55,647  WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component NODEMANAGER on ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:12:03,972  INFO [qtp-ambari-agent-52] HeartBeatHandler:683 - State of service component METRICS_MONITOR of service AMBARI_METRICS of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-43.ap-southeast-1.compute.internal
06 Oct 2016 13:12:03,974  INFO [qtp-ambari-agent-52] HeartBeatHandler:683 - State of service component METRICS_COLLECTOR of service AMBARI_METRICS of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-43.ap-southeast-1.compute.internal
06 Oct 2016 13:12:03,976  INFO [qtp-ambari-agent-52] HeartBeatHandler:683 - State of service component ZOOKEEPER_SERVER of service ZOOKEEPER of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-43.ap-southeast-1.compute.internal
06 Oct 2016 13:12:03,979  INFO [qtp-ambari-agent-52] HeartBeatHandler:683 - State of service component SECONDARY_NAMENODE of service HDFS of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-43.ap-southeast-1.compute.internal
06 Oct 2016 13:12:03,982  INFO [qtp-ambari-agent-52] HeartBeatHandler:683 - State of service component HISTORYSERVER of service MAPREDUCE2 of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-43.ap-southeast-1.compute.internal
06 Oct 2016 13:12:03,985  INFO [qtp-ambari-agent-52] HeartBeatHandler:683 - State of service component NAMENODE of service HDFS of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-43.ap-southeast-1.compute.internal
06 Oct 2016 13:12:03,991  INFO [qtp-ambari-agent-52] HeartBeatHandler:683 - State of service component APP_TIMELINE_SERVER of service YARN of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-43.ap-southeast-1.compute.internal
06 Oct 2016 13:12:03,994  INFO [qtp-ambari-agent-52] HeartBeatHandler:683 - State of service component RESOURCEMANAGER of service YARN of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-43.ap-southeast-1.compute.internal
06 Oct 2016 13:12:05,548  WARN [qtp-ambari-agent-52] SecurityFilter:103 - Request https://ip-172-27-3-42.ap-southeast-1.compute.internal:8440/ca doesn't match any pattern.
06 Oct 2016 13:12:05,548  WARN [qtp-ambari-agent-52] SecurityFilter:62 - This request is not allowed on this port: https://ip-172-27-3-42.ap-southeast-1.compute.internal:8440/ca
06 Oct 2016 13:12:05,962  INFO [qtp-ambari-agent-51] HeartBeatHandler:927 - agentOsType = redhat7
06 Oct 2016 13:12:06,001  INFO [qtp-ambari-agent-51] HostImpl:285 - Received host registration, host=[hostname=ip-172-27-3-44,fqdn=ip-172-27-3-44.ap-southeast-1.compute.internal,domain=ap-southeast-1.compute.internal,architecture=x86_64,processorcount=8,physicalprocessorcount=8,osname=redhat,osversion=7.2,osfamily=redhat,memory=62450636,uptime_hours=4001,mounts=(available=7132304,mountpoint=/,used=13826908,percent=66%,size=20959212,device=/dev/xvda2,type=xfs)(available=31333092,mountpoint=/dev,used=0,percent=0%,size=31333092,device=devtmpfs,type=devtmpfs)(available=31225316,mountpoint=/dev/shm,used=0,percent=0%,size=31225316,device=tmpfs,type=tmpfs)(available=30405548,mountpoint=/run,used=819768,percent=3%,size=31225316,device=tmpfs,type=tmpfs)(available=6245064,mountpoint=/run/user/1009,used=0,percent=0%,size=6245064,device=tmpfs,type=tmpfs)(available=1818844928,mountpoint=/data0,used=6173224,percent=1%,size=1922709696,device=/dev/xvdb,type=ext4)(available=1819170444,mountpoint=/data1,used=5847708,percent=1%,size=1922709696,device=/dev/xvdc,type=ext4)(available=1819149136,mountpoint=/data2,used=5869016,percent=1%,size=1922709696,device=/dev/xvdd,type=ext4)(available=1818622620,mountpoint=/data3,used=6395532,percent=1%,size=1922709696,device=/dev/xvde,type=ext4)(available=1818458600,mountpoint=/data4,used=6559552,percent=1%,size=1922709696,device=/dev/xvdf,type=ext4)(available=1819329432,mountpoint=/data5,used=5688720,percent=1%,size=1922709696,device=/dev/xvdg,type=ext4)(available=6245064,mountpoint=/run/user/1002,used=0,percent=0%,size=6245064,device=tmpfs,type=tmpfs)]
, registrationTime=1475730725962, agentVersion=2.2.1.0
06 Oct 2016 13:12:06,001  INFO [qtp-ambari-agent-51] TopologyManager:316 - TopologyManager.onHostRegistered: Entering
06 Oct 2016 13:12:06,002  INFO [qtp-ambari-agent-51] TopologyManager:318 - TopologyManager.onHostRegistered: host = ip-172-27-3-44.ap-southeast-1.compute.internal is already associated with the cluster or is currently being processed
06 Oct 2016 13:12:06,030  INFO [qtp-ambari-agent-51] HeartBeatHandler:994 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, disabledComponents=, enabledComponents=METRICS_COLLECTOR}
06 Oct 2016 13:12:16,463  INFO [qtp-ambari-agent-52] HeartBeatHandler:683 - State of service component HIVE_SERVER of service HIVE of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:12:16,467  INFO [qtp-ambari-agent-52] HeartBeatHandler:683 - State of service component WEBHCAT_SERVER of service HIVE of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:12:16,469  INFO [qtp-ambari-agent-52] HeartBeatHandler:683 - State of service component MYSQL_SERVER of service HIVE of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:12:16,472  INFO [qtp-ambari-agent-52] HeartBeatHandler:683 - State of service component METRICS_MONITOR of service AMBARI_METRICS of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:12:16,474  INFO [qtp-ambari-agent-52] HeartBeatHandler:683 - State of service component HIVE_METASTORE of service HIVE of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:12:16,475  INFO [qtp-ambari-agent-52] HeartBeatHandler:683 - State of service component NODEMANAGER of service YARN of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:12:16,477  INFO [qtp-ambari-agent-52] HeartBeatHandler:683 - State of service component DATANODE of service HDFS of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-44.ap-southeast-1.compute.internal
06 Oct 2016 13:12:23,611  WARN [qtp-ambari-agent-52] SecurityFilter:103 - Request https://ip-172-27-3-42.ap-southeast-1.compute.internal:8440/ca doesn't match any pattern.
06 Oct 2016 13:12:23,611  WARN [qtp-ambari-agent-52] SecurityFilter:62 - This request is not allowed on this port: https://ip-172-27-3-42.ap-southeast-1.compute.internal:8440/ca
06 Oct 2016 13:12:24,024  INFO [qtp-ambari-agent-55] HeartBeatHandler:927 - agentOsType = redhat7
06 Oct 2016 13:12:24,048  INFO [qtp-ambari-agent-55] HostImpl:285 - Received host registration, host=[hostname=ip-172-27-3-45,fqdn=ip-172-27-3-45.ap-southeast-1.compute.internal,domain=ap-southeast-1.compute.internal,architecture=x86_64,processorcount=8,physicalprocessorcount=8,osname=redhat,osversion=7.2,osfamily=redhat,memory=62450636,uptime_hours=4001,mounts=(available=7582428,mountpoint=/,used=13376784,percent=64%,size=20959212,device=/dev/xvda2,type=xfs)(available=31333092,mountpoint=/dev,used=0,percent=0%,size=31333092,device=devtmpfs,type=devtmpfs)(available=31225316,mountpoint=/dev/shm,used=0,percent=0%,size=31225316,device=tmpfs,type=tmpfs)(available=30757844,mountpoint=/run,used=467472,percent=2%,size=31225316,device=tmpfs,type=tmpfs)(available=6245064,mountpoint=/run/user/1009,used=0,percent=0%,size=6245064,device=tmpfs,type=tmpfs)(available=1819347548,mountpoint=/data0,used=5670604,percent=1%,size=1922709696,device=/dev/xvdb,type=ext4)(available=1818929436,mountpoint=/data1,used=6088716,percent=1%,size=1922709696,device=/dev/xvdc,type=ext4)(available=1818785956,mountpoint=/data2,used=6232196,percent=1%,size=1922709696,device=/dev/xvdd,type=ext4)(available=1818854456,mountpoint=/data3,used=6163696,percent=1%,size=1922709696,device=/dev/xvde,type=ext4)(available=1818324588,mountpoint=/data4,used=6693564,percent=1%,size=1922709696,device=/dev/xvdf,type=ext4)(available=1819333412,mountpoint=/data5,used=5684740,percent=1%,size=1922709696,device=/dev/xvdg,type=ext4)(available=6245064,mountpoint=/run/user/0,used=0,percent=0%,size=6245064,device=tmpfs,type=tmpfs)(available=6245064,mountpoint=/run/user/1002,used=0,percent=0%,size=6245064,device=tmpfs,type=tmpfs)]
, registrationTime=1475730744023, agentVersion=2.2.1.0
06 Oct 2016 13:12:24,049  INFO [qtp-ambari-agent-55] TopologyManager:316 - TopologyManager.onHostRegistered: Entering
06 Oct 2016 13:12:24,050  INFO [qtp-ambari-agent-55] TopologyManager:318 - TopologyManager.onHostRegistered: host = ip-172-27-3-45.ap-southeast-1.compute.internal is already associated with the cluster or is currently being processed
06 Oct 2016 13:12:24,067  INFO [qtp-ambari-agent-55] HeartBeatHandler:994 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, disabledComponents=, enabledComponents=METRICS_COLLECTOR}
06 Oct 2016 13:12:34,431  INFO [qtp-ambari-agent-55] HeartBeatHandler:683 - State of service component DATANODE of service HDFS of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-45.ap-southeast-1.compute.internal
06 Oct 2016 13:12:34,433  INFO [qtp-ambari-agent-55] HeartBeatHandler:683 - State of service component NODEMANAGER of service YARN of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-45.ap-southeast-1.compute.internal
06 Oct 2016 13:12:34,435  INFO [qtp-ambari-agent-55] HeartBeatHandler:683 - State of service component METRICS_MONITOR of service AMBARI_METRICS of cluster eim_edl_dev_cluster_1 has changed from UNKNOWN to STARTED at host ip-172-27-3-45.ap-southeast-1.compute.internal
06 Oct 2016 13:13:06,861  INFO [Thread-21] AbstractPoolBackedDataSource:462 - Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 1br43sa9j16aawh419l599e|58230224, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> org.postgresql.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 1br43sa9j16aawh419l599e|58230224, idleConnectionTestPeriod -> 50, initialPoolSize -> 3, jdbcUrl -> jdbc:postgresql://localhost/ambari, lastAcquisitionFailureDefaultUser -> null, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 5, maxStatements -> 0, maxStatementsPerConnection -> 120, minPoolSize -> 1, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> select 0, properties -> {user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> true, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ]
06 Oct 2016 13:13:06,913  INFO [Thread-21] JobStoreTX:861 - Freed 0 triggers from 'acquired' / 'blocked' state.
06 Oct 2016 13:13:06,921  INFO [Thread-21] JobStoreTX:871 - Recovering 0 jobs that were in-progress at the time of the last shut-down.
06 Oct 2016 13:13:06,921  INFO [Thread-21] JobStoreTX:884 - Recovery complete.
06 Oct 2016 13:13:06,921  INFO [Thread-21] JobStoreTX:891 - Removed 0 'complete' triggers.
06 Oct 2016 13:13:06,923  INFO [Thread-21] JobStoreTX:896 - Removed 0 stale fired job entries.
06 Oct 2016 13:13:06,924  INFO [Thread-21] QuartzScheduler:575 - Scheduler ExecutionScheduler_$_NON_CLUSTERED started.
8,280 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank muthukumar_siva
Rising Star

Created ‎10-11-2016 08:26 AM

Hi All,

The issue is fixed & it was a problem on the network firewall end in which they have done some changes which affected the ambari server ip which got NATing on the firewall. Today only they have responded with the details.

Really wanted to thank you for your valuable comments and time you have provided.

View solution in original post

7,035 Views
0 Kudos
11 REPLIES 11

avatar
author-rank chennuri_gouris
Super Collaborator

Created ‎10-06-2016 06:40 AM

Try to reset ambari admin password and have a try

https://community.hortonworks.com/questions/449/how-to-reset-ambari-admin-password.html

6,428 Views
0 Kudos

avatar
author-rank muthukumar_siva
Rising Star

Created ‎10-06-2016 06:52 AM

@chennuri gouri shankar

Yes tried that too with no luck.

6,428 Views
0 Kudos

avatar
Former Member

Created ‎10-06-2016 07:12 AM

@Muthukumar S

It must be a browser issue. Can you try with some other browser?

If you want to debug further then in the chroma browser you can try "Menu ---> More Tools ---> Developers Tools" and then debug the request/response when you login to console. To see if the browser is sending password or not?

Also please check if you are able to connect to ambari host port from the client machine (where you are running the browser)

telnet  ambariHost  8080
6,428 Views
0 Kudos

avatar
author-rank muthukumar_siva
Rising Star

Created ‎10-06-2016 09:09 AM

@jk We are not using chrome here and cant be installed. I have tried both on mozilla and IE in whic...

We have another cluster which is working fine on both the browsers. Only today it started giving the attached error in the initial description. Looks like somewhere something is broke and bit hard to find. It is an instance sitting on a AWS cloud. Checked with them seems all ok from their end

After i clear the cache and try to load the page, frist page comes with login prompt. 😞

e:\Users\Muthukumar_Sivagnana>telnet 172.27.3.42 8080
Connecting To 172.27.3.42...Could not open connection to the host, on port 8080:
 Connect failed

6,428 Views
0 Kudos

avatar
Former Member

Created ‎10-06-2016 09:55 AM

@Muthukumar S

If the "telnet 172.27.3.42 8080" is broken means there might be a N/W issue between the Ambari User Machine (Browser) and the AmbariServer host.

As it is placed in the AWS so can you please check if that host was restarted by any chance... which might cause an IP change.

Also can you please run the following command on your ambari server host to findout what all ports and IP Address it is listening?

## Find the AmbariServer PID
ps -ef | grep AmbariServer
 
## Then use that PID in the following command:
netstat -tnlpa | grep $PID

To findout if it is listening to correct IP and Port or not?

6,428 Views
0 Kudos

avatar
author-rank muthukumar_siva
Rising Star

Created ‎10-06-2016 10:10 AM

@jk

Host is not restarted, uptime is 166 days. Below are the outputs u have asked. Also im able to retrieve information using curl like below. Only login from ambari console throws that error.

$ curl -u admin:xxxxxxxx -H 'X-Requested-By: ambari' -X GET http://172.27.3.42:8080/api/v1/clusters/c1/services/ { "href" : "http://172.27.3.42:8080/api/v1/clusters/eim_edl_dev_cluster_1/services/", "items" : [ { "href" : "http://172.27.3.42:8080/api/v1/clusters/eim_edl_dev_cluster_1/services/AMBARI_METRICS", "ServiceInfo" : { "cluster_name" : "eim_edl_dev_cluster_1", "service_name" : "AMBARI_METRICS" } }, { "href" : "http://172.27.3.42:8080/api/v1/clusters/eim_edl_dev_cluster_1/services/HDFS", "ServiceInfo" : { "cluster_name" : "eim_edl_dev_cluster_1", "service_name" : "HDFS" } }, { "href" : "http://172.27.3.42:8080/api/v1/clusters/eim_edl_dev_cluster_1/services/HIVE", "ServiceInfo" : { "cluster_name" : "eim_edl_dev_cluster_1", "service_name" : "HIVE" } }, { "href" : "http://172.27.3.42:8080/api/v1/clusters/eim_edl_dev_cluster_1/services/MAPREDUCE2", "ServiceInfo" : { "cluster_name" : "eim_edl_dev_cluster_1", "service_name" : "MAPREDUCE2" }

 $ ps -ef | grep AmbariServer
ambari   12281     1  0 17:58 ?        00:00:00 sh -c /usr/jdk64/jdk1.8.0_60/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -Dsun.zip.disableMemoryMapping=true   -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.controller.AmbariServer > /var/log/ambari-server/ambari-server.out 2>&1 || echo $? > /var/run/ambari-server/ambari-server.exitcode &
ambari   12282 12281 13 17:58 ?        00:01:09 /usr/jdk64/jdk1.8.0_60/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -Dsun.zip.disableMemoryMapping=true -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.controller.AmbariServer

$ netstat -tnlpa | grep 12282
tcp6       0      0 :::8080                 :::*                    LISTEN      12282/java
tcp6       0      0 :::8440                 :::*                    LISTEN      12282/java
tcp6       0      0 :::8441                 :::*                    LISTEN      12282/java
tcp6       0      0 127.0.0.1:46872         127.0.0.1:5432          ESTABLISHED 12282/java
tcp6       0      0 127.0.0.1:46848         127.0.0.1:5432          ESTABLISHED 12282/java
tcp6       0      0 127.0.0.1:46813         127.0.0.1:5432          ESTABLISHED 12282/java
tcp6       0      0 172.27.3.42:8441        172.27.3.43:60598       ESTABLISHED 12282/java
tcp6       0      0 172.27.3.42:8441        172.27.3.44:34427       ESTABLISHED 12282/java
tcp6       0      0 127.0.0.1:46873         127.0.0.1:5432          ESTABLISHED 12282/java
tcp6       0      0 127.0.0.1:46808         127.0.0.1:5432          ESTABLISHED 12282/java
tcp6       0      0 127.0.0.1:46821         127.0.0.1:5432          ESTABLISHED 12282/java
tcp6       0      0 127.0.0.1:46874         127.0.0.1:5432          ESTABLISHED 12282/java
tcp6       0      0 172.27.3.42:8441        172.27.3.45:48750       ESTABLISHED 12282/java
6,428 Views
0 Kudos

avatar
author-rank muthukumar_siva
Rising Star

Created ‎10-07-2016 04:38 AM

@ ALL

I found a warning HTTP GET method message in

/var/log/ambari-server/ambari-server.out file.. 

  WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.ambari.server.api.services.BlueprintService.getBlueprints(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo), should not consume any entity.
  WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.ambari.server.api.services.BlueprintService.getBlueprint(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String), should not consume any entity.
  WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.ambari.server.api.services.ViewService.getViews(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo), should not consume any entity.
  WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.ambari.server.api.services.ViewService.getView(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String), should not consume any entity.
6,428 Views
0 Kudos

avatar
author-rank Shelton
Master Mentor

Created ‎10-08-2016 03:41 PM

The log below show your ambari is listening on 8440. Can you try 
telnet 172.27.3.42 8440
And also make sure the Ambari database derby or Mysql is running ! 
06Oct201613:12:23,611  WARN [qtp-ambari-agent-52]SecurityFilter:62-This request is not allowed on this port: https://ip-172-27-3-42.ap-southeast-1.compute.internal:8440/ca

Hope that helps

6,428 Views
0 Kudos

avatar
author-rank muthukumar_siva
Rising Star

Created ‎10-10-2016 03:22 AM

@Geoffrey Shelton Okot

Thank you for the comment. I tried 8440 and it is not going. 8080 im getting the login screen but getting the error like i mentioned above when i provide the credentials. Also 8080 is up and listening.

"Unable to connect to Ambari Server. Confirm Ambari Server is running and you can reach Ambari Server from this machine"

It is using postgresql and it is up and running.

Also i see this error on ambari-server.log. It is an aws instance.

"07 Oct 2016 17:54:17,897 INFO [main] Configuration:843 - Credential provider creation failed. Reason: Master key initialization failed. 10 Oct 2016 10:57:10,610 INFO [main] Configuration:843 - Credential provider creation failed. Reason: Master key initialization failed."

[root@ip-172-27-3-42.ap-southeast-1.compute.internal]:/home/ambari
$ service postgresql status -l
Redirecting to /bin/systemctl status  -l postgresql.service
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-10-07 17:41:32 SGT; 2 days ago
  Process: 2720 ExecStart=/usr/bin/pg_ctl start -D ${PGDATA} -s -o -p ${PGPORT} -w -t 300 (code=exited, status=0/SUCCESS)
  Process: 2714 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)
 Main PID: 2723 (postgres)
   CGroup: /system.slice/postgresql.service
           ├─ 2723 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
           ├─ 2724 postgres: logger process
           ├─ 2726 postgres: checkpointer process
           ├─ 2727 postgres: writer process
           ├─ 2728 postgres: wal writer process
           ├─ 2729 postgres: autovacuum launcher process
           ├─ 2730 postgres: stats collector process
           ├─12788 postgres: ambari ambari 127.0.0.1(54787) idle
           ├─12790 postgres: ambari ambari 127.0.0.1(54789) idle
           ├─12795 postgres: ambari ambari 127.0.0.1(54792) idle
           ├─12932 postgres: ambari ambari 127.0.0.1(54799) idle
           ├─12933 postgres: ambari ambari 127.0.0.1(54800) idle
           ├─12934 postgres: ambari ambari 127.0.0.1(54801) idle
           └─12935 postgres: ambari ambari 127.0.0.1(54802) idle

Oct 07 17:41:30 ip-172-27-3-42.ap-southeast-1.compute.internal systemd[1]: Starting PostgreSQL database server...
Oct 07 17:41:32 ip-172-27-3-42.ap-southeast-1.compute.internal systemd[1]: Started PostgreSQL database server.
6,428 Views
0 Kudos
Announcements
Product Announcements
[ANNOUNCE] Cloudera on Public Cloud Runtime 7.2.18 is GA!
What's New @ Cloudera
Cloudera Operational Database (COD) supports enhancements to...
Community Announcements
March 2024 Community Highlights
Community Announcements
Celebrating 100,000 Members: A Journey of Growth and Connect...
Product Announcements
[ANNOUNCE] New Cloudera JDBC Connector for Apache Hive 2.6.2...
View More Announcements