Support Questions
Find answers, ask questions, and share your expertise

How to resolve the Ambari issue when trying HDP Sandbox on AWS EC2?

How to resolve the Ambari issue when trying HDP Sandbox on AWS EC2?

Explorer

[Moving original question from Hadoop Core track to Cloud & Operations]

I created an image using the .ova file from HDP Sandbox 2.4 for virtual box. I am able to access the publicip:8888 link but not the 8080.

Upon checking the ambari-server logs, I get the following error:

05 Apr 2016 19:41:04,011  INFO [main] Configuration:746 - Reading password from existing file
05 Apr 2016 19:41:04,841  INFO [main] Configuration:746 - Reading password from existing file
05 Apr 2016 19:41:05,659  INFO [main] Configuration:746 - Reading password from existing file
05 Apr 2016 19:41:06,488  INFO [main] Configuration:746 - Reading password from existing file
05 Apr 2016 19:41:07,295  INFO [main] Configuration:746 - Reading password from existing file
05 Apr 2016 19:41:08,082  INFO [main] Configuration:746 - Reading password from existing file
05 Apr 2016 19:41:08,912  INFO [main] Configuration:746 - Reading password from existing file
05 Apr 2016 19:41:09,729  INFO [main] Configuration:746 - Reading password from existing file
05 Apr 2016 19:41:27,110  INFO [main] AmbariServer:173 - Found org/apache/ambari/server/controller/AmbariServer.class class in file:/usr/lib/ambari-server/ambari-server-2.2.1.0.161.jar!/org/apache/ambari/server/controller/AmbariServer.class
05 Apr 2016 19:41:27,163  INFO [main] Configuration:746 - Reading password from existing file
05 Apr 2016 19:41:27,171  INFO [main] Configuration:1076 - Hosts Mapping File null
05 Apr 2016 19:41:27,171  INFO [main] HostsMap:60 - Using hostsmap file null
05 Apr 2016 19:41:27,587  INFO [main] ControllerModule:193 - Detected POSTGRES as the database type from the JDBC URL
05 Apr 2016 19:41:29,042  INFO [main] ControllerModule:578 - Binding and registering notification dispatcher class org.apache.ambari.server.notifications.dispatchers.EmailDispatcher
05 Apr 2016 19:41:29,092  INFO [main] ControllerModule:578 - Binding and registering notification dispatcher class org.apache.ambari.server.notifications.dispatchers.SNMPDispatcher
05 Apr 2016 19:41:29,094  INFO [main] ControllerModule:578 - Binding and registering notification dispatcher class org.apache.ambari.server.notifications.dispatchers.AlertScriptDispatcher
05 Apr 2016 19:42:34,009 ERROR [main] DBAccessorImpl:102 - Error while creating database accessor 
org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:207)
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)
    at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:138)
    at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
    at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
    at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)
    at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
    at org.postgresql.Driver.makeConnection(Driver.java:410)
    at org.postgresql.Driver.connect(Driver.java:280)
    at java.sql.DriverManager.getConnection(DriverManager.java:571)
    at java.sql.DriverManager.getConnection(DriverManager.java:215)
    at org.apache.ambari.server.orm.DBAccessorImpl.<init>(DBAccessorImpl.java:83)
    at org.apache.ambari.server.orm.DBAccessorImpl$FastClassByGuice$86dbc63e.newInstance(<generated>)
    at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
    at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
    at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
    at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
    at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.Scopes$1$1.get(Scopes.java:65)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
    at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
    at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.Scopes$1$1.get(Scopes.java:65)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
    at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
    at com.google.inject.internal.InjectionRequestProcessor$StaticInjection$1.call(InjectionRequestProcessor.java:116)
    at com.google.inject.internal.InjectionRequestProcessor$StaticInjection$1.call(InjectionRequestProcessor.java:110)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
    at com.google.inject.internal.InjectionRequestProcessor$StaticInjection.injectMembers(InjectionRequestProcessor.java:110)
    at com.google.inject.internal.InjectionRequestProcessor.injectMembers(InjectionRequestProcessor.java:78)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:170)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
    at com.google.inject.Guice.createInjector(Guice.java:95)
    at com.google.inject.Guice.createInjector(Guice.java:72)
    at com.google.inject.Guice.createInjector(Guice.java:62)
    at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:803)
Caused by: java.net.ConnectException: Connection timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:579)
    at java.net.Socket.connect(Socket.java:528)
    at org.postgresql.core.PGStream.<init>(PGStream.java:60)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:101)
    ... 47 more

I notice, postgres processes related to ambari are not started:

ps -f -u postgres
UID        PID  PPID  C STIME TTY          TIME CMD
postgres  5414     1  0 19:36 ?        00:00:00 /usr/bin/postmaster -p 5432 -D /var/lib/pgsql/data
postgres  5416  5414  0 19:36 ?        00:00:00 postgres: logger process                        
postgres  5418  5414  0 19:36 ?        00:00:00 postgres: writer process                        
postgres  5419  5414  0 19:36 ?        00:00:00 postgres: wal writer process  

lsof -n -u postgres |grep LISTEN
postmaste 1532 postgres    3u  IPv4              20222      0t0    TCP *:postgres (LISTEN)
postmaste 1532 postgres    4u  IPv6              20223      0t0    TCP *:postgres (LISTEN)

How to resolve this error?

4 REPLIES 4

Re: How to resolve the Ambari issue when trying HDP Sandbox on AWS EC2?

Super Guru

@Piyush Routray

Can you please check /var/lib/psql/pg_hba.conf and update ident to trust as shown in below example

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
# "local" is for Unix domain socket connections only
local   all         all                               trust
# IPv4 local connections:
host    all         all         127.0.0.1/32          trust
host    all         all         100.200.300.50/32       trust
# IPv6 local connections:
host    all         all         ::1/128               ident

Re: How to resolve the Ambari issue when trying HDP Sandbox on AWS EC2?

Explorer

Yes, I have tried that and it didn't help.

Re: How to resolve the Ambari issue when trying HDP Sandbox on AWS EC2?

@Piyush Routray

The postgres process seem to be up and running:

postgres 5414 1 0 19:36?00:00:00/usr/bin/postmaster -p 5432-D /var/lib/pgsql/data

Please run netstat -ntupl | grep -i 5414 and ensure that port 5432 is indeed locked by postgres process.

Also try to connect to postgres and ensure that the connection is going ok - probably do a \connect ambari and \dt *

If the Postgres db is fine,

  1. review ambari.properties an ensure that the postgres is specified as teh ambari db
  2. check if iptables are running and turn off if its running
  3. Check /var/lib/pgsql/data/postgresql.conf and ensure that 'listen_addresses = '*' # what IP address(es) to listen on;' if not set this to * or the ipaddress of the sandbox
  4. Restart postgresql service [even if the above change is not done]

Re: How to resolve the Ambari issue when trying HDP Sandbox on AWS EC2?

@Piyush Routray

Just to make sure if you can also have a look at aws instances security rules, and make sure you have opened all ports for testing(time being). If it works then you can just add specific rules you need.

Below is the sample snap of security rules for aws -

3298-screen-shot-2016-04-09-at-104630-am.png