Member since
08-08-2017
1652
Posts
30
Kudos Received
11
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2058 | 06-15-2020 05:23 AM | |
| 17024 | 01-30-2020 08:04 PM | |
| 2220 | 07-07-2019 09:06 PM | |
| 8554 | 01-27-2018 10:17 PM | |
| 4837 | 12-31-2017 10:12 PM |
01-30-2020
05:32 AM
We have ambari cluster
we have issue with ambari server
example
ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...Ambari database consistency check started...Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start............................................................ERROR: Exiting with exit code 1.REASON: Server not yet listening on http port 8080 after 50 seconds. Exiting.
ambari server create the PID and service is up but server not listening on port 8080
we check the firewall and iptables but this services are down
we check the PostgreSQL service ( is running well )
systemctl status postgresql.service● postgresql.service - PostgreSQL database server
Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2020-01-30 12:49:26 UTC; 20min ago
we also checked the following
cat /etc/ambari-server/conf/password.dat
bigdata
grep 'ambari' /var/lib/pgsql/data/pg_hba.conf
local all ambari,mapred md5
host all ambari,mapred 0.0.0.0/0 md5
host all ambari,mapred ::/0 md5
grep 'jdbc' /etc/ambari-server/conf/ambari.properties
custom.postgres.jdbc.name=postgresql-jdbc.jar
previous.custom.postgres.jdbc.name=postgresql-jdbc.jar
server.jdbc.connection-pool=internal
server.jdbc.database=postgres
server.jdbc.database_name=ambari
server.jdbc.postgres.schema=ambari
server.jdbc.user.name=ambari
but seems they are ok
so what could be the reason that the default port 8080 of ambari server isn’t opened
30 Jan 2020 12:51:50,152 ERROR [main] DBAccessorImpl:119 - Error while creating database accessor
org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:207) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:138) at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) at org.postgresql.Driver.makeConnection(Driver.java:410) at org.postgresql.Driver.connect(Driver.java:280) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:247) at org.apache.ambari.server.orm.DBAccessorImpl.<init>(DBAccessorImpl.java:93) at org.apache.ambari.server.orm.DBAccessorImpl$$FastClassByGuice$$86dbc63e.newInstance(<generated>) at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.Scopes$1$1.get(Scopes.java:65) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.Scopes$1$1.get(Scopes.java:65) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.Scopes$1$1.get(Scopes.java:65) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) at com.google.inject.internal.InjectionRequestProcessor$StaticInjection$1.call(InjectionRequestProcessor.java:116) at com.google.inject.internal.InjectionRequestProcessor$StaticInjection$1.call(InjectionRequestProcessor.java:110) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) at com.google.inject.internal.InjectionRequestProcessor$StaticInjection.injectMembers(InjectionRequestProcessor.java:110) at com.google.inject.internal.InjectionRequestProcessor.injectMembers(InjectionRequestProcessor.java:78) at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:170) at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109) at com.google.inject.Guice.createInjector(Guice.java:95) at com.google.inject.Guice.createInjector(Guice.java:72) at com.google.inject.Guice.createInjector(Guice.java:62) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1045)Caused by: java.net.ConnectException: Connection timed out (Connection timed out) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at org.postgresql.core.PGStream.<init>(PGStream.java:60) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:101)
... 52 more
... View more
Labels:
- Labels:
-
Apache Ambari
01-28-2020
02:00 PM
Jay - can you help me with this post - https://community.cloudera.com/t5/Support-Questions/how-to-recover-bad-namenode-from-good-namenode/td-p/288471
... View more
01-28-2020
09:45 AM
we can verify the HDFS version on hadoop cluster
as the following:
su hdfs
hdfs version
but since hdfs client also installed on all data-nodes machines
we need to verify also the hdfs client version
any cli / command / binaries files , that can captured the hdfs client version ?
... View more
Labels:
01-15-2020
10:27 AM
We have 10 kafka machines with kafka version - 1.X
this kafka cluster version is part of HDP version - 2.6.5
We noticed that under /var/log/kafka/server.log the following message
ERROR Error while accepting connection {kafka.network.Accetpr} java.io.IOException: Too many open files
We saw also additionally
Broker 21 stopped fetcher for partition ...................... because they are in the failed log dir /kafka/kafka-logs {kafka.server.ReplicaManager}
and
WARN Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. this implies messages have arrived out of order. New: {epoch:0, offset:2227488}, Currnet: {epoch 2, offset:261} for Partition: cars-list-75 {kafka.server.epochLeaderEpocHFileCache}
so regarding to the issue -
ERROR Error while accepting connection {kafka.network.Accetpr} java.io.IOException: Too many open files
how to increase the MAX open files , in order to avoid this issue
... View more
Labels:
- Labels:
-
Apache Kafka
01-12-2020
11:24 AM
since metrics take space on the local disk ,
we want to move to distributed mode , so metrics data will be on HDFS and not on local disk
we start to work according to the link - https://cwiki.apache.org/confluence/display/AMBARI/AMS+-+distributed+mode
we are using ambari cluster 2.6.4 with active/stand by name node
so we start the configuration
first we set in ambari the following
ams-site, Set timeline.metrics.service.operation.mode = distributed
now we need to Edit ams-hbase-site,
hdfs://<nameservice-id>/user/ams/hbase
the question is - how to find the nameservice-id ?
since we have active/stand by name node
then how to know the - nameservice-id value?
... View more
Labels:
- Labels:
-
Apache Ambari
12-07-2019
12:19 PM
hi all
we have 5 kafka brokers
we can stop or start the kafka service by API
but how to stop specific kafka broker by REST API?
for example , in this pic we can see 5 kafka brokers , and we want to stop/start only the second kafka broker by REST API
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Kafka
12-03-2019
12:24 PM
Dear Jay so from your example , I understand that we need to capture the - "state" from the output for each kafka broker , while expected output should be OK am I right until now?
... View more
12-03-2019
04:48 AM
hi all
we want to identify kafka alert from ambari by rest api
Example when we get alert
what is the AMBARI REST API in order to verify alert on kafka broker01 or kafka broker02 , etc
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Kafka
12-02-2019
02:48 AM
yes we get the following: cat /etc/group | grep -i hadoop hadoop:x:1006:hive,livy,zookeeper,spark,ams,kafka,yarn,hcat,mapred cat /etc/group | grep -i airflow hdfs:x:1005:hdfs,hive,airflow airflow:x:1016: cat /etc/group | grep -i hdfs hdfs:x:1005:hdfs,hive,airflow let me know if you need additional info?
... View more
12-02-2019
02:09 AM
so when we do hdfs dfs -ls -R / | grep "airflow " | awk '{print $1" "$2" "$3" "$4" "}' we get: drwxrwx--- - airflow hadoop drwxrwx--- - airflow hadoop drwxrwx--- - airflow hadoop -rw-r----- 3 airflow hadoop -rw-r----- 3 airflow hadoop -rw-r----- 3 airflow hadoop drwxrwx--- - airflow hadoop -rw-r----- 3 airflow hadoop -rw-r----- 3 airflow hadoop -rw-r----- 3 airflow hadoop . . . . do you means to change every hadoop group to hdfs ? or in simple words - how to know the HDFS path , that we need to change?
... View more