Member since
08-08-2017
1652
Posts
30
Kudos Received
11
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1921 | 06-15-2020 05:23 AM | |
| 15487 | 01-30-2020 08:04 PM | |
| 2074 | 07-07-2019 09:06 PM | |
| 8121 | 01-27-2018 10:17 PM | |
| 4573 | 12-31-2017 10:12 PM |
01-30-2020
06:04 AM
more * LOG: database system was interrupted; last known up at 2020-01-22 23:31:37 UTC LOG: redo starts at 39/C7000080 LOG: consistent recovery state reached at 39/C70474A8 FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up FATAL: the database system is starting up LOG: could not open file "pg_xlog/000000010000003A00000023" (log file 58, segment 35): No such file or directory LOG: redo done at 3A/22036568 LOG: last completed transaction was at log time 2020-01-30 10:41:43.937312+00 FATAL: the database system is starting up LOG: autovacuum launcher started LOG: database system is ready to accept connections WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress WARNING: there is already a transaction in progress
... View more
01-30-2020
06:01 AM
now I see the correct syntax ( I use full FQDN ) /usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-9.3-1101-jdbc4.jar org.apache.ambari.server.DBConnectionVerification jdbc:postgresql://`hostname -f`:5432/ambari ambari bigdata org.postgresql.Driver Connected to DB Successfully!
... View more
01-30-2020
06:00 AM
now I see the correct syntax ( I use full FQDN ) /usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-9.3-1101-jdbc4.jar org.apache.ambari.server.DBConnectionVerification jdbc:postgresql://`hostname -f`:5432/ambari ambari bigdata org.postgresql.Driver Connected to DB Successfully!
... View more
01-30-2020
05:57 AM
sorry for my mistake the correct is /usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-9.3-1101-jdbc4.jar org.apache.ambari.server.DBConnectionVerification jdbc:postgresql://localhost:5432/ambari ambari bigdata org.postgresql.Driver but its hang ( not return prompt )
... View more
01-30-2020
05:54 AM
about the second test how to find the HOSTNAME of PostgresDB ?
... View more
01-30-2020
05:53 AM
Dear Jay here the first test /usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-42.2.2.jar org.apache.ambari.server.DBConnectionVerification jdbc:postgresql://localhost:5432/ambari ambari bigdata org.postgresql.Driver ERROR: Unable to connect to the DB. Please check DB connection properties. java.lang.ClassNotFoundException: org.postgresql.Driver
... View more
01-30-2020
05:32 AM
We have ambari cluster
we have issue with ambari server
example
ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...Ambari database consistency check started...Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start............................................................ERROR: Exiting with exit code 1.REASON: Server not yet listening on http port 8080 after 50 seconds. Exiting.
ambari server create the PID and service is up but server not listening on port 8080
we check the firewall and iptables but this services are down
we check the PostgreSQL service ( is running well )
systemctl status postgresql.service● postgresql.service - PostgreSQL database server
Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2020-01-30 12:49:26 UTC; 20min ago
we also checked the following
cat /etc/ambari-server/conf/password.dat
bigdata
grep 'ambari' /var/lib/pgsql/data/pg_hba.conf
local all ambari,mapred md5
host all ambari,mapred 0.0.0.0/0 md5
host all ambari,mapred ::/0 md5
grep 'jdbc' /etc/ambari-server/conf/ambari.properties
custom.postgres.jdbc.name=postgresql-jdbc.jar
previous.custom.postgres.jdbc.name=postgresql-jdbc.jar
server.jdbc.connection-pool=internal
server.jdbc.database=postgres
server.jdbc.database_name=ambari
server.jdbc.postgres.schema=ambari
server.jdbc.user.name=ambari
but seems they are ok
so what could be the reason that the default port 8080 of ambari server isn’t opened
30 Jan 2020 12:51:50,152 ERROR [main] DBAccessorImpl:119 - Error while creating database accessor
org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:207) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:138) at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) at org.postgresql.Driver.makeConnection(Driver.java:410) at org.postgresql.Driver.connect(Driver.java:280) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:247) at org.apache.ambari.server.orm.DBAccessorImpl.<init>(DBAccessorImpl.java:93) at org.apache.ambari.server.orm.DBAccessorImpl$$FastClassByGuice$$86dbc63e.newInstance(<generated>) at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.Scopes$1$1.get(Scopes.java:65) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.Scopes$1$1.get(Scopes.java:65) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.Scopes$1$1.get(Scopes.java:65) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) at com.google.inject.internal.InjectionRequestProcessor$StaticInjection$1.call(InjectionRequestProcessor.java:116) at com.google.inject.internal.InjectionRequestProcessor$StaticInjection$1.call(InjectionRequestProcessor.java:110) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) at com.google.inject.internal.InjectionRequestProcessor$StaticInjection.injectMembers(InjectionRequestProcessor.java:110) at com.google.inject.internal.InjectionRequestProcessor.injectMembers(InjectionRequestProcessor.java:78) at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:170) at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109) at com.google.inject.Guice.createInjector(Guice.java:95) at com.google.inject.Guice.createInjector(Guice.java:72) at com.google.inject.Guice.createInjector(Guice.java:62) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1045)Caused by: java.net.ConnectException: Connection timed out (Connection timed out) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at org.postgresql.core.PGStream.<init>(PGStream.java:60) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:101)
... 52 more
... View more
Labels:
- Labels:
-
Apache Ambari
01-28-2020
02:00 PM
Jay - can you help me with this post - https://community.cloudera.com/t5/Support-Questions/how-to-recover-bad-namenode-from-good-namenode/td-p/288471
... View more
01-28-2020
09:45 AM
we can verify the HDFS version on hadoop cluster
as the following:
su hdfs
hdfs version
but since hdfs client also installed on all data-nodes machines
we need to verify also the hdfs client version
any cli / command / binaries files , that can captured the hdfs client version ?
... View more
Labels:
01-15-2020
10:27 AM
We have 10 kafka machines with kafka version - 1.X
this kafka cluster version is part of HDP version - 2.6.5
We noticed that under /var/log/kafka/server.log the following message
ERROR Error while accepting connection {kafka.network.Accetpr} java.io.IOException: Too many open files
We saw also additionally
Broker 21 stopped fetcher for partition ...................... because they are in the failed log dir /kafka/kafka-logs {kafka.server.ReplicaManager}
and
WARN Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. this implies messages have arrived out of order. New: {epoch:0, offset:2227488}, Currnet: {epoch 2, offset:261} for Partition: cars-list-75 {kafka.server.epochLeaderEpocHFileCache}
so regarding to the issue -
ERROR Error while accepting connection {kafka.network.Accetpr} java.io.IOException: Too many open files
how to increase the MAX open files , in order to avoid this issue
... View more
Labels:
- Labels:
-
Apache Kafka