Member since
05-05-2017
11
Posts
20
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
748 | 08-01-2018 04:43 PM | |
1530 | 08-01-2018 02:26 PM |
08-08-2019
02:56 PM
Hi Could you compare with the number of files on the same period ? It could be an explanation. Look the size of the files too, little size files and empty files could be an other explanation. Regards.
... View more
08-08-2019
09:50 AM
Hi @Leonardo Araujo, don't know if you did it. About the collateral effect on 2.6.4, there is one to take in account, the Hive Acid activation change the transactional manager from : hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager To hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager The big impact is the way of Hive manage the locks even for no transactionnal tables as describe in Hive documentation about LockManager . This could dramatically increase processing times of your jobs using Hive. You could use the parameter hive.txn.strict.locking.mode to manage the behavior of the LockManager even if the documentation said that it is only available since Hive 2.2, use set -v to show presence of this parameter in Hive 1.2.1000 . You could hit 2 bugs too : HIVE-17900 : Compaction of ACID table might fail if table is partitioned by more than one column HIVE-18293 : Compaction of ACID table failed when using doas Hope this will help. Best regards, Eric.
... View more
10-24-2018
02:57 PM
1 Kudo
Hello Clement and David Yes, the Failed to get database default, returning NoSuchObjectException is not a very clear message, this mean that the default Hive database on hdfs is not reachable : hdfs://clustername/user/hive/warehouse/ So this is not a postgresql problem but a hdfs client problem. Regards
... View more
10-23-2018
03:13 PM
Hi David Did you solve this problem ? Best regards. Eric.
... View more
08-06-2018
07:08 AM
Hi Always use the newer version, less bugs, more security and functionality. Even if it is not in the matrix, I use postgresl 9.5 for ambari, hive, ranger, oozie, hue. Keep in mind that if you plan to upgrade to HDP 3 and ambari 2.7 you have to upgrade to postgresql 10.2. Regards.
... View more
08-01-2018
04:43 PM
1 Kudo
Hi You can practice the exam on AWS : https://fr.hortonworks.com/services/training/certification/hdpca-certification/ you have to follow this doc for using the AWS practice : https://2xbbhjxc6wk3v21p62t8n4d4-wpengine.netdna-ssl.com/wp-content/uploads/2015/04/HDPCA-PracticeExamGuide.pdf Just be carefull the AMI are not available on all the AWS zone, try us-east-1 (USA East, Virginie). Good luck. Regards. Eric.
... View more
08-01-2018
04:16 PM
Hi Have you check that Hue is listening on the right port 443 (https) ? 443 is the https default port, so your browser remove it from the url and replace it by https:// in front of your ip address. You could try with another port if you want to verify, ex: 15001 Regards
... View more
08-01-2018
02:54 PM
Hi First what is your database type ? In Ambari alerts you have a "Ambari Server Performance" alert with default treshold : warning : 3 000 ms critical : 5 000 ms So 8 ms should not issue an alert. What do you mean by latency ? The time to open a connection to the database ? The time for a simple request to answer ? On my Postgresql database I have many queries answers under 1 ms and others takes 3 ms to 9 ms, but I have other db that ambari ( hive, oozie, hue, ranger, ...) Regards. Eric
... View more
08-01-2018
02:26 PM
13 Kudos
Hi Here is the answer : After setup your postgresql external database access with ambari-server setup, you have to edit the file /etc/ambari-server/conf/ambari.properties and modify the server.jdbc.url parameter to add the activation of the ssl like this : server.jdbc.url=jdbc:postgresql://<HOSTNAME>:<PORT>/ambari?ssl=true Once done, you are able to start the ambari installation with a SSL connection to your external Postgresql database. For more information on all the options you could add on the connect string, you have the jdbc.postgresql documentation here : https://jdbc.postgresql.org/documentation/head/connect.html#ssl Eric
... View more
08-01-2018
02:20 PM
4 Kudos
When installing a cluster with a very secure environement you could want to setup an external Postgresql cluster database with HA and a connection pooler like pgbouncer for reducing the impact of opening new sessions on Postgresql. All the connections should be with SSL/TLS : Client -> Pgbouncer and Pgbouncer -> Postgresql The problem was that configuring Ambari with the ambari-server setup don't give you the oportunity to setup SSL connection and ambari is not able to connect to the database. You will find this error in the logs : 26 Jul 2018 18:56:39,202 ERROR [main] DBAccessorImpl:119 - Error while creating database accessor
org.postgresql.util.PSQLException: ERROR: SSL required
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:400)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:138)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:410)
at org.postgresql.Driver.connect(Driver.java:280)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.ambari.server.orm.DBAccessorImpl.<init>(DBAccessorImpl.java:93)
at org.apache.ambari.server.orm.DBAccessorImpl$$FastClassByGuice$$86dbc63e.newInstance(<generated>)
at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)
at com.google.inject.internal.InjectionRequestProcessor$StaticInjection$1.call(InjectionRequestProcessor.java:116)
at com.google.inject.internal.InjectionRequestProcessor$StaticInjection$1.call(InjectionRequestProcessor.java:110)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
at com.google.inject.internal.InjectionRequestProcessor$StaticInjection.injectMembers(InjectionRequestProcessor.java:110)
at com.google.inject.internal.InjectionRequestProcessor.injectMembers(InjectionRequestProcessor.java:78)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:170)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
at com.google.inject.Guice.createInjector(Guice.java:95)
at com.google.inject.Guice.createInjector(Guice.java:72)
at com.google.inject.Guice.createInjector(Guice.java:62)
at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1045)
So, the question is how to setup your ambari server for connecting to the database without desactivating SSL mode ? Eric.
... View more
Labels:
- Labels:
-
Apache Ambari
08-01-2018
08:13 AM
1 Kudo
If you want a Postgresql HA for your Ambari and the other components like Ranger, Hive, Oozie, you have to use an external postgresql database, not the embeded. In front of your Postgresql HA, use a connection pooler like pgbouncer or pgpool, so you will reduce the impact of opening a client connection to the database by keeping sessions open between the connection pooler and the database.
... View more