Member since
10-03-2016
9
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
480 | 11-08-2019 06:26 AM |
08-08-2020
09:17 AM
Thanks a tonn!!!! I was struggling with the same issue , it helped , Thanks
... View more
04-11-2020
06:48 AM
'HDP--3.1.4,The table containing the parquet timestamp which has hourly data ,hive server is pushing the hour data into into next date example is shown below , please check before and after 29 th Mar 2020 , where Mar 29 is the BST time settings with day light saving' | 2020-03-22 | 2020-03-22 00:00:59.0 | 2020-03-22 23:59:59.0 |
| 2020-03-23 | 2020-03-23 00:00:59.0 | 2020-03-23 23:59:59.0 |
| 2020-03-24 | 2020-03-24 00:00:59.0 | 2020-03-24 23:59:59.0 |
| 2020-03-25 | 2020-03-25 00:00:59.0 | 2020-03-25 23:59:59.0 |
| 2020-03-26 | 2020-03-26 00:00:59.0 | 2020-03-26 23:59:59.0 |
| 2020-03-27 | 2020-03-27 00:00:59.0 | 2020-03-27 23:59:59.0 |
| 2020-03-28 | 2020-03-28 00:00:59.0 | 2020-03-28 23:59:59.0 |
| 2020-03-29 | 2020-03-29 00:00:59.0 | 2020-03-30 00:59:59.0 |
| 2020-03-30 | 2020-03-30 01:00:59.0 | 2020-03-31 00:59:59.0 |
| 2020-03-31 | 2020-03-31 01:00:59.0 | 2020-04-01 00:59:59.0 |
| 2020-04-01 | 2020-04-01 01:00:59.0 | 2020-04-02 00:59:59.0 |
| 2020-04-02 | 2020-04-02 01:00:59.0 | 2020-04-03 00:59:59.0 |
... View more
11-08-2019
06:26 AM
Hi @rguruvannagari Thanks for alot for the reply , not sure if heap space is filled during compaction or ranger hive audit if we set hive authentication to none then it is ok , please see the following issue. https://community.cloudera.com/t5/Support-Questions/hive-metastore-is-not-responding-but-alive-with-the/m-p/282224 Thanks Nag
... View more
11-07-2019
05:50 AM
hive metastore is consuming complete heap(tried max heap : 24G) after sometime the metastore is not responding , the metastore is ok if we disable ranger hive authentication this issue is reported in (https://jira.apache.org/jira/browse/HIVE-20568) , It seems ranger plugin is causing memory leaks during audit when it encounters hive db names containing '_' just wondering to know if there is a work around for it in HDP-3.1.0 or the upgrade from HDP-3.1 to HDP-3.1.4 will fix it, Thanks in advance. Current Version and config Information : ======================== HDP-3.1 hive-3.1 Hive Authentication - Ranger JDK-1.8 mysql -7.5 mysql-connector-8.0.13.jar
... View more
Labels:
11-06-2019
07:11 AM
Version Information :
HDP -3.1
Hive -3.1
Mysql -5.7.25 , connector : mysql-8.0.13.*jar
Hive Metastore Heap Size : 8G
Hive Server Heap Size : 16 G
Concurrent connections : nearly - 40 (may be an idle process for long hours)
GC : G1GC
Hive metastore is going down for every two hours if we increase the hive metastore memory(16G) it is not responding after 4 hrs .Compactor is on and uses single thread ,Hive metastore has its GC filled and not responding is there any way we can improve the GC , Any thoughts on this please ,
Many Thanks
Error message :
===================
Caused by: HikariPool-1 - Connection is not available, request timed out after 213730ms. at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:602) at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195) at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145) at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:85) at org.datanucleus.store.rdbms.ConnectionProviderPriorityList.getConnection(ConnectionProviderPriorityList.java:57) at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:402) at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getXAResource(ConnectionFactoryImpl.java:361) at org.datanucleus.store.connection.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:316) at org.datanucleus.store.connection.AbstractConnectionFactory.getConnection(AbstractConnectionFactory.java:84) at org.datanucleus.store.rdbms.RDBMSStoreManager.getNucleusConnection(RDBMSStoreManager.java:1351) at org.datanucleus.api.jdo.JDOPersistenceManager.getDataStoreConnection(JDOPersistenceManager.java:2272); Caused by: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. at com.mysql.cj.protocol.FullReadInputStream.readFully(FullReadInputStream.java:67) at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:63) at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:45) at com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:549) Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. at org.datanucleus.store.connection.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:316) at org.datanucleus.store.connection.AbstractConnectionFactory.getConnection(AbstractConnectionFactory.java:84) Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. at org.datanucleus.store.connection.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:316) at org.datanucleus.store.connection.AbstractConnectionFactory.getConnection(AbstractConnectionFactory.java:84) Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. at org.datanucleus.store.connection.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:316) ~[datanucleus-core-4.1.17.jar:?] at org.datanucleus.store.connection.AbstractConnectionFactory.getConnection(AbstractConnectionFactory.java:84) ~[datanucleus-core-4.1.17.jar:?] Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. at org.datanucleus.store.connection.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:316) ~[datanucleus-core-4.1.17.jar:?] at org.datanucleus.store.connection.AbstractConnectionFactory.getConnection(AbstractConnectionFactory.java:84) ~[datanucleus-core-4.1.17.jar:?] Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. Binary file (standard input) matches
... View more
06-10-2019
08:17 AM
Trying to write an orc table from PySpark ,orc files to hive table with more than 200k columns which is failing with org.apache.spark.sql.AnalysisException:org.apache.hadoop.hive.metadata.HiveException :Unable to alter table Exception thrown flushing changes to datastore
... View more
05-28-2019
10:50 AM
It seems it is not included in HDP yet please use pig as alternative
... View more
12-11-2018
02:30 PM
Hi Sahina just wondering if you had any solution for this issue.
... View more
10-03-2016
06:32 PM
1 Kudo
do the following to increase the dfs size : Create multiple directories or mount points in the hdfs data path : by default ambari deployed cluster contain /hadoop/hdfs/data as the data directory , so with root privileges : create a directory 1) mkdir /hadoop/hdfs/data1 2) chown -R hdfs:hadoop /hadoop/hdfs/data1 3) chmod -R 777 /hadoop/hdfs/data1 now edit the hdfs configuration : 1) on the cluster click on hdfs , click on configs , in the settings add the directory separated by comma under the hdfs.data.dir property : ex : /hadoop/hdfs/data, /hadoop/hdfs/data1 save the changes and restart the effected That will increase the disk space , to increase further repeat the same (or) lvs resize /hadoop/hdfs/data directory , do the following to increase the dfs size : Create multiple directories or mount points in the hdfs data path : by default ambari deployed cluster contain /hadoop/hdfs/data as the data directory , so with root privileges : create a directory 1) mkdir /hadoop/hdfs/data1 2) chown -R hdfs:hadoop /hadoop/hdfs/data1 3) chmod -R 777 /hadoop/hdfs/data1 now edit the hdfs configuration : 1) on the cluster click on hdfs , click on configs , in the settings add the directory separated by comma under the hdfs.data.dir property : ex : /hadoop/hdfs/data, /hadoop/hdfs/data1 save the changes and restart the effected That will increase the disk space , to increase further repeat the same (or) lvs resize /hadoop/hdfs/data directory
... View more