Hello.
I am trying to read partitioned parquet table from s3 using impala. Table creating is successful:
create external table database.table
like parquet 's3a://bucket/table/dt=2020-11-06/file'
partitioned by (dt string)
stored as parquet
location "s3a://bucket/table";
Select to this table returns empty, but in fact bucket is not empty. Show partitions also returns empty result
I am trying to make it read partitions, but both queries get stuck and fail after `Read Timeout` from Metastore
ALTER TABLE database.table RECOVER PARTITIONS;
REFRESH database.table PARTITION (dt="2020-11-06");
Metastore logs (for ALTER TABLE):
2020-11-07T10:14:01,782 INFO [pool-5-thread-8] metastore.HiveMetaStore: 8: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
2020-11-07T10:14:01,783 INFO [pool-5-thread-8] metastore.ObjectStore: ObjectStore, initialize called
2020-11-07T10:14:01,927 INFO [pool-5-thread-8] metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL
2020-11-07T10:14:01,927 INFO [pool-5-thread-8] metastore.ObjectStore: Initialized ObjectStore
2020-11-07T10:14:01,927 INFO [pool-5-thread-8] metastore.HiveMetaStore: 8: add_partitions
2020-11-07T10:14:01,927 INFO [pool-5-thread-8] HiveMetaStore.audit: ugi=root ip=127.0.0.1 cmd=add_partitions
Metastore logs (for REFRESH):
2020-11-07T09:44:52,939 INFO [pool-5-thread-7] metastore.HiveMetaStore: 7: source:127.0.0.1 get_partition_by_name: tbl=hive.database.table part=dt=2020-11-06
2020-11-07T09:44:52,940 INFO [pool-5-thread-7] HiveMetaStore.audit: ugi=root ip=127.0.0.1 cmd=source:127.0.0.1 get_partition_by_name: tbl=hive.database.table part=dt=2020-11-06
2020-11-07T09:44:52,940 INFO [pool-5-thread-7] metastore.HiveMetaStore: 7: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
2020-11-07T09:44:52,941 INFO [pool-5-thread-7] metastore.ObjectStore: ObjectStore, initialize called
2020-11-07T09:44:53,014 INFO [pool-5-thread-7] metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL
2020-11-07T09:44:53,014 INFO [pool-5-thread-7] metastore.ObjectStore: Initialized ObjectStore
Impalad logs:
I1107 10:18:00.572021 3732 Frontend.java:1245] 5f4f65bc6d08c533:5e808dfc00000000] Analyzing query: ALTER TABLE database.table RECOVER PARTITIONS db: database
I1107 10:18:00.572461 3732 Frontend.java:1286] 5f4f65bc6d08c533:5e808dfc00000000] Analysis finished.
I1107 10:19:01.868741 3451 client-request-state.cc:187] 034a1b4883de285d:cf0f264b00000000] ImpalaRuntimeException: Error making 'add_partition' RPC to Hive Metastore:
CAUSED BY: TTransportException: java.net.SocketTimeoutException: Read timed out
CAUSED BY: SocketTimeoutException: Read timed out
Can you please suggest, what could have gone wrong?
Thank you
