Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

bucket y delta in hive

Labels:
avatar
author-rank pacosoplas
Master Collaborator

Created on ‎06-04-2016 11:22 AM - edited ‎08-19-2019 03:29 AM

Hi:

I have lunched 5 batches, one per month and in hive directories i can see that why??? inside de delta there are all the bucket, is corret???

4756-snip20160604-1.png

4757-snip20160604-2.png

6,184 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank ekoifman
Super Collaborator

Created ‎06-17-2016 07:09 PM

both compactions are important. The should run minor compactions more frequently to combine delta files into fewer deltas. From the listing above major compaction ran and produced the base file. All the delta files in the listing should be garbage collected by the system. Could you provide output of "show locks" and "show transactions"? The only thing that should prevent deletion of these delta files is a query that is still running which the "show" commands should indicate.

View solution in original post

6,022 Views
17 REPLIES 17

avatar
author-rank pacosoplas
Master Collaborator

Created on ‎06-04-2016 12:30 PM - edited ‎08-19-2019 03:28 AM

hI:

Thanks, i was cheking my configuration and i can see that,i put run compactor true and 1 thread, also 5 delta files, but still there are 5 directories file why??

4759-snip20160604-5.png

4758-snip20160604-4.png

3,130 Views
0 Kudos

avatar
author-rank ekoifman
Super Collaborator

Created ‎06-05-2016 03:36 AM

Try looking at your metastore log file for any errors from ql.Initiator or ql.Worker.

3,130 Views

avatar
author-rank pacosoplas
Master Collaborator

Created ‎06-06-2016 02:39 AM 

2016-06-05 09:58:26,880 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 0, isactive true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:938)


2016-06-05 09:58:26,880 DEBUG [Thread-8]: compactor.CompactorThread (CompactorThread.java:findUserToRunAs(153)) - Determining who to run the job as.


2016-06-05 09:58:26,880 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20810


2016-06-05 09:58:26,881 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20810


2016-06-05 09:58:26,881 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 1ms


2016-06-05 09:58:26,881 DEBUG [Thread-8]: compactor.CompactorThread (CompactorThread.java:findUserToRunAs(158)) - Running job as hive


2016-06-05 09:58:26,881 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20811


2016-06-05 09:58:26,881 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20811


2016-06-05 09:58:26,881 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 0ms


2016-06-05 09:58:26,881 DEBUG [Thread-8]: io.AcidUtils (AcidUtils.java:getAcidState(439)) - in directory hdfs://xxxxxx/RSI/tables/logs/journey_entidad_importe base = null deltas = 5


2016-06-05 09:58:26,881 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20812


2016-06-05 09:58:26,882 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20812


2016-06-05 09:58:26,882 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 1ms


2016-06-05 09:58:26,882 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20813


2016-06-05 09:58:26,882 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20813


2016-06-05 09:58:26,882 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 0ms


2016-06-05 09:58:26,883 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20814


2016-06-05 09:58:26,883 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20814


2016-06-05 09:58:26,883 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 0ms


2016-06-05 09:58:26,883 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20815


2016-06-05 09:58:26,884 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20815


2016-06-05 09:58:26,884 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 1ms


2016-06-05 09:58:26,884 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20816


2016-06-05 09:58:26,884 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20816


2016-06-05 09:58:26,885 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 1ms


2016-06-05 09:58:26,885 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20817


2016-06-05 09:58:26,885 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20817


2016-06-05 09:58:26,885 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 0ms


2016-06-05 09:58:26,886 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20818


2016-06-05 09:58:26,886 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20818


2016-06-05 09:58:26,886 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 0ms


2016-06-05 09:58:26,886 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20819


2016-06-05 09:58:26,886 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20819


2016-06-05 09:58:26,886 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 0ms


2016-06-05 09:58:26,887 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20820


2016-06-05 09:58:26,887 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20820


2016-06-05 09:58:26,887 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 0ms


2016-06-05 09:58:26,887 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20821


2016-06-05 09:58:26,888 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20821


2016-06-05 09:58:26,888 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 1ms


2016-06-05 09:58:26,888 INFO  [Thread-8]: compactor.Initiator (Initiator.java:run(89)) - Checking to see if we should compact default.temp_canal_bucketed_v4


2016-06-05 09:58:26,888 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Open transaction: count = 1, isActive = true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:936)


2016-06-05 09:58:26,888 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Open transaction: count = 2, isActive = true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:999)


2016-06-05 09:58:26,889 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 1, isactive true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1007)


2016-06-05 09:58:26,889 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 0, isactive true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:938)


2016-06-05 09:58:26,889 INFO  [Thread-8]: compactor.Initiator (Initiator.java:run(94)) - Can't find table default.temp_canal_bucketed_v4, assuming it's a temp table or has been dropped and moving on.


2016-06-05 09:58:26,889 INFO  [Thread-8]: compactor.Initiator (Initiator.java:run(89)) - Checking to see if we should compact default.temp_wordcount_v3


2016-06-05 09:58:26,889 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Open transaction: count = 1, isActive = true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:936)


2016-06-05 09:58:26,889 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Open transaction: count = 2, isActive = true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:999)


2016-06-05 09:58:26,890 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 1, isactive true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1007)


2016-06-05 09:58:26,890 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 0, isactive true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:938)


2016-06-05 09:58:26,890 INFO  [Thread-8]: compactor.Initiator (Initiator.java:run(94)) - Can't find table default.temp_wordcount_v3, assuming it's a temp table or has been dropped and moving on.


2016-06-05 09:58:26,890 INFO  [Thread-8]: compactor.Initiator (Initiator.java:run(89)) - Checking to see if we should compact default.temp_canal_bucketed_v3


2016-06-05 09:58:26,890 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Open transaction: count = 1, isActive = true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:936)


2016-06-05 09:58:26,890 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Open transaction: count = 2, isActive = true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:999)


2016-06-05 09:58:26,892 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 1, isactive true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1007)


2016-06-05 09:58:26,895 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 0, isactive true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:938)


2016-06-05 09:58:26,896 DEBUG [Thread-8]: compactor.CompactorThread (CompactorThread.java:findUserToRunAs(153)) - Determining who to run the job as.


2016-06-05 09:58:26,896 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20822


2016-06-05 09:58:26,896 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20822


2016-06-05 09:58:26,896 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 0ms


2016-06-05 09:58:26,896 DEBUG [Thread-8]: compactor.CompactorThread (CompactorThread.java:findUserToRunAs(158)) - Running job as dangulo


2016-06-05 09:58:26,896 INFO  [Thread-8]: compactor.Initiator (Initiator.java:checkForCompaction(208)) - Going to initiate as user dangulo


2016-06-05 09:58:26,896 DEBUG [Thread-8]: security.UserGroupInformation (UserGroupInformation.java:logPrivilegedAction(1681)) - PrivilegedAction as:dangulo (auth:PROXY) via hive (auth:SIMPLE) from:org.apache.hadoop.hive.ql.txn.compactor.Initiator.checkForCompaction(Initiator.java:211)


2016-06-05 09:58:26,896 DEBUG [Thread-8]: hdfs.BlockReaderLocal (DFSClient.java:<init>(457)) - dfs.client.use.legacy.blockreader.local = false


2016-06-05 09:58:26,896 DEBUG [Thread-8]: hdfs.BlockReaderLocal (DFSClient.java:<init>(460)) - dfs.client.read.shortcircuit = true


2016-06-05 09:58:26,897 DEBUG [Thread-8]: hdfs.BlockReaderLocal (DFSClient.java:<init>(463)) - dfs.client.domain.socket.data.traffic = false


2016-06-05 09:58:26,897 DEBUG [Thread-8]: hdfs.BlockReaderLocal (DFSClient.java:<init>(466)) - dfs.domain.socket.path = /var/lib/hadoop-hdfs/dn_socket


2016-06-05 09:58:26,897 DEBUG [Thread-8]: retry.RetryUtils (RetryUtils.java:getDefaultRetryPolicy(75)) - multipleLinearRandomRetry = null


2016-06-05 09:58:26,897 DEBUG [Thread-8]: ipc.Client (ClientCache.java:getClient(63)) - getting client out of cache: org.apache.hadoop.ipc.Client@1e2b326c


2016-06-05 09:58:26,897 DEBUG [Thread-8]: sasl.DataTransferSaslUtil (DataTransferSaslUtil.java:getSaslPropertiesResolver(183)) - DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection


2016-06-05 09:58:26,897 DEBUG [Thread-8]: ipc.Client (Client.java:<init>(432)) - The ping interval is 60000 ms.


2016-06-05 09:58:26,898 DEBUG [Thread-8]: ipc.Client (Client.java:setupIOstreams(702)) - Connecting to xxxxxx


2016-06-05 09:58:26,898 DEBUG [IPC Client (324702500) connection to xxxxxx from dangulo]: ipc.Client (Client.java:run(972)) - IPC Client (324702500) connection to xxxxxx from dangulo: starting, having connections 2


2016-06-05 09:58:26,900 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from dangulo sending #20823


2016-06-05 09:58:26,905 DEBUG [IPC Client (324702500) connection to xxxxxx from dangulo]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from dangulo got value #20823


2016-06-05 09:58:26,905 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 8ms


2016-06-05 09:58:26,907 DEBUG [Thread-8]: io.AcidUtils (AcidUtils.java:getAcidState(439)) - in directory hdfs://xxxxxx/apps/hive/warehouse/temp_canal_bucketed_v3 base = null deltas = 0


2016-06-05 09:58:26,907 DEBUG [Thread-8]: compactor.Initiator (Initiator.java:determineCompactionType(271)) - delta size: 0 base size: 25034508 threshold: 0.1 will major compact: false


2016-06-05 09:58:26,907 INFO  [Thread-8]: compactor.Initiator (Initiator.java:run(89)) - Checking to see if we should compact default.journey_entidad_acid


2016-06-05 09:58:26,907 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Open transaction: count = 1, isActive = true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:936)


2016-06-05 09:58:26,907 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Open transaction: count = 2, isActive = true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:999)


2016-06-05 09:58:26,909 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 1, isactive true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1007)


2016-06-05 09:58:26,913 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 0, isactive true at:


	org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:938)


2016-06-05 09:58:26,914 DEBUG [Thread-8]: compactor.CompactorThread (CompactorThread.java:findUserToRunAs(153)) - Determining who to run the job as.


2016-06-05 09:58:26,914 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20824


2016-06-05 09:58:26,914 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20824


2016-06-05 09:58:26,914 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 0ms


2016-06-05 09:58:26,914 DEBUG [Thread-8]: compactor.CompactorThread (CompactorThread.java:findUserToRunAs(158)) - Running job as hive


2016-06-05 09:58:26,914 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20825


2016-06-05 09:58:26,915 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20825


2016-06-05 09:58:26,915 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 1ms


2016-06-05 09:58:26,915 DEBUG [Thread-8]: io.AcidUtils (AcidUtils.java:getAcidState(439)) - in directory hdfs://xxxxxx/RSI/tables/logs/journey_entidad_ACID base = null deltas = 5


2016-06-05 09:58:26,915 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20826


2016-06-05 09:58:26,915 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20826


2016-06-05 09:58:26,915 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 0ms


2016-06-05 09:58:26,915 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20827


2016-06-05 09:58:26,916 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20827


2016-06-05 09:58:26,916 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 1ms


2016-06-05 09:58:26,916 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20828


2016-06-05 09:58:26,916 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20828


2016-06-05 09:58:26,916 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 0ms


2016-06-05 09:58:26,917 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20829

Did you see any error???

thanks

3,130 Views
0 Kudos

avatar
author-rank pacosoplas
Master Collaborator

Created ‎06-06-2016 03:20 PM

Hi:

Still not the compaction have been executed and any error on the metastore log as you can see above.

3,130 Views
0 Kudos

avatar
author-rank ekoifman
Super Collaborator

Created ‎06-06-2016 07:09 PM

Your Compactor Initiator seems to be running. I don't see any errors.

2016-06-05 09:58:26,907 INFO  [Thread-8]: compactor.Initiator (Initiator.java:run(89)) - Checking to see if we should compact default.journey_entidad_acid

What version of Hive is this?

3,130 Views
0 Kudos

avatar
author-rank pacosoplas
Master Collaborator

Created on ‎06-17-2016 05:04 PM - edited ‎08-19-2019 03:28 AM

hi: today i execute the manual compactor and i seem it work but what happend now with the other delta files??

it created a base directory but there are alot files i need to delete somenthing??? what is better major o minor compaction??

ALTER TABLE canal_acid  COMPACT 'Major'

5100-snip20160617-5.png

3,130 Views
0 Kudos

avatar
author-rank ekoifman
Super Collaborator

Created ‎06-17-2016 07:09 PM

both compactions are important. The should run minor compactions more frequently to combine delta files into fewer deltas. From the listing above major compaction ran and produced the base file. All the delta files in the listing should be garbage collected by the system. Could you provide output of "show locks" and "show transactions"? The only thing that should prevent deletion of these delta files is a query that is still running which the "show" commands should indicate.

6,023 Views

avatar
author-rank pacosoplas
Master Collaborator

Created ‎06-18-2016 10:01 AM

hi:

yuo are right now just tnere is a base file 🙂 many thanks

3,130 Views
0 Kudos
Announcements
What's New @ Cloudera
[RELEASED] Cloudera Streaming Analytics 1.14 for Cloudera Pu...
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
View More Announcements