I mean if i need to change into the table, i need to delete everything and insert again?? ... View more

2016-06-05 09:58:26,880 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 0, isactive true at: org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:938) 2016-06-05 09:58:26,880 DEBUG [Thread-8]: compactor.CompactorThread (CompactorThread.java:findUserToRunAs(153)) - Determining who to run the job as. 2016-06-05 09:58:26,880 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20810 2016-06-05 09:58:26,881 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20810 2016-06-05 09:58:26,881 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 1ms 2016-06-05 09:58:26,881 DEBUG [Thread-8]: compactor.CompactorThread (CompactorThread.java:findUserToRunAs(158)) - Running job as hive 2016-06-05 09:58:26,881 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20811 2016-06-05 09:58:26,881 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20811 2016-06-05 09:58:26,881 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 0ms 2016-06-05 09:58:26,881 DEBUG [Thread-8]: io.AcidUtils (AcidUtils.java:getAcidState(439)) - in directory hdfs://xxxxxx/RSI/tables/logs/journey_entidad_importe base = null deltas = 5 2016-06-05 09:58:26,881 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20812 2016-06-05 09:58:26,882 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20812 2016-06-05 09:58:26,882 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 1ms 2016-06-05 09:58:26,882 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20813 2016-06-05 09:58:26,882 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20813 2016-06-05 09:58:26,882 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 0ms 2016-06-05 09:58:26,883 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20814 2016-06-05 09:58:26,883 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20814 2016-06-05 09:58:26,883 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 0ms 2016-06-05 09:58:26,883 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20815 2016-06-05 09:58:26,884 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20815 2016-06-05 09:58:26,884 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 1ms 2016-06-05 09:58:26,884 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20816 2016-06-05 09:58:26,884 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20816 2016-06-05 09:58:26,885 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 1ms 2016-06-05 09:58:26,885 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20817 2016-06-05 09:58:26,885 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20817 2016-06-05 09:58:26,885 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 0ms 2016-06-05 09:58:26,886 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20818 2016-06-05 09:58:26,886 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20818 2016-06-05 09:58:26,886 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 0ms 2016-06-05 09:58:26,886 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20819 2016-06-05 09:58:26,886 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20819 2016-06-05 09:58:26,886 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 0ms 2016-06-05 09:58:26,887 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20820 2016-06-05 09:58:26,887 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20820 2016-06-05 09:58:26,887 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 0ms 2016-06-05 09:58:26,887 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20821 2016-06-05 09:58:26,888 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20821 2016-06-05 09:58:26,888 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 1ms 2016-06-05 09:58:26,888 INFO [Thread-8]: compactor.Initiator (Initiator.java:run(89)) - Checking to see if we should compact default.temp_canal_bucketed_v4 2016-06-05 09:58:26,888 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Open transaction: count = 1, isActive = true at: org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:936) 2016-06-05 09:58:26,888 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Open transaction: count = 2, isActive = true at: org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:999) 2016-06-05 09:58:26,889 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 1, isactive true at: org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1007) 2016-06-05 09:58:26,889 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 0, isactive true at: org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:938) 2016-06-05 09:58:26,889 INFO [Thread-8]: compactor.Initiator (Initiator.java:run(94)) - Can't find table default.temp_canal_bucketed_v4, assuming it's a temp table or has been dropped and moving on. 2016-06-05 09:58:26,889 INFO [Thread-8]: compactor.Initiator (Initiator.java:run(89)) - Checking to see if we should compact default.temp_wordcount_v3 2016-06-05 09:58:26,889 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Open transaction: count = 1, isActive = true at: org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:936) 2016-06-05 09:58:26,889 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Open transaction: count = 2, isActive = true at: org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:999) 2016-06-05 09:58:26,890 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 1, isactive true at: org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1007) 2016-06-05 09:58:26,890 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 0, isactive true at: org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:938) 2016-06-05 09:58:26,890 INFO [Thread-8]: compactor.Initiator (Initiator.java:run(94)) - Can't find table default.temp_wordcount_v3, assuming it's a temp table or has been dropped and moving on. 2016-06-05 09:58:26,890 INFO [Thread-8]: compactor.Initiator (Initiator.java:run(89)) - Checking to see if we should compact default.temp_canal_bucketed_v3 2016-06-05 09:58:26,890 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Open transaction: count = 1, isActive = true at: org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:936) 2016-06-05 09:58:26,890 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Open transaction: count = 2, isActive = true at: org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:999) 2016-06-05 09:58:26,892 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 1, isactive true at: org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1007) 2016-06-05 09:58:26,895 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 0, isactive true at: org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:938) 2016-06-05 09:58:26,896 DEBUG [Thread-8]: compactor.CompactorThread (CompactorThread.java:findUserToRunAs(153)) - Determining who to run the job as. 2016-06-05 09:58:26,896 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20822 2016-06-05 09:58:26,896 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20822 2016-06-05 09:58:26,896 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 0ms 2016-06-05 09:58:26,896 DEBUG [Thread-8]: compactor.CompactorThread (CompactorThread.java:findUserToRunAs(158)) - Running job as dangulo 2016-06-05 09:58:26,896 INFO [Thread-8]: compactor.Initiator (Initiator.java:checkForCompaction(208)) - Going to initiate as user dangulo 2016-06-05 09:58:26,896 DEBUG [Thread-8]: security.UserGroupInformation (UserGroupInformation.java:logPrivilegedAction(1681)) - PrivilegedAction as:dangulo (auth:PROXY) via hive (auth:SIMPLE) from:org.apache.hadoop.hive.ql.txn.compactor.Initiator.checkForCompaction(Initiator.java:211) 2016-06-05 09:58:26,896 DEBUG [Thread-8]: hdfs.BlockReaderLocal (DFSClient.java:<init>(457)) - dfs.client.use.legacy.blockreader.local = false 2016-06-05 09:58:26,896 DEBUG [Thread-8]: hdfs.BlockReaderLocal (DFSClient.java:<init>(460)) - dfs.client.read.shortcircuit = true 2016-06-05 09:58:26,897 DEBUG [Thread-8]: hdfs.BlockReaderLocal (DFSClient.java:<init>(463)) - dfs.client.domain.socket.data.traffic = false 2016-06-05 09:58:26,897 DEBUG [Thread-8]: hdfs.BlockReaderLocal (DFSClient.java:<init>(466)) - dfs.domain.socket.path = /var/lib/hadoop-hdfs/dn_socket 2016-06-05 09:58:26,897 DEBUG [Thread-8]: retry.RetryUtils (RetryUtils.java:getDefaultRetryPolicy(75)) - multipleLinearRandomRetry = null 2016-06-05 09:58:26,897 DEBUG [Thread-8]: ipc.Client (ClientCache.java:getClient(63)) - getting client out of cache: org.apache.hadoop.ipc.Client@1e2b326c 2016-06-05 09:58:26,897 DEBUG [Thread-8]: sasl.DataTransferSaslUtil (DataTransferSaslUtil.java:getSaslPropertiesResolver(183)) - DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection 2016-06-05 09:58:26,897 DEBUG [Thread-8]: ipc.Client (Client.java:<init>(432)) - The ping interval is 60000 ms. 2016-06-05 09:58:26,898 DEBUG [Thread-8]: ipc.Client (Client.java:setupIOstreams(702)) - Connecting to xxxxxx 2016-06-05 09:58:26,898 DEBUG [IPC Client (324702500) connection to xxxxxx from dangulo]: ipc.Client (Client.java:run(972)) - IPC Client (324702500) connection to xxxxxx from dangulo: starting, having connections 2 2016-06-05 09:58:26,900 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from dangulo sending #20823 2016-06-05 09:58:26,905 DEBUG [IPC Client (324702500) connection to xxxxxx from dangulo]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from dangulo got value #20823 2016-06-05 09:58:26,905 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 8ms 2016-06-05 09:58:26,907 DEBUG [Thread-8]: io.AcidUtils (AcidUtils.java:getAcidState(439)) - in directory hdfs://xxxxxx/apps/hive/warehouse/temp_canal_bucketed_v3 base = null deltas = 0 2016-06-05 09:58:26,907 DEBUG [Thread-8]: compactor.Initiator (Initiator.java:determineCompactionType(271)) - delta size: 0 base size: 25034508 threshold: 0.1 will major compact: false 2016-06-05 09:58:26,907 INFO [Thread-8]: compactor.Initiator (Initiator.java:run(89)) - Checking to see if we should compact default.journey_entidad_acid 2016-06-05 09:58:26,907 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Open transaction: count = 1, isActive = true at: org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:936) 2016-06-05 09:58:26,907 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Open transaction: count = 2, isActive = true at: org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:999) 2016-06-05 09:58:26,909 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 1, isactive true at: org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1007) 2016-06-05 09:58:26,913 DEBUG [Thread-8]: metastore.ObjectStore (ObjectStore.java:debugLog(6848)) - Commit transaction: count = 0, isactive true at: org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:938) 2016-06-05 09:58:26,914 DEBUG [Thread-8]: compactor.CompactorThread (CompactorThread.java:findUserToRunAs(153)) - Determining who to run the job as. 2016-06-05 09:58:26,914 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20824 2016-06-05 09:58:26,914 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20824 2016-06-05 09:58:26,914 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 0ms 2016-06-05 09:58:26,914 DEBUG [Thread-8]: compactor.CompactorThread (CompactorThread.java:findUserToRunAs(158)) - Running job as hive 2016-06-05 09:58:26,914 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20825 2016-06-05 09:58:26,915 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20825 2016-06-05 09:58:26,915 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 1ms 2016-06-05 09:58:26,915 DEBUG [Thread-8]: io.AcidUtils (AcidUtils.java:getAcidState(439)) - in directory hdfs://xxxxxx/RSI/tables/logs/journey_entidad_ACID base = null deltas = 5 2016-06-05 09:58:26,915 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20826 2016-06-05 09:58:26,915 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20826 2016-06-05 09:58:26,915 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 0ms 2016-06-05 09:58:26,915 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20827 2016-06-05 09:58:26,916 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20827 2016-06-05 09:58:26,916 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getListing took 1ms 2016-06-05 09:58:26,916 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20828 2016-06-05 09:58:26,916 DEBUG [IPC Client (324702500) connection to xxxxxx from hive]: ipc.Client (Client.java:receiveRpcResponse(1092)) - IPC Client (324702500) connection to xxxxxx from hive got value #20828 2016-06-05 09:58:26,916 DEBUG [Thread-8]: ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(250)) - Call: getFileInfo took 0ms 2016-06-05 09:58:26,917 DEBUG [IPC Parameter Sending Thread #242]: ipc.Client (Client.java:run(1035)) - IPC Client (324702500) connection to xxxxxx from hive sending #20829 Did you see any error??? thanks ... View more

Yes i restarted all hive services, i didnt try do manually, ill wait more days to see if its working, if not ill do manually. any other suggeston? o parameter that is wrong on my picture?? ... View more

Many thanks, so, i conclusion, my files are hive in ORC format, so all the computation will be in memory right?? Also where keep drill the hive metastore??? thanks ... View more

hI: Thanks, i was cheking my configuration and i can see that,i put run compactor true and 1 thread, also 5 delta files, but still there are 5 directories file why?? ... View more

Hi: and i can do that manually?? how? ... View more

so, this month it will be in production, so what happend if the diary batch fail this day??? ... View more

Hi: whaMany thanks i create that and it faster now, but i cant do ACID operation, so, what can i do?? CREATE EXTERNAL TABLE IF NOT EXISTS journey_oficina( FECHAOPRCNF timestamp, codnrbeenf string, CODINTERNO string, CODTXF string, FREQ BIGINT ) CLUSTERED BY (codnrbeenf) SORTED BY (FECHAOPRCNF) INTO 60 BUCKETS stored as ORC LOCATION '/RSI/tables/logs/journey_oficina' TBLPROPERTIES ("immutable"="false", "orc.create.index"="true","orc.compress"="ZLIB", "orc.stripe.size"="67108864","orc.row.index.stride"="25000"); thanks ... View more