Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive - create index fail when loading data to index table step

Hive - create index fail when loading data to index table step

New Contributor

- Hi Community, i'm new in Hadoop development and there's certainly something i do badly. I try to create an index in a partitionned table with MR mode. So when i rebuild the index, i get a Hive exception : "Unable to alter partition. alter is not possible". Is it regular the alter partition step when altering the index in rebuild index process ? Then why hive can't alter the partition of the index table ? Thanks a lot for your help

Here additional information :

------------------------------------------------------------------------------------------------------------------------------------------------------------------ #ENVOIRONNEMENT# ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Hive 1.2.1.2.4

HDFS 2.7.1.2.4 ------------------------------------------------------------------------------------------------------------------------------------------------------------------

#CREATE TABLE#

------------------------------------------------------------------------------------------------------------------------------------------------------------------

CREATE TABLE 3C_DWH.NMSBROADLOGRCP ( IBROADLOGID INT, IDELIVERYID INT, IFLAGS SMALLINT, IMSGID INT, IRECIPIENTID INT, ISERVICEID INT, ISTATUS SMALLINT, SADDRESS STRING, TSEVENT TIMESTAMP, TSLASTMODIFIED TIMESTAMP, IRESULTAT_APPEL SMALLINT, SCONTRATNUMERO_DU_CONTRAT STRING, SMESSAGE STRING, SSEGMENTCODE STRING, LB_RECORD_SRC STRING, TS_RECORD_CREATION TIMESTAMP, TS_RECORD_MODIFICATION TIMESTAMP, TS_RECORD_ARCHIVE TIMESTAMP ) PARTITIONED BY (DT_MONTH_EVENT STRING) CLUSTERED BY (IRECIPIENTID) INTO 16 BUCKETS STORED AS ORC tblproperties ( 'orc.compress'='SNAPPY');

------------------------------------------------------------------------------------------------------------------------------------------------------------------

#CREATE INDEX#

------------------------------------------------------------------------------------------------------------------------------------------------------------------

DROP INDEX IF EXISTS nmsbroadlogrcp_index ON nmsbroadlogrcp;

DROP TABLE IF EXISTS nmsbroadlogrcp_index_table;

CREATE INDEX nmsbroadlogrcp_index ON TABLE nmsbroadlogrcp (ideliveryid) AS 'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler' WITH DEFERRED REBUILD IN TABLE msbroadlogrcp_index_table --PARTITIONED BY (dt_month_event) ;

------------------------------------------------------------------------------------------------------------------------------------------------------------------

#REBUILD INDEX#

------------------------------------------------------------------------------------------------------------------------------------------------------------------

ALTER INDEX nmsbroadlogrcp_index ON nmsbroadlogrcp REBUILD;

------------------------------------------------------------------------------------------------------------------------------------------------------------------

#ERROR LOG#

------------------------------------------------------------------------------------------------------------------------------------------------------------------

... Launching Job 14 out of 14 Number of reduce tasks not specified. Estimated from input data size: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1476195381098_0037, Tracking URL = .../proxy/application_1476195381098_0037/ Kill Command = /usr/hdp/2.4.0.0-169/hadoop/bin/hadoop job -kill job_1476195381098_0037 Hadoop job information for Stage-53: number of mappers: 1; number of reducers: 1 2016-10-12 15:43:17,751 Stage-53 map = 0%, reduce = 0% 2016-10-12 15:43:24,032 Stage-53 map = 100%, reduce = 0%, Cumulative CPU 3.28 sec 2016-10-12 15:43:30,425 Stage-53 map = 100%, reduce = 100%, Cumulative CPU 6.83 sec MapReduce Total cumulative CPU time: 6 seconds 830 msec Ended Job = job_1476195381098_0037 Loading data to table nmsbroadlogrcp_index_table partition (dt_month_event=201504) org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter partition. alter is not possible at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition(Hive.java:578) at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition(Hive.java:549) at org.apache.hadoop.hive.ql.index.IndexMetadataChangeTask.execute(IndexMetadataChangeTask.java:74) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1720) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1477) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1254) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: InvalidOperationException(message:alter is not possible) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_partition_result$alter_partition_resultStandardScheme.read(ThriftHiveMetastore.java) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_partition_result$alter_partition_resultStandardScheme.read(ThriftHiveMetastore.java) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_partition_result.read(ThriftHiveMetastore.java) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_alter_partition(ThriftHiveMetastore.java:2343) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.alter_partition(ThriftHiveMetastore.java:2328) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partition(HiveMetaStoreClient.java:1311) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156) at com.sun.proxy.$Proxy5.alter_partition(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition(Hive.java:573) ... 21 more Error changing index table/partition metadata Unable to alter partition. alter is not possible FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.index.IndexMetadataChangeTask MapReduce Jobs Launched: Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 5.85 sec HDFS Read: 18869 HDFS Write: 220 SUCCESS Stage-Stage-5: Map: 1 Reduce: 1 Cumulative CPU: 12.29 sec HDFS Read: 2008962 HDFS Write: 8606223 SUCCESS Stage-Stage-9: Map: 1 Reduce: 2 Cumulative CPU: 27.79 sec HDFS Read: 4571103 HDFS Write: 23252947 SUCCESS Stage-Stage-13: Map: 1 Reduce: 3 Cumulative CPU: 41.62 sec HDFS Read: 7247982 HDFS Write: 40002638 SUCCESS Stage-Stage-17: Map: 2 Reduce: 7 Cumulative CPU: 100.59 sec HDFS Read: 15265419 HDFS Write: 98994678 SUCCESS Stage-Stage-21: Map: 2 Reduce: 8 Cumulative CPU: 110.13 sec HDFS Read: 18435647 HDFS Write: 119484997 SUCCESS Stage-Stage-25: Map: 2 Reduce: 9 Cumulative CPU: 125.57 sec HDFS Read: 21760655 HDFS Write: 136252522 SUCCESS Stage-Stage-29: Map: 2 Reduce: 8 Cumulative CPU: 109.23 sec HDFS Read: 18313591 HDFS Write: 117124457 SUCCESS Stage-Stage-33: Map: 2 Reduce: 8 Cumulative CPU: 109.1 sec HDFS Read: 18744411 HDFS Write: 118529966 SUCCESS Stage-Stage-37: Map: 2 Reduce: 9 Cumulative CPU: 125.2 sec HDFS Read: 21799320 HDFS Write: 136408779 SUCCESS Stage-Stage-41: Map: 2 Reduce: 6 Cumulative CPU: 85.11 sec HDFS Read: 16577146 HDFS Write: 86599094 SUCCESS Stage-Stage-45: Map: 1 Reduce: 1 Cumulative CPU: 5.76 sec HDFS Read: 61542 HDFS Write: 29594 SUCCESS Stage-Stage-49: Map: 1 Reduce: 1 Cumulative CPU: 5.67 sec HDFS Read: 45472 HDFS Write: 9478 SUCCESS Stage-Stage-53: Map: 1 Reduce: 1 Cumulative CPU: 6.83 sec HDFS Read: 50505 HDFS Write: 13731 SUCCESS Total MapReduce CPU Time Spent: 14 minutes 30 seconds 740 msec

2 REPLIES 2
Highlighted

Re: Hive - create index fail when loading data to index table step

I don't see this issue on my environment.

See if you are missing step below because Tez does not support create or rebuild:-

set hive.execution.engine=mr;

Re: Hive - create index fail when loading data to index table step

New Contributor

Thank you for your concern. I already modified this option before the process. I've setted it to map reduce. the log attached is whith this. So the process can't success so long as the alter partition 's not finished.