Member since
10-16-2015
6
Posts
2
Kudos Received
0
Solutions
01-11-2017
09:06 PM
Your create table statement should define partition name. CREATE EXTERNAL TABLE table_name (
...
)
PARTITIONED BY (moat_date STRING);
... View more
01-11-2017
07:25 PM
@suresh krish Please share your table schema( describe table table_name) and the select query Regards, Anubhav
... View more
01-11-2017
07:13 PM
2 Kudos
Hi @Saikrishna Tarapareddy 1. You can add partitions to existing table using ALTER TABLE <table_name> ADD PARTITION (date=' ') LOCATION '/path' 2. For dynamic partitioning: --Set following two properties for your Hive session: SET hive.exec.dynamic.partition=true;
SET hive.exec.dynamic.partition.mode=nonstrict; --Create your staging table without Partition create external table table_name (col_01,col_02) row format delimited fields terminated by ","; --load data to your Hive Table: load data local inpath '/path' into table table_name; --Create production table with the columns you want to partition upon. In this case you want to partition upon date column: create external table table_name (col_01,col_02) partitioned by (date_part string) row format delimited fields terminated by ','; insert overwrite table table_name partition(date_part) select col_01,col_02 from staging_table; --Verify data is correctly populated in partition: hive> select * from table_name where date_part="20170111";
Regards, Anubhav
... View more
11-15-2016
09:05 PM
@Ashnee Sharma Try restarting Hiveserver2. Also are you able to run Hive cli ?
... View more
10-19-2016
07:34 PM
We are trying to distcp between two clusters and getting I/O errors. On failed attempts while trying to copy a file, AM tries to re-run and succeeds sometimes on 2nd or 3rd attempt but copy fails completely after 4 attempts due to default setting on AM. What is strange is that there is not particular pattern in failures. Distcp on same files succeeds if retried after some time. Any pointers would be helpful. Using HDP-2.4.2 (Namenode HA) 2016-10-19 17:31:51,011 FATAL [IPC Server handler 14 on 41818] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1476385594016_120501_m_000012_0 - exited : java.io.IOException: File copy failed: hftp://a.b.c.d:50070/user/hive/warehouse/attribution_impsclicks_daily/action=click/wrt_dt=2016-09-25/000089_0 --> hdfs://kandula/user/hive/warehouse/attribution_impsclicks_daily/action=click/wrt_dt=2016-09-25/000089_0
at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:285)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:253)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying hftp://a.b.c.d:50070/user/hive/warehouse/attribution_impsclicks_daily/action=click/wrt_dt=2016-09-25/000089_0 to hdfs://kandula/user/hive/warehouse/attribution_impsclicks_daily/action=click/wrt_dt=2016-09-25/000089_0
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:281)
... 10 more
Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.net.SocketTimeoutException: Read timed out
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.getInputStream(RetriableFileCopyCommand.java:302)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:247)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:183)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:123)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
... View more
Labels:
- Labels:
-
Apache Hadoop