Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive export encrypted table

Solved Go to solution

Hive export encrypted table

New Contributor

Trying to export a hive table from an encrypted HDFS.

The distcp portion of the export fails. Every file segment throws one of these...

2020-03-26 14:36:17,124 ERROR [IPC Server handler 25 on 37768] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1585157114387_3676_m_000000_0 - exited : java.io.IOException: File copy failed: hdfs://myserver/warehouse/tablespace/tmp/hive/e18e8f44-9d0e-4f10-9490-2703bbdc84e9/_tmp_space.db/7579e917-2de7-4923-887a-df739f3a98e1/000006_0 --> hdfs://myserver/warehouse/tablespace/hive-export/username/tablename/data/000006_0
	at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:263)
	at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:220)
	at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://myserver/warehouse/tablespace/tmp/hive/e18e8f44-9d0e-4f10-9490-2703bbdc84e9/_tmp_space.db/7579e917-2de7-4923-887a-df739f3a98e1/000006_0 to hdfs://myserver/warehouse/tablespace/hive-export/username/tablename/data/000006_0
	at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
	at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:259)
	... 10 more
Caused by: java.io.IOException: Checksum mismatch between hdfs://myserver/warehouse/tablespace/tmp/hive/e18e8f44-9d0e-4f10-9490-2703bbdc84e9/_tmp_space.db/7579e917-2de7-4923-887a-df739f3a98e1/000006_0 and hdfs://myserver/warehouse/tablespace/hive-export/username/tablename/data/.distcp.tmp.attempt_1585157114387_3676_m_000000_0.
	at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:261)
	at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:153)
	at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
	at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
	... 11 more

 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Hive export encrypted table

New Contributor

hive > set distcp.options.skipcrccheck=;

hive > export ...blah, blah blah;

success!

Thanks to Cloudera Support, M. Green.  Super fast!

View solution in original post

2 REPLIES 2
Highlighted

Re: Hive export encrypted table

New Contributor

Should have mentioned: this is my export command (beeline)

0: jdbc:hive2://myhiveserver,d> export table mydatabase.mytable to '/warehouse/tablespace/hive-export/username/tablename';

 

The HDFS encryption zone is on /warehouse/tablespace

Highlighted

Re: Hive export encrypted table

New Contributor

hive > set distcp.options.skipcrccheck=;

hive > export ...blah, blah blah;

success!

Thanks to Cloudera Support, M. Green.  Super fast!

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here