Reply
Explorer
Posts: 21
Registered: ‎04-15-2014

HBASE_HOME for sqoop import

Hello,

 

I'm using CDH 5.0.1


i'm trying to import data into hbase using sqoop.
i launch my sqoop line-command:

 

sudo -u hdfs  sqoop import --connect jdbc:mysql://10.0.0.221/db --username XXX --password XXX --table test -m 1 --target-dir /user/import  --incremental lastmodified --check-column date --append --hbase-table forHive --column-family infos

 



During the log, i have this error:
Error during import: HBase jars are not present in classpath, cannot import to HBase!

I set my $HBASE_HOME : export HBASE_HOME=/usr/lib/hbase
after that i change my hbase_home because i use CDH with this string:
export HBASE_HOME=/opt/cloudera/parcels/CDH/lib/hbase
and i also try:
export HBASE_HOME=/opt/cloudera/parcels/CDH/lib/hbase/lib

But in every case i have this error:
 ERROR tool.ImportTool: Error during import: HBase jars are not present in classpath, cannot import to HBase!

What kind of HBASE_HOME must i set?

and is possible use builk-load HBase with sqoop for massive import?

thanks

Posts: 1,885
Kudos: 424
Solutions: 299
Registered: ‎07-31-2013

Re: HBASE_HOME for sqoop import

How are you setting your environment exactly? When you use a command like "sudo -u hdfs cmd…" the local environment variables you have set in shell will not transfer through to the sudo-based execution (different shell).

Try logging in as 'hdfs' first, via 'su hdfs' or 'sudo -u hdfs /bin/bash', then export the HBASE_HOME to the parcel or package path and retry your Sqoop command execution.

Also read http://stackoverflow.com/questions/8633461/how-to-keep-environment-variables-when-using-sudo
New Contributor
Posts: 2
Registered: ‎01-03-2019

Re: HBASE_HOME for sqoop import

I also have the same issue in CDH 5.13, do you have the solution now? thanks.
Highlighted
Champion
Posts: 768
Registered: ‎05-16-2016

Re: HBASE_HOME for sqoop import

I have not tested ? but would you consider deploying a  Hbase gateway role on the host you are ruining ? 

New Contributor
Posts: 2
Registered: ‎01-03-2019

Re: HBASE_HOME for sqoop import

HBase service have been added before. After I add /usr/lib/hbase/lib/* in YARN Application Classpath, the error gone, but dispaly a new new error:

2019-01-04 17:31:27,834 [main] INFO org.apache.sqoop.orm.CompilationManager - HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/hadoop-mapreduce
2019-01-04 17:31:31,952 [main] ERROR org.apache.sqoop.tool.ImportTool - Import failed: java.io.IOException: Error returned by javac
at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:222)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:494)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:187)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:170)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:81)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Announcements
New solutions