Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Incremental Imported records not reflected in HBase using Sqoop

Incremental Imported records not reflected in HBase using Sqoop

New Contributor

Hi All,

Objective:

Automate the process of loading incremental data from SQL server to HBase using Sqoop job and schedule it in Oozie.

I have created a sqoop incremental job

sqoop import --connect "jdbc:sqlserver://x.x.x.x:1433;database=test" --username sa --password qwert --table testing --incremental append --check-column id --last-value 71 -m 1

Sqoop Job

sqoop job --create incjobsnew -- import --connect "jdbc:sqlserver://x.x.x.x:1433;database=test" --username sa --password qwert --table testing --incremental append --check-column id --last-value 71 -m 1

i have create it as a job and when i execute the job, it shows the exact number of records has been retrieved successfully.

But if i check in hbase table records are not reflected. I was unable to find the issue.

Please do the needful

9 REPLIES 9

Re: Incremental Imported records not reflected in HBase using Sqoop

Expert Contributor

Have you first tried a single import into HBase? I don't see the hbase arguments in your commands.

To import into hbase, you need to use arguments —hbase-table, possibly —hbase-row-key and —column-family

https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_importing_data_into_hbase

Then I would try adding the incremental arguments to the command.

Re: Incremental Imported records not reflected in HBase using Sqoop

New Contributor

Hi Bhagan,

I have imported the table into hbase with command in the link give.

I have to do incremental import to an existing hbase table using sqoop job.

Re: Incremental Imported records not reflected in HBase using Sqoop

Expert Contributor

All indications are that the incremental update to hbase should work. Can you provide me with the latest sqoop command that you are using that includes the hbase arguments and the incremental update arguments? Also, then provide a line(s) from the output that shows success. And perhaps show the result of a get from hbase that shows the data is not there?

Re: Incremental Imported records not reflected in HBase using Sqoop

New Contributor

@bhagan.

Hi Bhagan,

This are the steps i followed.

Step 1:

Importing a Table To HBase

sqoop import --connect "jdbc:sqlserver://x.x.x.x:1433;database=test" --username sa -P --table employee --hbase-table employee --hbase-create-table --column-family cf --hbase-row-key id -m 1

Step 2:

SQOOP HBASE INCREMENTAL IMPORT

sqoop import --connect "jdbc:sqlserver://x.x.x.x:1433;database=test" --username sa -P --table employee --incremental append --check-column id --last-value 71 -m 1

Step 3:

SQOOP JOB CREATION FOR HBASE INCREMENT

sqoop job --create incjobsnew -- import --connect "jdbc:sqlserver://x.x.x.x:1433;database=test" --username sa -P --table employee --incremental append --check-column id --last-value 71 -m 1.

When i execute sqoop job

sqoop job --exec incjobsnew.

Sqoop command runs successfully and it show the exact number of records retrieved successfully. When i check in hbase for the records. It doesn't show the retrieved results.

Could you tell where is the mistake done.

I need to automate this sqoop job in Oozie to run a particular time interval daily.

Re: Incremental Imported records not reflected in HBase using Sqoop

Expert Contributor

@Magesh Kumar

Hi Magesh, It looks like you need to add the hbase arguments to your incremental command. By default, Sqoop will import a table named to a directory named inside your home directory in HDFS. For example, if your username is someuser, then the import tool will write to /user/someuser/employee/(files).

Check that directory and see if your incremental updates are appearing there. If so, add the hbase arguments to the incremental command and let me know the results.

Re: Incremental Imported records not reflected in HBase using Sqoop

New Contributor

@bhagan

Hi Bhagan,

I have imported the data to hbase and incremental data has been loading into /user/hadooopuser/employee directory successfully. When i write the select query in hbase using phoenix the incremental records are not getting displayed. But when i see in the HDFS Path (/user/hadooopuser/employee) i was able to view the records latest records.

Highlighted

Re: Incremental Imported records not reflected in HBase using Sqoop

Expert Contributor

Magesh, Ah, Phoenix. It looks like this is a known issue, and it will be addressed in an upcoming release of SQOOP.

https://issues.apache.org/jira/browse/SQOOP-2649

Re: Incremental Imported records not reflected in HBase using Sqoop

New Contributor

Hi bhagan

@bhagan

I have tried the same incremental import in hive and i see the values incremented in the hdfs directory when i run the sqoop job. but when i check it in the hive table Eg) Select * from employee.

The values are not reflected.

Now i need to schedule the sqoop job in order to automate it.

Please do the needful.

Re: Incremental Imported records not reflected in HBase using Sqoop

Expert Contributor

Magesh, I'm not clear. Have you added the hbase options when doing an hbase incremental update? Have you added the hive options in your incremental hive update?