Support Questions

Find answers, ask questions, and share your expertise

Sqoop time transferred

avatar
Rising Star

Hello guys, I'm importing a table to Hbase via Sqoop import, and i want to do some experimentations and measure the time written, based on some options in Hbase like Number of regions, Max. file size, pre-splitting, others... Can i rely on the number (315.1576 seconds) that i'm showing in this picture:

13871-sqoop-import.png

Thanks

1 ACCEPTED SOLUTION

avatar
Rising Star

You can view the job completion details if u launch <resourcemangerhost>:8080/cluster/apps/FINISHED

look for the respective Mapreduce associated to the sqoop job.

View solution in original post

9 REPLIES 9

avatar
Rising Star

You can time your command and compare numbers if you do not trust the number reported by Sqoop MR Import job 🙂

time IMPORT_COMMAND (Linux)

avatar
Rising Star

Thanks for the suggestion @vrodionov.

So, there is no other way to measure the time that sqoop spends to import data to hbase?

avatar
Rising Star

What I suggested is to compare both times. If they close enough, than you can rely on both. If there is a significant discrepancy than I would go with Unix timing.

avatar
Rising Star

You can view the job completion details if u launch <resourcemangerhost>:8080/cluster/apps/FINISHED

look for the respective Mapreduce associated to the sqoop job.

avatar
Rising Star
@rkandula

Shouldn't be <resourcemanagerhost>:8088/cluster/apps/FINISHED ?

avatar
Rising Star

sorry typo, 8088 is the port

avatar
Rising Star

yup, i figured out, but thanks @rkandula, i tested it and it worked. But now i have 3 types of different times:

214.51 seconds from shell

real 4m40.985s from Linux command

user 0m19.121s from Linux command

sys 0m5.181s from Linux command

2m50 seconds elapsed time from mapreduce job

What time should i consider about the real time of writing to hbase table?

avatar
Rising Star

i am not sure about unix command,what is the time duration u see in resourcemanager url

avatar
Rising Star

I see elapsed time 2m50 seconds and in the bottom : Aggregate Resource Allocation:53607 MB-seconds, 213 vcore-seconds