Support Questions

Find answers, ask questions, and share your expertise
Announcements
Welcome to the upgraded Community! Read this blog to see What’s New!

Sqoop time transferred

avatar
Rising Star

Hello guys, I'm importing a table to Hbase via Sqoop import, and i want to do some experimentations and measure the time written, based on some options in Hbase like Number of regions, Max. file size, pre-splitting, others... Can i rely on the number (315.1576 seconds) that i'm showing in this picture:

13871-sqoop-import.png

Thanks

1 ACCEPTED SOLUTION

avatar
Cloudera Employee

You can view the job completion details if u launch <resourcemangerhost>:8080/cluster/apps/FINISHED

look for the respective Mapreduce associated to the sqoop job.

View solution in original post

9 REPLIES 9

avatar
Contributor

You can time your command and compare numbers if you do not trust the number reported by Sqoop MR Import job 🙂

time IMPORT_COMMAND (Linux)

avatar
Rising Star

Thanks for the suggestion @vrodionov.

So, there is no other way to measure the time that sqoop spends to import data to hbase?

avatar
Contributor

What I suggested is to compare both times. If they close enough, than you can rely on both. If there is a significant discrepancy than I would go with Unix timing.

avatar
Cloudera Employee

You can view the job completion details if u launch <resourcemangerhost>:8080/cluster/apps/FINISHED

look for the respective Mapreduce associated to the sqoop job.

avatar
Rising Star
@rkandula

Shouldn't be <resourcemanagerhost>:8088/cluster/apps/FINISHED ?

avatar
Cloudera Employee

sorry typo, 8088 is the port

avatar
Rising Star

yup, i figured out, but thanks @rkandula, i tested it and it worked. But now i have 3 types of different times:

214.51 seconds from shell

real 4m40.985s from Linux command

user 0m19.121s from Linux command

sys 0m5.181s from Linux command

2m50 seconds elapsed time from mapreduce job

What time should i consider about the real time of writing to hbase table?

avatar
Cloudera Employee

i am not sure about unix command,what is the time duration u see in resourcemanager url

avatar
Rising Star

I see elapsed time 2m50 seconds and in the bottom : Aggregate Resource Allocation:53607 MB-seconds, 213 vcore-seconds

Labels