question Sqoop time transferred in Archives of Support Questions (Read Only)

Sqoop time transferred

fmm_pires — Sun, 18 Aug 2019 10:56:35 GMT

Hello guys, I'm importing a table to Hbase via Sqoop import, and i want to do some experimentations and measure the time written, based on some options in Hbase like Number of regions, Max. file size, pre-splitting, others... Can i rely on the number (315.1576 seconds) that i'm showing in this picture:

Thanks

Re: Sqoop time transferred

vrodionov — Wed, 22 Mar 2017 05:09:25 GMT

You can time your command and compare numbers if you do not trust the number reported by Sqoop MR Import job 🙂

time IMPORT_COMMAND (Linux)

Re: Sqoop time transferred

fmm_pires — Wed, 22 Mar 2017 05:34:58 GMT

Thanks for the suggestion @vrodionov.

So, there is no other way to measure the time that sqoop spends to import data to hbase?

Re: Sqoop time transferred

vrodionov — Wed, 22 Mar 2017 06:26:05 GMT

What I suggested is to compare both times. If they close enough, than you can rely on both. If there is a significant discrepancy than I would go with Unix timing.

Re: Sqoop time transferred

rkandula — Fri, 24 Mar 2017 01:11:41 GMT

You can view the job completion details if u launch <resourcemangerhost>:8080/cluster/apps/FINISHED

look for the respective Mapreduce associated to the sqoop job.

Re: Sqoop time transferred

fmm_pires — Fri, 24 Mar 2017 03:55:45 GMT

@rkandula

Shouldn't be <resourcemanagerhost>:8088/cluster/apps/FINISHED ?

Re: Sqoop time transferred

rkandula — Fri, 24 Mar 2017 03:58:55 GMT

sorry typo, 8088 is the port

Re: Sqoop time transferred

fmm_pires — Fri, 24 Mar 2017 04:29:07 GMT

yup, i figured out, but thanks @rkandula, i tested it and it worked. But now i have 3 types of different times:

214.51 seconds from shell

real 4m40.985s from Linux command

user 0m19.121s from Linux command

sys 0m5.181s from Linux command

2m50 seconds elapsed time from mapreduce job

What time should i consider about the real time of writing to hbase table?

Re: Sqoop time transferred

rkandula — Fri, 24 Mar 2017 04:34:30 GMT

i am not sure about unix command,what is the time duration u see in resourcemanager url

Re: Sqoop time transferred

fmm_pires — Fri, 24 Mar 2017 04:37:12 GMT

I see elapsed time 2m50 seconds and in the bottom : Aggregate Resource Allocation:53607 MB-seconds, 213 vcore-seconds