Support Questions
Find answers, ask questions, and share your expertise

Sqoop jobs are failing while loading more data

Re: Sqoop jobs are failing while loading more data

@Nirvana India Please do me a favor and download HDP sandbox. It will save you lot of time and efforts. Also, it will be easier for us to help you.

Re: Sqoop jobs are failing while loading more data

@Nirvana India When you run sqoop job then its displaying the output on screen...We need to see at what point the jobs are failing.

Re: Sqoop jobs are failing while loading more data

@Nirvana India Please add me on linkedin and we can plan a webex.

Re: Sqoop jobs are failing while loading more data

Mentor

@Nirvana India you are most likely killing your SQL server. Do not do that, stress testing a VM wont get you any results you can use in production. Sqoop works well and performance depends on many factors, you are not seeing the whole picture with this POC. You are on the other hand stressing your database server as depending how you've written your sqoop, have 4 mappers by default per job, so you're running 20 mappers at the server. Your jobs are timing out as db server is choking.

Re: Sqoop jobs are failing while loading more data

@Artem Ervits

We are running 4 jobs each has switch -m 2. 2 Jobs has source database server on Server1 and other 2 has Server2.

As i can see in the job bowser, 3 out of the 4 jobs are running and 1 job is queued up. total of 6 mapper are working on 6 processor machine at the same time.

Sql server profiler is showing no incoming queries on the source database. I don't think database instance is choking.

Re: Sqoop jobs are failing while loading more data

Mentor