Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

sqoop into more parts with small sizes

New Contributor

I have sqoop stmt with 10 mappers. All the data is going into 10 parts in hadoop with each part exceeding the 1GB. I want to divide the data in multiple files of smaller parts, needless to say more than 10, something like 50 files of 200MB each. However due to DB bottleneck issue, I cant create more than 10 mappers in a sqoop. Let me know if their is any easy solution.

1 REPLY 1

Super Guru
@Pavan Ebbadi

Have you tried --direct-split-size to limit the size of the files written? It takes an argument in bytes so you would give --direct-split-size 200000000