Support Questions

Find answers, ask questions, and share your expertise

Any guidelines for disk space allocation for tez job Spills?

avatar

What is the disk space requirement for job spills. Is there any guideline for the same? Or is it just Job dependent - if so, how to determine the same?

1 ACCEPTED SOLUTION

avatar
Super Collaborator

In general I configure disk space allocation for tez job spills the same way as Yarn intermediate data.

Please find here some discussions regarding how to configure it:

http://community.hortonworks.com/questions/2230/recommended-size-for-yarnnodemanagerresourcelocal.ht...

http://community.hortonworks.com/questions/1405/can-you-please-advise-about-how-best-to-use-this-s.h...

View solution in original post

1 REPLY 1

avatar
Super Collaborator

In general I configure disk space allocation for tez job spills the same way as Yarn intermediate data.

Please find here some discussions regarding how to configure it:

http://community.hortonworks.com/questions/2230/recommended-size-for-yarnnodemanagerresourcelocal.ht...

http://community.hortonworks.com/questions/1405/can-you-please-advise-about-how-best-to-use-this-s.h...