Skip to main content
dbrys
dbrys
Silver ⭐️
May 24, 2023
Solved

Support for external shuffle services

Hi, does anyone knows whether there are any restrictions on using external shuffle services on CDP?

For instance, are the following external shuffle services supported?:

- uber rss

- apache uniffle

Bharati
BharatiAnswer
Gold ⭐️
May 24, 2023

At the moment both uber rss and apache uniffle are not supported in CDP.

Dynamic resource allocation requires an external shuffle service that runs on each worker node as an auxiliary service of NodeManager. This service is started automatically; no further steps are needed. spark.shuffle.service.enabled=true enables the external shuffle service. The external shuffle service preserves shuffle files written by executors so that the executors can be deallocated without losing work. Must be enabled if dynamic allocation is enabled.

View in thread

3 replies

DianaTorres
Community Manager
DianaTorres
Community Manager
May 24, 2023

@dbrys Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Spark experts @Bharati and @jagadeesan  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Translate
Regards, Diana Torres, Senior Community Moderator
    Bharati
    Gold ⭐️
    BharatiAnswer
    Gold ⭐️
    May 24, 2023

    At the moment both uber rss and apache uniffle are not supported in CDP.

    Dynamic resource allocation requires an external shuffle service that runs on each worker node as an auxiliary service of NodeManager. This service is started automatically; no further steps are needed. spark.shuffle.service.enabled=true enables the external shuffle service. The external shuffle service preserves shuffle files written by executors so that the executors can be deallocated without losing work. Must be enabled if dynamic allocation is enabled.

    Translate
      dbrys
      dbrysAuthor
      Silver ⭐️
      May 30, 2023

      thanks for the information, we're running on yarn so I guess the steps mentioned in https://spark.apache.org/docs/3.0.0-preview2/running-on-yarn.html#configuring-the-external-shuffle-service must be executed to configure the external shuffle service on each worker node. Are there any plans to support any other external shuffle service like uber rss or apache uniffle in the future?

      Translate
      Terms & ConditionsAccessibility statement