Skip to main content
Manado
Bronze ⭐️⭐️⭐️
October 11, 2023
Question

Access to —files in driver in spark on k8s

Hi, I’m running spark on k8s and would love to access files that are submitted with —files on the driver (not executor). I know files are uploaded to /tmp/spark-XXXX/ folder that is random, but I can’t find any environment variable that I can use in driver to access them. 

the reason I want this is to pass a config file in —files with some parameters , and I want to access this file before I initialize spark context.

1 reply

VidyaSargur
Community Manager
Community Manager
October 11, 2023

@Manado, Welcome to our community! To help you get the best possible answer, I have tagged in our Spark experts @RangaReddy @Babasaheb who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Translate
Regards, Vidya Sargur, Community Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.