- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Access to —files in driver in spark on k8s
- Labels:
-
Apache Spark
Created on 10-10-2023 05:33 PM - edited 10-10-2023 06:07 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi, I’m running spark on k8s and would love to access files that are submitted with —files on the driver (not executor). I know files are uploaded to /tmp/spark-XXXX/ folder that is random, but I can’t find any environment variable that I can use in driver to access them.
the reason I want this is to pass a config file in —files with some parameters , and I want to access this file before I initialize spark context.
Created 10-10-2023 09:56 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Manado, Welcome to our community! To help you get the best possible answer, I have tagged in our Spark experts @RangaReddy @Babasaheb who may be able to assist you further.
Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.
Regards,
Vidya Sargur,Community Manager
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:
