Member since
10-10-2023
1
Post
0
Kudos Received
0
Solutions
10-10-2023
05:33 PM
Hi, I’m running spark on k8s and would love to access files that are submitted with —files on the driver (not executor). I know files are uploaded to /tmp/spark-XXXX/ folder that is random, but I can’t find any environment variable that I can use in driver to access them. the reason I want this is to pass a config file in —files with some parameters , and I want to access this file before I initialize spark context.
... View more
Labels:
- Labels:
-
Apache Spark