Support Questions
Find answers, ask questions, and share your expertise

How to concatenate date to file in pyspark

Explorer

Hello community, I have created the following pyspark query:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('ops').getOrCreate()
df = spark.read.csv('/home/packt/Downloads/Spark_DataFrames/HumanResources_vEmployeeDepartment.csv',inferSchema=True,header=True)
df.createOrReplaceTempView('HumanResources_vEmployeeDepartment')
counts = spark.sql("""SELECT
FirstName
,LastName
,JobTitle
FROM HumanResources_vEmployeeDepartment
ORDER BY FirstName, LastName DESC""")
counts.coalesce(1).write.csv("/home/packt/Downloads/myresults3.csv") 

I would like to add the current date and time to the file called myresults3. I think the code would look something like the following:

counts.coalesce(1).write.csvCONCAT("/home/packt/Downloads/'myresults3'-CURRENTDATE.csv")

I'm sure I'm way off the mark with the above attempt, but I'm sure you can see what I'm trying to achieve. Any help will be appreciated.

Cheers Carlton

2 REPLIES 2

@Carlton Patterson

You can use the python's datetime package to obtain the current date.

import datetime 
currentdate = datetime.datetime.now().strftime("%Y-%m-%d") 
print currentdate 

2018-08-13

And then use the currentdate in output file name.

counts.coalesce(1).write.csvCONCAT("/home/packt/Downloads/myresults3-" + currentdate + ".csv")

Hope this helps.

P.S. If you want date and time use: datetime.datetime.now().strftime("%Y-%m-%d %H:%M")

As this is duplicate, Please delete/close this question.

; ;