Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to concatenate date to file in pyspark

How to concatenate date to file in pyspark

Explorer

Hello community, I have created the following pyspark query:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('ops').getOrCreate()
df = spark.read.csv('/home/packt/Downloads/Spark_DataFrames/HumanResources_vEmployeeDepartment.csv',inferSchema=True,header=True)
df.createOrReplaceTempView('HumanResources_vEmployeeDepartment')
counts = spark.sql("""SELECT
FirstName
,LastName
,JobTitle
FROM HumanResources_vEmployeeDepartment
ORDER BY FirstName, LastName DESC""")
counts.coalesce(1).write.csv("/home/packt/Downloads/myresults3.csv") 

I would like to add the current date and time to the file called myresults3. I think the code would look something like the following:

counts.coalesce(1).write.csvCONCAT("/home/packt/Downloads/'myresults3'-CURRENTDATE.csv")

I'm sure I'm way off the mark with the above attempt, but I'm sure you can see what I'm trying to achieve. Any help will be appreciated.

Cheers Carlton

2 REPLIES 2
Highlighted

Re: How to concatenate date to file in pyspark

@Carlton Patterson

You can use the python's datetime package to obtain the current date.

import datetime 
currentdate = datetime.datetime.now().strftime("%Y-%m-%d") 
print currentdate 

2018-08-13

And then use the currentdate in output file name.

counts.coalesce(1).write.csvCONCAT("/home/packt/Downloads/myresults3-" + currentdate + ".csv")

Hope this helps.

P.S. If you want date and time use: datetime.datetime.now().strftime("%Y-%m-%d %H:%M")

Highlighted

Re: How to concatenate date to file in pyspark

As this is duplicate, Please delete/close this question.

Don't have an account?
Coming from Hortonworks? Activate your account here