Support Questions
Find answers, ask questions, and share your expertise

How to concatenate date to file in pyspark

How to concatenate date to file in pyspark

Explorer

Hello community,

I have created the following pyspark query:

from pyspark.sql import SparkSession

spark
= SparkSession.builder.appName('ops').getOrCreate()df
=
spark.read.csv('/home/packt/Downloads/Spark_DataFrames/HumanResources_vEmployeeDepartment.csv',inferSchema=True,header=True)df.createOrReplaceTempView('HumanResources_vEmployeeDepartment')counts
= spark.sql("""SELECTFirstName,LastName,JobTitleFROM
HumanResources_vEmployeeDepartmentORDER
BY FirstName, LastName DESC""")counts.coalesce(1).write.csv("/home/packt/Downloads/myresults3.csv") 

I would like to add the current date and time to the file called myresults3.

I think the code would look something like the following:

counts.coalesce(1).write.csvCONCAT("/home/packt/Downloads/'myresults3'-CURRENTDATE.csv")

I'm sure I'm way off the mark with the above attempt, but I'm sure you can see what I'm trying to achieve.

Any help will be appreciated.

Cheers

Carlton

2 REPLIES 2
Highlighted

Re: How to concatenate date to file in pyspark

@Carlton Patterson

You can use the python's datetime package to obtain the current date.

import datetime 
currentdate = datetime.datetime.now().strftime("%Y-%m-%d") 
print currentdate 

2018-08-13

And then use the currentdate in output file name.

counts.coalesce(1).write.csvCONCAT("/home/packt/Downloads/myresults3-" + currentdate + ".csv")

Hope this helps.

P.S. If you want date and time use: datetime.datetime.now().strftime("%Y-%m-%d %H:%M")

Highlighted

Re: How to concatenate date to file in pyspark

As this is duplicate, Please delete/close this question.