I have created the following pyspark query:
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('ops').getOrCreate()df = spark.read.csv('/home/packt/Downloads/Spark_DataFrames/HumanResources_vEmployeeDepartment.csv',inferSchema=True,header=True)df.createOrReplaceTempView('HumanResources_vEmployeeDepartment')counts = spark.sql("""SELECTFirstName,LastName,JobTitleFROM HumanResources_vEmployeeDepartmentORDER BY FirstName, LastName DESC""")counts.coalesce(1).write.csv("/home/packt/Downloads/myresults3.csv")
I would like to add the current date and time to the file called myresults3.
I think the code would look something like the following:
I'm sure I'm way off the mark with the above attempt, but I'm sure you can see what I'm trying to achieve.
Any help will be appreciated.
You can use the python's datetime package to obtain the current date.
import datetime currentdate = datetime.datetime.now().strftime("%Y-%m-%d") print currentdate 2018-08-13
And then use the currentdate in output file name.
counts.coalesce(1).write.csvCONCAT("/home/packt/Downloads/myresults3-" + currentdate + ".csv")
Hope this helps.
P.S. If you want date and time use: datetime.datetime.now().strftime("%Y-%m-%d %H:%M")