@Plop564 I am not an expert in Spark, but my understand is below: 1. I will have 100 output files >>> this depends how many partitions you have in your original DF. "coalesce" can only reduce number of partitions, so if you have less than 100 partitions before, then it won't do anything, as "coalesce" does not do shuffling. If you want to guarantee number of output files, I believe "repartition" function is better. 2. Each single CSV file is locally sorted, I mean by the "date" column ascending >>> Yes 3. Files are globally sorted, I mean CSV part-0000 have "date" inferior to CSV part-0001, CSV part-0001 have "date" inferior to CSV part-0002 and so on .. >>> I believe it is also Yes, but will wait for other Spark experts to confirm. Cheers Eric ... View more

Good luck @pramana! ... View more

@Mekaam, Glad that it helped. Cheers Eric ... View more

@gimp077 , Did you mean that "REFRESH" takes time, and eventually you can see the update data, but just some delay? How big is the table? I mean in terms of number of partitions and number of files in HDFS? Eric ... View more

@Mekaam, Can you please add quotes around the JDBC connection string? So like below: beeline -u 'jdbc:hive2://slave1:10000;ssl=true;sslTrustStore=/var/run/cloudera-scm-agent/process/72-hive-HIVESERVER2/cm-auto-host_keystore.jks;trustStorePassword=yeap4IhJzRvK5gBGVMeTahoL21BNmBF2TSi46pbQTP6' I believe without quotes it will cause issues. If still not working, check HS2 log to see what it complains on the server side. Cheers Eric ... View more