@EricL    Thanks for your assistance. Regards Wert   ... View more

@deekshant    To debug Namenode issue you need to check below -   1. Check active namenode[NN] logs [for time when it got reboot] 2. Check active NN zkfc logs [same time - if you see any issue] 3. Check for standby NN logs at same time if you see any error 4. Check for standby NN zkfc logs for any error at same timestamp 5. Check for Active NN .out file for any warnings/error 6. Check for system logs "/var/log/message" for any issue at particular moment of time.   You will find error in one of the above file. accordingly you can go for RCA. Do revert if you need further help. ... View more

@Timothyw0 , No, you can't update or delete in DF. You have to use filter/transform DF and create a new DF. Cheers Eric ... View more

    Exactly I wanted!   Thank you @EricL  ... View more

Hi @wret_1311, Thanks for your response and I appreciate for confirming the solution. I'm glad, it helped you 🙂 ... View more

Hi @DataMike, Yes, once CM is down, you can't update the configuration unless you "hack" into CM's backend database, which is also down in our scenario. So if you setup MySQL in such a way that CM will connect to LB / VIP host, which will route to the MySQL host that you want to, so in the event of one MySQL instance is down, simply update LB to only go to the working one. This avoids the needs to update the CM config. Hope that can help. Cheers Eric ... View more

thanks, i suppose one could just move that to wherever you want the home to be. this was part of a hack deployment, so ultimately not needed. ... View more

@Plop564 I am not an expert in Spark, but my understand is below: 1. I will have 100 output files >>> this depends how many partitions you have in your original DF. "coalesce" can only reduce number of partitions, so if you have less than 100 partitions before, then it won't do anything, as "coalesce" does not do shuffling. If you want to guarantee number of output files, I believe "repartition" function is better. 2. Each single CSV file is locally sorted, I mean by the "date" column ascending >>> Yes 3. Files are globally sorted, I mean CSV part-0000 have "date" inferior to CSV part-0001, CSV part-0001 have "date" inferior to CSV part-0002 and so on .. >>> I believe it is also Yes, but will wait for other Spark experts to confirm. Cheers Eric ... View more

Solved After Re-Installing Cloudera manager & CDH 5.16 ... View more