My expertise is not in hadoop but rather online communities, support and social media. Interests include: photography, travel, movies and watching sports.
Hi @mrTao It is not good idea to use entire memory Cluster YARN memory. You can tune the memory from spark side by adjusting the memory using following parameters: --conf spark.executor.instances=5
--conf spark.driver.memory=10g
--conf spark.driver.memoryOverhead=1g
--conf spark.executor.memory=10g
--conf spark.executor.memoryOverhead=1g With the above memory configuration, YARN will allocate 66gb(Executor memory 11gb * Executor Instances(5) + Driver memory 11gb = 55g + 11g = 66g). Better check your spark-submit once again and tune the above parameters according to your requirement.
... View more
@Manju1993 Welcome to our community! To help you get the best possible answer, I have tagged in our Spark experts @RangaReddy @Babasaheb who may be able to assist you further. Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.
... View more
Welcome to the community @prabhu456. Hopefully you have already resolved this, but if not, perhaps this older solved question may get you closer.
Cloudera Navigator - Identify unused db, tables, files, folders, etc
... View more
@anoop89 This is an unrelated issue to this original thread. Please start a new question. Fell free to @ckumar and @MattWho in your question so we get notified. This issue is related to authorization of your user. Thanks, Matt
... View more
Welcome to the community @supersonic-2021 . As this is an older post, we recommend starting a new thread. The new thread will provide the opportunity to provide details specific to your environment that could aid others in providing a more accurate answer to your question.
... View more
@jagan20, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.
... View more
id, name, balance, join_date, notes 1, John, 48.23, 04/03/2007 "Our very first customer!" 2, Jane, 1245.89, 08/22/2009, 3, Frank Franklin, "48481.29", 04/04/2016, the logic may fail in the above case, if column contains multiple rows.
... View more
