About manikandanjeyab

manikandanjeyab · ‎11-27-2018

Hi @Felix Albani, Tanx for your response. I reffered this site i'm thinking my issues is related to the story. Cheers, MJ

jsensharma · ‎09-17-2018

@Manikandan Jeyabal It is hard to provide the exact values based on the data size. However you can refer to the following article to understand Executor Memory/Core/Resource Optimization. https://community.hortonworks.com/articles/42803/spark-on-yarn-executor-resource-allocation-optimiz.html https://dzone.com/articles/apache-spark-on-yarn-resource-planning

edu_vikassri · ‎09-17-2018

Hi @vgarg, Since 0.11 hive has a NVL function nvl(T value, T default_value) which says Returns default value if value is null else returns value

manikandanjeyab · ‎08-27-2018

Hi Issue got resolved, i'm trying to perform Group by operation inside a Columns literal, group by itself will produce a new columns instead writing a query like i asked above we have to change our query accordingly as follow. inputDf = inputDf.groupBy(col("tableName"),col("runDate")) .agg(sum(when(col("MinBusinessDate") < col("runDate") && col("MaxBusinessDate") > col("runDate"), when(col("business_date") > col("runDate"), col("rowCount")))).alias("NewColumnName"))

manikandanjeyab · ‎08-27-2018

Hi Issue got resolved, i'm trying to perform Group by operation inside a Columns literal, group by itself will produce a new columns instead writing a query like i asked above we have to change our query accordingly as follow. inputDf = inputDf.groupBy(col("tableName"),col("runDate")) .agg(sum(when(col("MinBusinessDate")< col("runDate")&& col("MaxBusinessDate")> col("runDate"), when(col("business_date")> col("runDate"), col("rowCount")))).alias("NewColumnName"))

manikandanjeyab · ‎08-25-2018

Hi @Felix Albani yes your suggestion works fine, i think i have to extend my object as app to make it work. Tanx Bro.

manikandanjeyab · ‎08-24-2018

Hi Aditya, Tanx for your reply, is there any other posts or blog can i find how to implement the schema evolution in ORC ?

manikandanjeyab · ‎07-09-2018

Hi @Sandeep Nemuri, I think i found the answer from Stackoverflow site. Tanx for you info.

manikandanjeyab · ‎07-09-2018

Tanx for your info @Felix Albani

manikandanjeyab · ‎06-13-2018

It's already installed and the issue resolved now, tanx for you response.

Online	Offline
Last Visited	‎09-18-2019 09:50 AM

Member Since	‎03-26-2017 12:39 PM
Last Visited	‎09-18-2019 09:50 AM
Posts	61
Kudos received	1

Cloudera Community

Re: spark scala Dataframe adding new column Error

Re: Unsupported literal type class in Apache Spark...

Re: Spark SQL Error connecting Hive Metastore

Re: Spark throws "Invalid Sync" Error when trying ...

Re: how to configure spark submit configurations b...

Re: Hive set Null value to Empty string

Re: Unsupported literal type class in Apache Spark...

Re: spark scala Dataframe adding new column Error

Re: spark + scala schema creattion error

Re: How to handle Schema Evolution in ORC file fo...

Re: How to take record count when writing as ORC f...

Re: How to Read Files from Remote Hadoop HA cluste...

Re: Error When Running spark Submit: "native snapp...