question Re: data frames and data sets in Support Questions

data frames and data sets

roshanbi — Wed, 23 Jun 2021 14:56:05 GMT

Hello Everyone,

can you please tell me the difference between DataFrames and DataSets (with examples)?

The explanations is still unclear

http://spark.apache.org/docs/2.4.0/sql-programming-guide.html

Thanks,

Roshan

Re: data frames and data sets

roshanbi — Wed, 23 Jun 2021 15:04:53 GMT

I have been working with Oracle databases, in what way is DataFrames and DataSets similar to Oracle? Are they similar to views?

Re: data frames and data sets

RangaReddy — Thu, 24 Jun 2021 11:25:20 GMT

Hi you can find following tutorial.

https://www.cloudera.com/tutorials/dataframe-and-dataset-examples-in-spark-repl.html

Re: data frames and data sets

roshanbi — Thu, 24 Jun 2021 13:43:50 GMT

Hi @RangaReddy

thanks a lot for sharing the link. It will help me a lot.

Can you please advise why we have to include df (data frame name) before each column?

df.select(df("name"), df("age") + 1).show()

I noticed in groupBy() there is no df.

Grateful if you can clarify this.

Thanks,

Roshan

Re: data frames and data sets

RangaReddy — Thu, 24 Jun 2021 15:21:39 GMT

Hi @roshanbi

Apache Spark offers several methods to use when selecting a column.

Scala Spark:

// Scala import org.apache.spark.sql.functions.{expr, col, column} // 6 ways to select a column df.select(df.col("ColumnName")) df.select(col("ColumnName")) df.select(column("ColumnName")) df.select(`ColumnName) df.select($"ColumnName") df.select(expr("ColumnName"))

PySpark:

# Python from pyspark.sql.functions import expr, col, column # 4 ways to select a column df.select(df.ColumnName) df.select(col("ColumnName")) df.select(column("ColumnName")) df.select(expr("ColumnName"))

Re: data frames and data sets

RangaReddy — Thu, 24 Jun 2021 15:23:24 GMT

Hi @roshanbi

If you are satisfied with my answer please Accept as Solution.