data frames and data sets

[#1]

Hello Everyone,

can you please tell me the difference between DataFrames and DataSets (with examples)?

The explanations is still unclear

http://spark.apache.org/docs/2.4.0/sql-programming-guide.html

Thanks,

Roshan

[#2 · Accepted solution]

Hi you can find following tutorial.

https://www.cloudera.com/tutorials/dataframe-and-dataset-examples-in-spark-repl.html

[#3]

I have been working with Oracle databases, in what way is DataFrames and DataSets similar to Oracle? Are they similar to views?

[#4 · Accepted solution]

Hi you can find following tutorial.

https://www.cloudera.com/tutorials/dataframe-and-dataset-examples-in-spark-repl.html

[#5]

Hi @RangaReddy 

 thanks a lot for sharing the link. It will help me a lot.

Can you please advise why we have to include df (data frame name) before each column?

df.select(df("name"), df("age") + 1).show()

 I noticed in groupBy() there is no df.

Grateful if you can clarify this.

Thanks,

Roshan

[#6 · Accepted solution]

Hi @roshanbi 

Apache Spark offers several methods to use when selecting a column.

Scala Spark:

// Scala
import org.apache.spark.sql.functions.{expr, col, column}
// 6 ways to select a column
df.select(df.col("ColumnName"))
df.select(col("ColumnName"))
df.select(column("ColumnName")) 
df.select(`ColumnName)
df.select($"ColumnName")
df.select(expr("ColumnName"))

PySpark:

# Python
from pyspark.sql.functions import expr, col, column
# 4 ways to select a column
df.select(df.ColumnName)
df.select(col("ColumnName"))
df.select(column("ColumnName"))
df.select(expr("ColumnName"))

[#7]

Hi @roshanbi 

If you are satisfied with my answer please Accept as Solution.