Spark is an incredible asset for removing information, running changes, and stacking the outcomes in an information store.
Spark runs calculations in equal so execution is extremely quick and groups can be scaled up for enormous information. Spark's local API and spark-daria's Etl Definition object take into consideration rich meanings of ETL rationale.
Extract
Assume you have an information pool of Parquet records. Here's some model code that will bring the information lake, channel the information, and afterward repartition the information subset.
val dataLakeDF = spark.read.parquet(" s3a://some-bucket/foo ") val extractDF = dataLakeDF .where(col("mood") === "happy") .repartition(10000)
Transform
We can characterize a custom change work that accepts a DataFrame as a contention and returns a DataFrame to change the extractDF. Custom change capacities are reusable and effectively testable, so this makes a top notch codebase.
Load
We can utilize the Spark DataFrame essayists to characterize a nonexclusive capacity that composes a DataFrame to a given area in S3.
def exampleWriter()(df: DataFrame): Unit = { val path = "s3a://some-bucket/extracts/bar" df.write.mode(SaveMode.Overwrite).parquet(path) }
Defining ETL
How about we launch the EtlDefinition case class characterized in spark-daria and utilize the procedure() technique to execute the ETL code.
val etl = new EtlDefinition( sourceDF = extractDF, transform = model(), write = exampleWriter() )
Here’s how to execute the ETL code:
etl.process()
Investigate the technique marks of the EtlDefinition contentions and ensure you see how the capacities we've characterized fit into this form.
case class EtlDefinition( sourceDF: DataFrame, transform: (DataFrame => DataFrame), write: (DataFrame => Unit), metadata: scala.collection.mutable.Map[String, Any] = scala.collection.mutable.Map[String, Any]() ) { def process(): Unit = { write(sourceDF.transform(transform)) } }
... View more
