Spark is an incredible asset for removing information, running changes, and stacking the outcomes in an information store. Spark runs calculations in equal so execution is extremely quick and groups can be scaled up for enormous information. Spark's local API and spark-daria's Etl Definition object take into consideration rich meanings of ETL rationale. Extract Assume you have an information pool of Parquet records. Here's some model code that will bring the information lake, channel the information, and afterward repartition the information subset.   val dataLakeDF = spark.read.parquet(" s3a://some-bucket/foo ") val extractDF = dataLakeDF .where(col("mood") === "happy") .repartition(10000) Transform We can characterize a custom change work that accepts a DataFrame as a contention and returns a DataFrame to change the extractDF. Custom change capacities are reusable and effectively testable, so this makes a top notch codebase.   Load We can utilize the Spark DataFrame essayists to characterize a nonexclusive capacity that composes a DataFrame to a given area in S3. def exampleWriter()(df: DataFrame): Unit = { val path = "s3a://some-bucket/extracts/bar" df.write.mode(SaveMode.Overwrite).parquet(path) } Defining ETL How about we launch the EtlDefinition case class characterized in spark-daria and utilize the procedure() technique to execute the ETL code.   val etl = new EtlDefinition( sourceDF = extractDF, transform = model(), write = exampleWriter() )   Here’s how to execute the ETL code: etl.process()   Investigate the technique marks of the EtlDefinition contentions and ensure you see how the capacities we've characterized fit into this form.   case class EtlDefinition( sourceDF: DataFrame, transform: (DataFrame => DataFrame), write: (DataFrame => Unit), metadata: scala.collection.mutable.Map[String, Any] = scala.collection.mutable.Map[String, Any]() ) { def process(): Unit = { write(sourceDF.transform(transform)) } }     ... View more

NiFi is one option to accomplish what you need. You can find an  example here for a generic SQL database moving data "in real-time" to Hive.  If you could describe your use case in more details, the community could assist you better.   ... View more

Here we have listed a few ETL tools both, traditional and Open source you can have a look at them and see for yourself which one suits your use case.   1. Panoply: Panoply is the main cloud ETL supplier and data warehouse blend. With 100+ data connectors, ETL and data ingestion is quick and simple, with only a couple of snaps and a login among you and your recently coordinated data. In the engine, Panoply is really utilizing an ELT approach (instead of conventional ETL), which makes data ingestion a lot quicker and progressively powerful, since you don't need to trust that change will finish before stacking your data. What's more, since Panoply fabricates oversaw cloud data warehouses for each client, you won't have to set up a different goal to store all the data you pull in utilizing Panoply's ELT procedure. On the off chance that you'd preferably utilize Panoply's rich arrangement of data gatherers to set up ETL pipelines into a current data warehouse, Panoply can likewise oversee ETL forms for your Azure SQL Data Warehouse. 2. Stitch: Stitch is a self-administration ETL data pipeline. The Stitch API can reproduce data from any source, and handle mass and gradual data refreshes. Stitch additionally gives a replication motor that depends on various techniques to convey data to clients. Its REST API underpins JSON or travel, which empowers programmed recognition and standardization of settled report structures into social constructions. Stitch can associate with Amazon Redshift engineering, Google BigQuery design, and Postgres design - and incorporates with BI apparatuses. Stitch is normally intended to gather, change and burden Google examination data into its own framework, to naturally give business bits of knowledge on crude data. 3. Sprinkle:  Sprinkle is a SaaS platform providing ETL tool for organisations.Their easy to use UX and code free mode of operations makes it easy for technical and non technical users to ingest data from multiple data sources and drive real time insights on the data. Their Free Trial enables users to first try the platform and then pay if it fulfils the requirement.   Some of the open source tools include   1. Heka: Heka is an open source programming framework for elite data gathering, investigation, observing and detailing. Its principle part is a daemon program known as 'hekad' that empowers the usefulness of social occasion, changing over, assessing, preparing and conveying data. Heka is written in the 'Go' programming language, and has worked in modules for contributing, disentangling, separating, encoding and yielding data. These modules have various functionalities and can be utilized together to assemble a total pipeline. Heka utilizes Advanced Message Queuing Protocol (AMQP) or TCP to transport data starting with one area then onto the next. It tends to be utilized to stack and parse log records from a document framework, or to perform constant investigation, charting and inconsistency recognition on a data stream. 2. Logstash: Logstash is an open source data handling pipeline that ingests data from numerous sources at the same time, changing the source data and store occasions into ElasticSearch as a matter of course. Logstash is a piece of an ELK stack. The E represents Elasticsearch, a JSON-based hunt and investigation motor, and the K represents Kibana, which empowers data perception. Logstash is written in Ruby and gives a JSON-like structure which has a reasonable division between inner items. It has a pluggable structure highlighting more than 200 modules, empowering the capacity to blend, coordinate and arrange offices over various information, channels and yield. This instrument can be utilized for BI, or in data warehouses with bring, change and putting away occasion capacities. 3. Singer: Singer's open source, order line ETL instrument permits clients to assemble measured ETL pipelines utilizing its "tap" and "target" modules. Rather than building a solitary, static ETL pipeline, Singer gives a spine that permits clients to interface data sources to capacity goals. With a huge assortment of pre-constructed taps, the contents that gather datapoints from their unique sources, and a broad choice of pre-fabricated focuses on, the contents that change and burden data into pre-determined goals, Singer permits clients to compose succinct, single-line ETL forms that can be adjusted on the fly by trading taps and focuses in and out. ... View more

@srijitachaturve Based on my experience of evaluating ETL tools, I have found the below tools to be helpful while joining data across tables. Panoply: Panoply is the main cloud ETL supplier and data warehouse blend. With 100+ data connectors, ETL and data ingestion is quick and simple, with only a couple of snaps and a login among you and your recently coordinated data. In the engine, Panoply is really utilizing an ELT approach (as opposed to customary ETL), which makes data ingestion a lot quicker and increasingly powerful, since you don't need to trust that change will finish before stacking your data. What's more, since Panoply constructs oversaw cloud data warehouses for each client, you won't have to set up a different goal to store all the data you pull in utilizing Panoply's ELT procedure. On the off chance that you'd preferably utilize Panoply's rich arrangement of data authorities to set up ETL pipelines into a current data warehouse, Panoply can likewise oversee ETL forms for your Azure SQL Data Warehouse. Atom: Atom, from ironSource, is a data pipeline platform that permits data gushing in close to ongoing, into a data warehouse. Atom empowers data stream customization, in view of necessities that help oversee data all the more proficiently. Atom's change code is written in Python, which helps transform crude logs into queryable fields and bits of knowledge. It gives an assortment layer, which supports sending data from any source and in any configuration to show up to the objective data storehouse close to constant. Atom likewise has delay and play alternatives. A solid resumption of data stream without losing a solitary occasion is a significant capacity of Atom, as far as keeping up data trustworthiness. Sprinkle Data:  Sprinkle is a SaaS platform providing ETL tool for organisations.Their easy to use UX and code free mode of operations makes it easy for technical and non technical users to ingest data from multiple data sources and drive real time insights on the data. Their Free Trial enables users to first try the platform and then pay if it fulfils the requirement. ... View more

Hi @venkatgalikm,   Based on your requirements, I have listed a few tools that suits your business requirements.   Stitch: Stitch is a self-administration ETL data pipeline arrangement worked for engineers. The Stitch API can reproduce data from any source, and handle mass and steady data refreshes. Stitch likewise gives a replication motor that depends on various methodologies to convey data to clients. Its REST API bolsters JSON or travel, which empowers programmed identification and standardization of settled record structures into social constructions. Stitch can interface with Amazon Redshift design, Google BigQuery engineering, and Postgres engineering - and incorporates with BI tools. Stitch is regularly intended to gather, change and burden Google examination data into its own framework, to consequently give business bits of knowledge on crude data. Alooma: Alooma offers a venture scale data reconciliation stage with extraordinary ETL tools worked in. The organization puts a solid spotlight on fast pipeline development, data quality checking and blunder dealing with to guarantee that clients don't lose or degenerate data in a possibly mistake inclined ETL process, however it likewise offers the adaptability to mediate and compose your own contents to screen, clean and move your data varying. As referenced, Alooma is intended for big business scale activities, so in case you're a little startup with a little working financial plan, Alooma most likely isn't for you.  ETL Leap: Based on AWS engineering, etleap makes it simple to gather data from a wide scope of sources and burden them into your Redshift or Snowflake data stockroom. Its point-and-snap, no code interface makes it a solid match for data groups that need a great deal of authority over their ETL forms, however don't really need high IT overhead. Since it's coordinated with AWS, etleap additionally makes it simple to scale your data distribution center all over with the equivalent simple to-utilize interface, while simultaneously dealing with your ETL streams on the fly. When data has been gathered utilizing one or a considerable lot of its 50+ data mixes, clients can likewise exploit etleap's graphical data fighting interface or fire up the SQL editorial manager for data displaying and change. Organization and booking highlights make dealing with all your ETL pipelines and procedures as simple as the snap of a catch. Notwithstanding its SaaS offering, etleap additionally gives an adaptation that can be facilitated all alone VPC. Blendo: Blendo offers a cloud-put together ETL tool centered with respect to letting clients get their data into distribution centers as fast as conceivable utilizing their set-up of exclusive data connectors. Blendo's ETL-as-an administration item makes it simple to pull data in from a wide range of data sources including S3 cans, CSVs, and an enormous cluster of outsider data sources like Google Analytics, Mailchimp, Salesforce and numerous others. When you've set up the approaching finish of the data pipeline, you can stack it into various diverse stockpiling goals, including Redshift, BigQuery, MS SQL Server, Panoply and Snowflake. Sprinkle Data:  Sprinkle is a SaaS platform providing ETL tool for organisations.Their easy to use UX and code free mode of operations makes it easy for technical and non technical users to ingest data from multiple data sources and drive real time insights on the data. Their Free Trial enables users to first try the platform and then pay if it fulfils the requirement. ... View more

Please find below list of a few leading ETL Tools   Matillion: Matillion's ETL tool is, as indicated by its designers, reason worked for cloud data distribution centers, so it could be an especially solid decision for clients who are particularly keen on stacking data into Amazon Redshift, Google BigQuery or Snowflake. With more than 70 local data source incorporations, just as a discretionary no-code graphical interface, Matillion makes stacking your data into your distribution center of decision basic and clear. It likewise robotizes the data changes you'll require so as to prepare your data for examination with your preferred BI tool. Matillion is charged hourly for use, so it could likewise be especially appealing for those with a ton of ETL personal time. Talend: Talend open source data combination programming items give programming to coordinate, purge, cover and profile data. Talend has a GUI that empowers dealing with countless source frameworks utilizing standard connectors. It additionally has Master Data Management (MDM) usefulness, which permits associations to have a solitary, predictable and exact perspective on key venture data. This can make better straightforwardness over a business, and lead to better operational proficiency, promoting viability and consistence. Fivetran: Fivetran is a completely overseen data pipeline with a web interface that incorporates data from SaaS administrations and databases into a solitary data stockroom. It gives direct reconciliation and sends data over a direct secure association utilizing a modern reserving layer. This reserving layer assists with moving data starting with one point then onto the next while never putting away a duplicate on the application server. Fivetran doesn't force any data limit, and can be utilized to bring together an organization's data and incorporate all sources to decide Key Performance Indicators (KPIs) over a whole association. Stitch: Stitch is a self-administration ETL data pipeline arrangement worked for engineers. The Stitch API can recreate data from any source, and handle mass and steady data refreshes. Stitch additionally gives a replication motor that depends on numerous methodologies to convey data to clients. Its REST API underpins JSON or travel, which empowers programmed identification and standardization of settled record structures into social outlines. Stitch can associate with Amazon Redshift design, Google BigQuery engineering, and Postgres design - and incorporates with BI tools. Stitch is ordinarily intended to gather, change and burden Google examination data into its own framework, to naturally give business bits of knowledge on crude data. Sprinkle Data:  Sprinkle is a SaaS platform providing ETL tool for organisations.Their easy to use UX and code free mode of operations makes it easy for technical and non technical users to ingest data from multiple data sources and drive real time insights on the data. Their Free Trial enables users to first try the platform and then pay if it fulfils the requirement. ... View more