- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
How does spark support connecting to two different databases in a single app
- Labels:
-
Apache Spark
Created ‎10-01-2018 04:40 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm thinking to connect spark from MySQL to Postgres in one single app could anyone help me how to do it I have tried adding jars of both the DB's in the classpath but either of the one works, please shed light or guide me on this
Created ‎10-02-2018 06:03 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You can create 2 different dataframes with 2 different jdbc sources(one with mysql and another one with postgres) and make sure both the jdbc jars are present in spark's classpath.
Created ‎10-02-2018 06:03 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You can create 2 different dataframes with 2 different jdbc sources(one with mysql and another one with postgres) and make sure both the jdbc jars are present in spark's classpath.
Created ‎10-04-2018 08:42 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Sandeep Nemuri my use case was to move the MySQL data to PostgreSQL for which my cluster has 5tb ram and data may grow (as of now 200Million) is that achievable from spark and will that affect the cluster.
Previously data transfer was from sqoop to hive and then any target db
