Support Questions
Find answers, ask questions, and share your expertise

Reducing load on RDBMS while using sparkSQl

New Contributor

Is there a way to reduce load on RDBMS while using sparkSQl as each time we need to query from database?

1 REPLY 1

New Contributor

In general, any query being on the same source, like a table should be cached for you to avoid unnecessary IO and you can cache the using -

https://docs.databricks.com/spark/latest/spark-sql/language-manual/cache-table.html

If this is not relevant, you can maintain a cache of all the results in memory using your constructs, which is also easily done using different cache libraries you can pick off the shelf.

Hope it helps.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.