Support Questions
Find answers, ask questions, and share your expertise

Reducing load on RDBMS while using sparkSQl

Reducing load on RDBMS while using sparkSQl

New Contributor

Is there a way to reduce load on RDBMS while using sparkSQl as each time we need to query from database?

1 REPLY 1
Highlighted

Re: Is there a way to reduce load on RDBMS while using sparkSQl as each time we need to query from database?

New Contributor

In general, any query being on the same source, like a table should be cached for you to avoid unnecessary IO and you can cache the using -

https://docs.databricks.com/spark/latest/spark-sql/language-manual/cache-table.html

If this is not relevant, you can maintain a cache of all the results in memory using your constructs, which is also easily done using different cache libraries you can pick off the shelf.

Hope it helps.