Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Reducing load on RDBMS while using sparkSQl

New Contributor

Is there a way to reduce load on RDBMS while using sparkSQl as each time we need to query from database?

1 REPLY 1

New Contributor

In general, any query being on the same source, like a table should be cached for you to avoid unnecessary IO and you can cache the using -

https://docs.databricks.com/spark/latest/spark-sql/language-manual/cache-table.html

If this is not relevant, you can maintain a cache of all the results in memory using your constructs, which is also easily done using different cache libraries you can pick off the shelf.

Hope it helps.