In general, any query being on the same source, like a table should be cached for you to avoid unnecessary IO and you can cache the using - https://docs.databricks.com/spark/latest/spark-sql/language-manual/cache-table.html If this is not relevant, you can maintain a cache of all the results in memory using your constructs, which is also easily done using different cache libraries you can pick off the shelf. Hope it helps.
... View more
