question Re: Spark SQL in-memory space managment in Archives of Support Questions (Read Only)

Spark SQL in-memory space managment

mothi86 — Fri, 20 Jan 2017 00:38:24 GMT

Hi. Considering a spark sql or data set with 400 columns and 1 million rows. Not all rows have all 400 columns populated and essentially they cant be as not null columns as well. Need to understand if null value consumes space in memory and if so how much does it take. Do we have any fact sheet or article of all data types size in bytes or bits.

Re: Spark SQL in-memory space managment

cstanca — Wed, 22 Feb 2017 06:21:20 GMT

@Mothilal marimuthu

You did not specify whether you are talking about RDD, Datasets or Dataframe.

Anyhow, let't assume RDD. It is not like a columnar database where you account only for the key-value. This is a row-based format. There is cost associated with empty values. I cannot tell you the exact cost because it depends on your data types, but there is cost to it.

Why don't you run yourself a test. Persist your test RDD (small) with all values completed, then one with partial values, some of them null. Again, the data type matters. You can experiment by using null values on columns of the same type, then another RDD for a different type, etc.

rdd.persist(StorageLevel.MEMORY_AND_DISK)

Re: Spark SQL in-memory space managment

mothi86 — Sat, 22 Apr 2017 01:12:35 GMT

Thanks @Constantin Stanca