@vincentV NA Shuffle data is serialized over the network so when deserialized its spilled to memory and this metric is aggregated on the shuffle spilled (memory) that you see in the UI Please review this: http://apache-spark-user-list.1001560.n3.nabble.com/What-is-shuffle-spill-to-memory-td10158.html "Shuffle spill (memory) is the size of the deserialized form of the data in memory at the time when we spill it, whereas shuffle spill (disk) is the size of the serialized form of the data on disk after we spill it. This is why the latter tends to be much smaller than the former. Note that both metrics are aggregated over the entire duration of the task (i.e. within each task you can spill multiple times)." HTH *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer. ... View more

@vivek jain Good to hear that. If you think the answer and followups have helped please take a moment to login and mark as "Accepted" ... View more

@vivek jain Could you try running the following steps and see if that works: https://community.hortonworks.com/articles/147327/accessing-hbase-tables-and-querying-on-dataframes.html including table creation? ... View more

@vivek jain Please run the following from HBase shell: hbase> scan 'tableName', {'LIMIT' => 5} Also check what the describe table prints: bhase> describe ‘tableName’ Make sure you are using case-sensitive name when referencing table from spark code. HTH ... View more