Support Questions
Find answers, ask questions, and share your expertise

Spark Topology / Hierarchy Mapping

I have a dataset like this:

"DeviceA","Connection1"
"Connection1","Connection2"
"Connection2","Connection3"
"DeviceB","Connection3"
"DeviceC","Connection2"
"DeviceD","Connection4"
"Connection4","Connection6"
"Connection6","Connection8"

I need to transform it so that each device has a hierarchy, like this:

"DeviceA","Connection1","Connection2","Connection3"
"DeviceB","Connection3"
"DeviceC","Connection2","Connection3"
"DeviceD","Connection4","Connection6","Connection8"

I'm trying to find the most efficient solution, since the topology can be very large, so I am trying to available multiple iterations (if that's possible).

Does anyone have any advice for transforming this data?

I am working with Spark (scala) and the data can be in RDD, Dataframes, or Datasets (whichever is optimal for this job)

Thanks!