Process an RDD through the databricks xml jar.

[#1]

I have a need to process some xml data in Spark ( 1.6) using the databricks xml jar. My problem is the data source adds "xmlns: /data/path/d" to the root element tag and this extra verbiage makes the databricks xml parser not parse a node. If I remove the extra verbiage and leave a normal tag like <tag1\> , the parser parses fine. I would like to load the file to an RDD, replaceAll on the verbiage, and then run the RDD through the databricks xml parser to create a dataframe. So, the main question is I'm not sure how to load the RDD into the databricks xml jar. I only see examples of files being loaded.

[#2]

@kenny creed,

Can you please share a sample xml file with "xmlns: /data/path/d" to try out a solution?

Thanks

Vinod

[#3]

problem.xml

[#4]

Please try this and let me know. I've tested in spark 1.6.3.

./bin/spark-shell --packages com.databricks:spark-xml_2.10:0.4.1

scala> sqlContext.read.format("com.databricks.spark.xml").option("rowTag","WSAOnRoad").load("file:///root/problem.xml").show(false)

[#5]

@kenny creed

Using sample xml given below, I'm able to parse and get the result

sqlContext.read.format("com.databricks.spark.xml").option("rowTag", "root").load("file:///root/testxml/data.xml").show()

<root xmlns:h="http://www.w3.org/TR/html4/"
xmlns:f="https://www.abc.com/furniture">

<h:table>
<h:tr>
<h:td>Apples</h:td>
<h:td>Bananas</h:td>
</h:tr>
</h:table>

<f:table>
<f:name>Coffee Table</f:name>
<f:width>80</f:width>
<f:length>120</f:length>
</f:table>

</root

[#6]

if SLA is not the constraint, you can save RDD as temporary file and read it again via databricks. If you are running it via zeppelin dashboard, you can invoke shell interpreter and use sed to do an infile replace for xmlns: prior to reading it on your dataframe.

[#7]

Yes, I realize I could write it to disk but was trying to avoid that if possible. I am not using Zeppellin.

[#8]

@kenny creed

Using a sample xml file with "xmlns: /data/path/d" to the root element tag, I'm able to parse it with this code

sqlContext.read.format("com.databricks.spark.xml").option("rowTag", "root").load("file:///testxml/data.xml").show()


//Sample XML file
<root xmlns:h="http://www.w3.org/TR/html4/"
xmlns:f="https://www.abc.com/furniture">

<h:table>
<h:tr>
<h:td>Apps</h:td>
<h:td>bean</h:td>
</h:tr>
</h:table>

<f:table>
<f:name>tables</f:name>
<f:width>80</f:width>
<f:length>120</f:length>
</f:table>
</root>
  

If it is not solving your current issue, please share your sample xml file to understand the xml content better.

[#9]

I went back and checked my uploaded file and I too can parse it. I am thinking there may be some type of hidden characters somewhere in my file. It is too big to upload the complete file. Is there a way to map the file as an RDD, replaceAll, and input it through the Databricks parser?

[#10]

@kenny creed

You may try the APIs suggested in this link : https://github.com/databricks/spark-xml/issues/114#issuecomment-215301575