https://community.cloudera.com/t5/Support-Questions/com-databricks-spark-xml-parsing-xml-takes-a-very-long-time/m-p/130447#M93133 <P>Hello All,</P><P>I require to import and parse xml files in Hadoop.</P><P>I have an old pig 'REGEX_EXTRACT' script parser that works fine but takes a sometime to run, arround 10-15mins.</P><P>In the last 6 months, I have started to use spark, with large success in improving run time. So I am trying to move the old pig script into spark using databricks xml parser. Mentioned in the following posts: <A href="http://community.hortonworks.com/questions/71538/parsing-xml-in-spark-rdd.html" target="_blank">http://community.hortonworks.com/questions/71538/parsing-xml-in-spark-rdd.html</A> <A href="http://community.hortonworks.com/questions/66678/how-to-convert-spark-dataframes-into-xml-files.html" target="_blank">http://community.hortonworks.com/questions/66678/how-to-convert-spark-dataframes-into-xml-files.html</A> The version used is; <A href="http://github.com/databricks/spark-xml/tree/branch-0.3" target="_blank">http://github.com/databricks/spark-xml/tree/branch-0.3</A> </P><P>The script I try to run is similar to: </P><PRE>import org.apache.spark.{SparkConf, SparkContext} import org.apache.spark.sql.hive.orc._ import org.apache.spark.sql._ import org.apache.hadoop.fs._ import com.databricks.spark import com.databricks.spark.xml import org.apache.spark.sql.functions._ import org.apache.spark.sql.SQLContext import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType} // drop table val dfremove = hiveContext.sql("DROP TABLE FileExtract") // Create schema val xmlSchema = StructType(Array( StructField("Text1", StringType, nullable = false), StructField("Text2", StringType, nullable = false), StructField("Text3", StringType, nullable = false), StructField("Text4", StringType ,nullable = false), StructField("Text5", StringType, nullable = false), StructField("Num1", IntegerType, nullable = false), StructField("Num2", IntegerType, nullable = false), StructField("Num3", IntegerType, nullable = false), StructField("Num4", IntegerType, nullable = false), StructField("Num5", IntegerType, nullable = false), StructField("Num6", IntegerType, nullable = false), StructField("AnotherText1", StringType, nullable = false), StructField("Num7", IntegerType, nullable = false), StructField("Num8", IntegerType, nullable = false), StructField("Num9", IntegerType, nullable = false), StructField("AnotherText2", StringType, nullable = false) )) // Read file val df = hiveContext.read.format("com.databricks.spark.xml").option("rootTag", "File").option("rowTag", "row").schema(xmlSchema).load("hdfs://MyCluster/RawXMLData/RecievedToday/File/Files.tar.gz") // select val selectedData = df.select("Text1", "Text2", "Text3", "Text4", "Text5", "Num1", "Num2", "Num3", "Num4", "Num5", "Num6", "AnotherText1", "Num7", "Num8", "Num9", "AnotherText2" ) selectedData.write.format("orc").mode(SaveMode.Overwrite).saveAsTable("FileExtract") </PRE><P>The xml file looks similar to:</P><PRE>&lt;?xml version="1.0"?&gt; &lt;File&gt; &lt;row&gt; &lt;Text1&gt;something here&lt;/Text1&gt; &lt;Text2&gt;something here&lt;/Text2&gt; &lt;Text3&gt;something here&lt;/Text3&gt; &lt;Text4&gt;something here&lt;/Text4&gt; &lt;Text5&gt;something here&lt;/Text5&gt; &lt;Num1&gt;2&lt;/Num1&gt; &lt;Num2&gt;1&lt;/Num2&gt; &lt;Num3&gt;1&lt;/Num3&gt; &lt;Num4&gt;0&lt;/Num4&gt; &lt;Num5&gt;1&lt;/Num5&gt; &lt;Num6&gt;0&lt;/Num6&gt; &lt;AnotherText1&gt;something here&lt;/AnotherText1&gt; &lt;Num7&gt;2&lt;/Num7&gt; &lt;Num8&gt;0&lt;/Num8&gt; &lt;Num9&gt;0&lt;/Num9&gt; &lt;AnotherText2&gt;something here&lt;/AnotherText2&gt; &lt;/row&gt; &lt;row&gt; &lt;Text1&gt;something here&lt;/Text1&gt; &lt;Text2&gt;something else here&lt;/Text2&gt; &lt;Text3&gt;something new here&lt;/Text3&gt; &lt;Text4&gt;something here&lt;/Text4&gt; &lt;Text5&gt;something here&lt;/Text5&gt; &lt;Num1&gt;2&lt;/Num1&gt; &lt;Num2&gt;1&lt;/Num2&gt; &lt;Num3&gt;1&lt;/Num3&gt; &lt;Num4&gt;0&lt;/Num4&gt; &lt;Num5&gt;1&lt;/Num5&gt; &lt;Num6&gt;0&lt;/Num6&gt; &lt;AnotherText1&gt;something here&lt;/AnotherText1&gt; &lt;Num7&gt;2&lt;/Num7&gt; &lt;Num8&gt;0&lt;/Num8&gt; &lt;Num9&gt;0&lt;/Num9&gt; &lt;AnotherText2&gt;something here&lt;/AnotherText2&gt; &lt;/row&gt; ... ... &lt;/File&gt;</PRE><P>Many xml files are zipped together. Hence the tar.gz file. </P><P>This runs. However for a 400MB file it takes 50mins to finish. </P><P>Does anyone have an idea why it is so slow, or how I may speed it up? I am running on a 7 machine cluster with about 120GB Yarn memory, with hortonworks HDP-2.5.3.0 and spark 1.6.2.</P><P>Many thanks in Advance!</P> Mon, 16 Jan 2017 18:03:21 GMT antin_leszczysz 2017-01-16T18:03:21Z