Member since
03-27-2017
9
Posts
0
Kudos Received
0
Solutions
01-02-2019
10:22 AM
Hi Nilesh, I know it is too late to answer, still lets add a solution for future issues Please try --conf "spark.executor.extraJavaOptions=-XX:+UseG1GC -Djava.security.auth.login.config=kafka_jaas.conf" instead of --conf "spark.executor.extraJavaOptions=-XX:+UseG1GC -Djava.security.auth.login.config=/home/user/sparktest/kafka_jaas.conf" Also ensure to use just the file name 'user.headless.keytab' in jaas file (not absolute file path) . eg: KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
doNotPrompt=true
useTicketCache=false
principal="userprincipalname"
useKeyTab=true
serviceName="kafka"
keyTab="user.headless.keytab"
client=true;
};
Thanks Vinod
... View more
04-17-2018
02:21 PM
@Venkata Sudheer Kumar M Can you please share the spark documentation which refers "SPARK_YARN_DIST_FILES" ? In Spark 2.2 code, I couldn't locate usage of this env variable.
... View more
08-26-2017
05:08 AM
Hi Anilkumar, While submitting spark application , you need to pass your Teradata jdbc Driver jar file that with --jar option Thanks Vinod
... View more
08-20-2017
08:01 AM
@kenny creed You may try the APIs suggested in this link : https://github.com/databricks/spark-xml/issues/114#issuecomment-215301575
... View more
08-18-2017
06:24 PM
@kenny creed Using a sample xml file with "xmlns: /data/path/d" to the root element tag, I'm able to parse it with this code sqlContext.read.format("com.databricks.spark.xml").option("rowTag", "root").load("file:///testxml/data.xml").show()
//Sample XML file
<root xmlns:h="http://www.w3.org/TR/html4/"
xmlns:f="https://www.abc.com/furniture">
<h:table>
<h:tr>
<h:td>Apps</h:td>
<h:td>bean</h:td>
</h:tr>
</h:table>
<f:table>
<f:name>tables</f:name>
<f:width>80</f:width>
<f:length>120</f:length>
</f:table>
</root>
If it is not solving your current issue, please share your sample xml file to understand the xml content better.
... View more
08-18-2017
06:24 PM
@kenny creed Using sample xml given below, I'm able to parse and get the result sqlContext.read.format("com.databricks.spark.xml").option("rowTag", "root").load("file:///root/testxml/data.xml").show() <root xmlns:h="http://www.w3.org/TR/html4/"
xmlns:f="https://www.abc.com/furniture">
<h:table>
<h:tr>
<h:td>Apples</h:td>
<h:td>Bananas</h:td>
</h:tr>
</h:table>
<f:table>
<f:name>Coffee Table</f:name>
<f:width>80</f:width>
<f:length>120</f:length>
</f:table>
</root
... View more
08-18-2017
02:01 AM
Please try this and let me know. I've tested in spark 1.6.3. ./bin/spark-shell --packages com.databricks:spark-xml_2.10:0.4.1 scala> sqlContext.read.format("com.databricks.spark.xml").option("rowTag","WSAOnRoad").load("file:///root/problem.xml").show(false)
... View more
08-17-2017
04:16 PM
@kenny creed, Can you please share a sample xml file with "xmlns: /data/path/d" to try out a solution? Thanks Vinod
... View more
05-10-2017
04:35 PM
Updated link : https://github.com/hortonworks/data-tutorials/blob/master/tutorials/hdf/hdf-2.1/realtime-event-processing-in-hadoop-with-nifi-kafka-and-storm/assets/IoT_Lab_Series_DataFlow.xml
... View more