Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

SparkSQL - org.apache.spark.sql.catalyst.types.StructField fails

avatar
Explorer

I am trying to use certain functionality from SparkSQL ( namely “programmatically specifying a schema” as described in the Spark 1.1.0 documentation)

 

I am getting the following error:

 

 

15/03/10 17:00:16 INFO storage.BlockManagerMaster: Updated info of block broadcast_2_piece0

15/03/10 17:00:16 INFO spark.SparkContext: Created broadcast 2 from broadcast at MSP.scala:52

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.types.StructField.<init>(Ljava/lang/String;Lorg/apache/spark/sql/catalyst/types/DataType;Z)

at MSP$$anonfun$3.apply(MSP.scala:57)

 

The code is:

val schemaString="ATR1,ATTR2,....”

 

line 57:

  val schema =

  StructType(

    schemaString.split(",").map(fieldName => StructField(fieldName, StringType, true)))

 

 

Why this fails ? I am using Spark 1.2

1 ACCEPTED SOLUTION

avatar
Master Collaborator

In general, NoSuchMethodError in Java means you compiled against one version of something, and executed against a different version. Check your build.

View solution in original post

1 REPLY 1

avatar
Master Collaborator

In general, NoSuchMethodError in Java means you compiled against one version of something, and executed against a different version. Check your build.