Support Questions
Find answers, ask questions, and share your expertise

Which “JAR” file do i need to be able to import “org.apache.parquet” in Scala?

Which “JAR” file do i need to be able to import “org.apache.parquet” in Scala?

New Contributor

When I try this:

scala> import org.apache.parquet

It errors out:

<console>:23: error: object parquet is not a member of package org.apache
       import org.apache.parquet

Question - which jar do i need to include in spark conf for this import to work?

Note, this works fine:

scala> import org.apache.hadoop
import org.apache.hadoop

CDH jars i have access to:

$ ls /opt/cloudera/parcels/CDH-5.14.4-1.cdh5.14.4.p3503.3712/jars -al|grep parq
-rw-r--r--.  1 root root     12717 Jun 24  2019 kite-morphlines-hadoop-parquet-avro-1.0.0-cdh5.14.4.jar
-rw-r--r--.  1 root root    106448 Jun 24  2019 parquet-avro-1.5.0-cdh5.14.4.jar
-rw-r--r--.  1 root root     25490 Jun 24  2019 parquet-cascading-1.5.0-cdh5.14.4.jar
-rw-r--r--.  1 root root    956035 Jun 24  2019 parquet-column-1.5.0-cdh5.14.4.jar
-rw-r--r--.  1 root root     41084 Jun 24  2019 parquet-common-1.5.0-cdh5.14.4.jar
-rw-r--r--.  1 root root    278926 Jun 24  2019 parquet-encoding-1.5.0-cdh5.14.4.jar
-rw-r--r--.  1 root root    384620 Jun 24  2019 parquet-format-2.1.0-cdh5.14.4.jar
-rw-r--r--.  1 root root    132777 Jun 24  2019 parquet-format-2.1.0-cdh5.14.4-javadoc.jar
-rw-r--r--.  1 root root      6474 Jun 24  2019 parquet-format-2.1.0-cdh5.14.4-sources.jar
-rw-r--r--.  1 root root     23679 Jun 24  2019 parquet-generator-1.5.0-cdh5.14.4.jar
-rw-r--r--.  1 root root    212644 Jun 24  2019 parquet-hadoop-1.5.0-cdh5.14.4.jar
-rw-r--r--.  1 root root   2776911 Jun 24  2019 parquet-hadoop-bundle-1.5.0-cdh5.14.4.jar
-rw-r--r--.  1 root root    927867 Jun 24  2019 parquet-jackson-1.5.0-cdh5.14.4.jar
-rw-r--r--.  1 root root     84853 Jun 24  2019 parquet-pig-1.5.0-cdh5.14.4.jar
-rw-r--r--.  1 root root   2855960 Jun 24  2019 parquet-pig-bundle-1.5.0-cdh5.14.4.jar
-rw-r--r--.  1 root root     49233 Jun 24  2019 parquet-protobuf-1.5.0-cdh5.14.4.jar
-rw-r--r--.  1 root root     33088 Jun 24  2019 parquet-scala_2.10-1.5.0-cdh5.14.4.jar
-rw-r--r--.  1 root root     22932 Jun 24  2019 parquet-scrooge_2.10-1.5.0-cdh5.14.4.jar
-rw-r--r--.  1 root root      6287 Jun 24  2019 parquet-test-hadoop2-1.5.0-cdh5.14.4.jar
-rw-r--r--.  1 root root    207188 Jun 24  2019 parquet-thrift-1.5.0-cdh5.14.4.jar
-rw-r--r--.  1 root root     67029 Jun 24  2019 parquet-tools-1.5.0-cdh5.14.4.jar
Don't have an account?