Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark Linage with Spline doesnt work

Highlighted

Spark Linage with Spline doesnt work

Super Collaborator

After try to integrate Spark - Spline and Atlas with Spark Shell:

spark-shell  --master yarn --driver-java-options='-Dspline.persistence.factory=za.co.absa.spline.persistence.atlas.AtlasPersistenceFactory' --files /usr/hdp/2.6.4.0-91/kafka/conf/producer.properties --conf 'spark.driver.extraJavaOptions=-Datlas.kafka.bootstrap.servers=lbkbd4.liberbank.cloud:6667,lbkbd2.liberbank.cloud:6667,lbkbd3.liberbank.cloud:6667 -Dbootstrap.servers=lbkbd4.liberbank.cloud:6667,lbkbd2.liberbank.cloud:6667,lbkbd3.liberbank.cloud:6667 -Dspline.persistence.factory=za.co.absa.spline.persistence.atlas.AtlasPersistenceFactory -Datlas.kafka.auto.commit.enable=false -Datlas.kafka.hook.group.id=atlas -Datlas.kafka.zookeeper.connect=lbkbd1.liberbank.cloud:2181 -Datlas.kafka.zookeeper.connection.timeout.ms=30000 -Datlas.kafka.zookeeper.session.timeout.ms=60000 -Datlas.kafka.zookeeper.sync.time.ms=20 -Dcluster.name=lbkhdpbigsql -Dabsolute.base.path=hdfs://lbkbd1.liberbank.cloud:8020' --driver-class-path /tmp/spline-core-0.3.1.jar

the next error appear:

scala> import za.co.absa.spline.core.SparkLineageInitializer._
import za.co.absa.spline.core.SparkLineageInitializer._


scala> spark.enableLineageTracking()
error: missing or invalid dependency detected while loading class file 'SparkLineageInitializer.class'.
Could not access type Logging in value org.slf4s,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'SparkLineageInitializer.class' was compiled against an incompatible version of org.slf4s.

Can anyone help me please?

1 REPLY 1

Re: Spark Linage with Spline doesnt work

New Contributor

i am also facing similar issue ! Have you got any resolution yet on this.

kind regards

sameer

Don't have an account?
Coming from Hortonworks? Activate your account here