Support Questions
Find answers, ask questions, and share your expertise

Spark Linage with Spline doesnt work

Super Collaborator

After try to integrate Spark - Spline and Atlas with Spark Shell:

spark-shell  --master yarn --driver-java-options='-Dspline.persistence.factory=za.co.absa.spline.persistence.atlas.AtlasPersistenceFactory' --files /usr/hdp/2.6.4.0-91/kafka/conf/producer.properties --conf 'spark.driver.extraJavaOptions=-Datlas.kafka.bootstrap.servers=lbkbd4.liberbank.cloud:6667,lbkbd2.liberbank.cloud:6667,lbkbd3.liberbank.cloud:6667 -Dbootstrap.servers=lbkbd4.liberbank.cloud:6667,lbkbd2.liberbank.cloud:6667,lbkbd3.liberbank.cloud:6667 -Dspline.persistence.factory=za.co.absa.spline.persistence.atlas.AtlasPersistenceFactory -Datlas.kafka.auto.commit.enable=false -Datlas.kafka.hook.group.id=atlas -Datlas.kafka.zookeeper.connect=lbkbd1.liberbank.cloud:2181 -Datlas.kafka.zookeeper.connection.timeout.ms=30000 -Datlas.kafka.zookeeper.session.timeout.ms=60000 -Datlas.kafka.zookeeper.sync.time.ms=20 -Dcluster.name=lbkhdpbigsql -Dabsolute.base.path=hdfs://lbkbd1.liberbank.cloud:8020' --driver-class-path /tmp/spline-core-0.3.1.jar

the next error appear:

scala> import za.co.absa.spline.core.SparkLineageInitializer._
import za.co.absa.spline.core.SparkLineageInitializer._


scala> spark.enableLineageTracking()
error: missing or invalid dependency detected while loading class file 'SparkLineageInitializer.class'.
Could not access type Logging in value org.slf4s,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'SparkLineageInitializer.class' was compiled against an incompatible version of org.slf4s.

Can anyone help me please?

1 REPLY 1

New Contributor

i am also facing similar issue ! Have you got any resolution yet on this.

kind regards

sameer