Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi spark jobs are failing in Docker container

Highlighted

Nifi spark jobs are failing in Docker container

New Contributor

Hello,

I have migrated a nifi flow from nifi host to Nifi docker image. This flow execute few spark jobs which are running perfectly fine when i run them locally but in Dockers these processors are failing with following errors. I have mounted all the volumes but still these jobs are failing. Any idea how to fix these..?

ls: cannot access '/usr/hdp//hadoop/lib': No such file or directory

Exception in thread "main" java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxx