Thanks @Benassi10 for providing the context. Much appreciated.   We are discussing this internally to see what can cause such issues. One theory is that we enabled support for Spark Lineage in CDS 2.3 and if the cm-agent doesn't create /var/log/spar2/lineage directory (for some reasons) you can see this behaviour. If lineage is not important, can you try running the shell with lineage disabled?   spark2-shell  --conf spark.lineage.enabled=false   If you don't want to disable lineage, another workaround would be to change the lineage directory to /tmp  in CM > Spark2 > Configuration > GATEWAY Lineage Log Directory > /tmp , followed by redeploying the client configuration.   Let us know if the above helps. I will update the thread once I have more information on the fix. ... View more

@Swasg  by any chance are you using the package name in the spark-shell? Something like  spark-shell --packages org.apache.spark:spark-streaming-kafka-0-8-assembly_2.11-2.3.0   The error suggests that the format should be in the form of 'groupId:artifactId:version' but in your case it's 'groupId:artifactId-version'. If you are using the package in the command line or somewhere in your configuration, please modify it to: org.apache.spark:spark-streaming-kafka-0-8-assembly_2.11:2.3.0   ... View more

Thanks for reporting. Care to share the full error for the lineage file missing, please? I quickly tested an upgrade from 2.2 to 2.3 but didn't hit this. A full error stack trace would certainly help. ... View more

Try this: http://site.clairvoyantsoft.com/installing-sparkr-on-a-hadoop-cluster/ ... View more

@hedy thanks for sharing.   The workaround you received makes sense when you are not using any cluster manager(?)   Local mode ( --master local[i] ) is generally seen if you want to test or debug something quickly since there will be only one JVM launched on the node from where you are running pyspark and this JVM will act as driver, executor, and master -> all-in-one. But of course with local mode, you lose the scalability and resource management that a cluster manager provides. If you want to debug why simultaneous spark shells are not working when using Spark-On-Yarn, we need to diagnose this from YARN perspective (troubleshooting steps shared in the last post). Let us know. ... View more

  WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.   ^ This generally means that the problem is beyond the port mapping ( i.e either with queue configuration/ available resources/YARN level).   Assuming that you are using spark1.6, I'd suggest to temporarily change the shell logging level to INFO and see if that gives a hint. The easy and quick way to do this would be to edit /etc/spark/conf/log4j.properties from the node you are running pyspark and modify the log level from WARN to INFO.   # vi /etc/spark/conf/log4j.properties shell.log.level=INFO $ spark-shell .... 18/04/10 20:40:50 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 18/04/10 20:40:50 INFO util.Utils: Successfully started service 'SparkUI' on port 4041. 18/04/10 20:40:50 INFO client.RMProxy: Connecting to ResourceManager at host-xxx.cloudera.com/10.xx.xx.xx:8032 18/04/10 20:40:52 INFO impl.YarnClientImpl: Submitted application application_1522940183682_0060 18/04/10 20:40:54 INFO yarn.Client: Application report for application_1522940183682_0060 (state: ACCEPTED) 18/04/10 20:40:55 INFO yarn.Client: Application report for application_1522940183682_0060 (state: ACCEPTED) 18/04/10 20:40:56 INFO yarn.Client: Application report for application_1522940183682_0060 (state: ACCEPTED) 18/04/10 20:40:57 INFO yarn.Client: Application report for application_1522940183682_0060 (state: ACCEPTED)     Next, open the Resource Manager UI and check the state of the Application (i.e your second invocation of pyspark) -- whether it's is registered but just stuck in ACCEPTED state like this:       If yes, look at the Cluster Metrics row at the top of the RM UI page and see if there are enough resources available:       Now kill the first pyspark session and check if the second session changes the state RUNNING in the RM UI. If yes, look at the queue placement rules and stats in Cloudera Manager > Yarn > Resource Pools Usage (and Configuration)         Hopefully, this would give us some more clues. Let us know how it goes? Feel free to share the screen-shots from the RM UI and spark-shell INFO logging. ... View more