Getting error in Python3 kernel for pyspark with HDP 2.6.2

[#1]

Hello,

We have a cluster with HDP 2.6.2.0 and we face an issue when running a Python 3 script.

See with the following trace :

	

	18/04/10 10:40:38 WARN ScriptBasedMapping: Exception running /etc/hadoop/conf/topology_script.py 172.16.10.2 
	

	ExitCodeException exitCode=1:  File "/etc/hadoop/conf/topology_script.py", line 63
	

	  print rack
	

	  ^
	

	SyntaxError: Missing parentheses in call to 'print' 

	

	   at org.apache.hadoop.util.Shell.runCommand(Shell.java:944)
	

	   at org.apache.hadoop.util.Shell.run(Shell.java:848)
	

	   at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1142)
	

	   at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:251)
	

	   at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:188)
	

	   at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
	

	   at org.apache.hadoop.yarn.util.RackResolver.coreResolve(RackResolver.java:101)
	

	   at org.apache.hadoop.yarn.util.RackResolver.resolve(RackResolver.java:81)
	

	   at org.apache.spark.scheduler.cluster.YarnScheduler.getRackForHost(YarnScheduler.scala:37)
	

	   at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$resourceOffers$1.apply(TaskSchedulerImpl.scala:306)
	

	   at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$resourceOffers$1.apply(TaskSchedulerImpl.scala:295)
	

	   at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	

	   at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
	

	   at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	

	   at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	

	   at org.apache.spark.scheduler.TaskSchedulerImpl.resourceOffers(TaskSchedulerImpl.scala:295)
	

	   at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.org$apache$spark$scheduler$cluster$CoarseGrainedSchedulerBackend$DriverEndpoint$$makeOffers(CoarseGrainedSchedulerBackend.scala:226)
	

	   at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint$$anonfun$receive$1.applyOrElse(CoarseGrainedSchedulerBackend.scala:142)
	

	   at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:117)
	

	   at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
	

	   at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
	

	   at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:213)
	

	   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	

	   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	

	   at java.lang.Thread.run(Thread.java:745)

[#2]

@shobhit shukla

As per the Support Metrics of HDP 2.6.2 then tested & certified Python versions are as following:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_support-matrices/content/ch_matrices-amb...

For SLES 11: Python 2.6.x

For SLES 12: Python 2.7.x

For CentOS 7, Ubuntu 14, Ubuntu 16 and Debian 7: Python 2.7.x

.

With Python 2.7 i see the script running file as :

# python --version
Python 2.7.5
# python /etc/hadoop/conf/topology_script.py
/default-rack

.