Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

spark-submit results in HTTP erro

avatar
New Contributor

When I run spark-submit (spark-submit --class scala.TestSubmit --master yarn --deploy-mode cluster --executor-memory 4G hdfs://xx.x.xxx.xx:4059/user/myaccount/untitled5.jar) with a jar file in the cluster, the submit seems to be working, but the scala spark environment where I expect the output results in an HTTP error:

scala> WARN HttpParser: Illegal character 0x9 in state=METHOD for buffer HeapByteBuffer@4421e6d9[p=5,l=25,c=8192,r=20]={hrpc\t<<<\x00\xDf\x00\x00\x00\x0e\n\x08\x02\x10\x00\x18A"\x00(\x01\x02\x10\x01>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 18/06/05 09:52:03 WARN HttpParser: bad HTTP parsed: 400 Illegal character 0x9 for HttpChannelOverHttp@56f4b278{r=0,c=false,a=IDLE,uri=null}

Can anyone point me in the right direction as to where to look to fix this? I'm just trying to run spark-submit on my jar files that are uploaded in the cluster.

Thankyou

1 REPLY 1

avatar

@Nick Giammanco As per WARN message you shared HttpParser is failing to parse the response from a HTTP request I think.
Could you check If same code runs fine on local mode --master local? If same code runs fine in local mode perhaps there is a firewall/network issue when you run the code in yarn-cluster mode (driver and executors will run on cluster worker nodes), which causes the HTTP requests to fail. Another point to check is whether you need to use an http-proxy.

HTH