- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
spark-submit results in HTTP erro
- Labels:
-
Apache Spark
-
Apache YARN
Created ‎06-05-2018 02:21 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
When I run spark-submit (spark-submit --class scala.TestSubmit --master yarn --deploy-mode cluster --executor-memory 4G hdfs://xx.x.xxx.xx:4059/user/myaccount/untitled5.jar) with a jar file in the cluster, the submit seems to be working, but the scala spark environment where I expect the output results in an HTTP error:
scala> WARN HttpParser: Illegal character 0x9 in state=METHOD for buffer HeapByteBuffer@4421e6d9[p=5,l=25,c=8192,r=20]={hrpc\t<<<\x00\xDf\x00\x00\x00\x0e\n\x08\x02\x10\x00\x18A"\x00(\x01\x02\x10\x01>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 18/06/05 09:52:03 WARN HttpParser: bad HTTP parsed: 400 Illegal character 0x9 for HttpChannelOverHttp@56f4b278{r=0,c=false,a=IDLE,uri=null}
Can anyone point me in the right direction as to where to look to fix this? I'm just trying to run spark-submit on my jar files that are uploaded in the cluster.
Thankyou
Created ‎06-05-2018 02:28 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Nick Giammanco As per WARN message you shared HttpParser is failing to parse the response from a HTTP request I think.
Could you check If same code runs fine on local mode --master local? If same code runs fine in local mode perhaps there is a firewall/network issue when you run the code in yarn-cluster mode (driver and executors will run on cluster worker nodes), which causes the HTTP requests to fail. Another point to check is whether you need to use an http-proxy.
HTH
