Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

running parallel jobs in HUE raise packet_write_wait: Connection to UNKNOWN port 0: Broken pipe

running parallel jobs in HUE raise packet_write_wait: Connection to UNKNOWN port 0: Broken pipe

New Contributor

running parallel jobs in HUE raise

packet_write_wait: Connection to UNKNOWN port 0: Broken pipe

Error in DataNode & NameNode

 

but when "running jobs is linear" is working fine.

any idea to fix this

 

3 REPLIES 3

Re: running parallel jobs in HUE raise packet_write_wait: Connection to UNKNOWN port 0: Broken pipe

Champion
Do you mean connecting to Hue concurrently? Or is that fine but multiple users running jobs concurrently? Does the latter work outside of Hue?

what is your backend DB for Hue? I believe the default derby db has concurrency issues.

Highlighted

Re: running parallel jobs in HUE raise packet_write_wait: Connection to UNKNOWN port 0: Broken pipe

New Contributor

Q: Do you mean connecting to Hue concurrently?

No , i mean running jobs parallelly under only one user.

 

Q: is that fine but multiple users running jobs concurrently

I didn't test with multiple user. i am running only one main job (parallelly 5x5 times) from a single user.

that means that one main job splitted into 5 groups and each group can have 5 sub-jobs. when the main job is running it will take one group at a time and run all 5 sub-jobs parallelly by creating individual "ssh connection session" under "ssh user@hadoop_en" .

 

Q: Does the latter work outside of Hue?

yes, i can "ssh user@hadoop_en"  into that node and run that main job (sh main_job.sh) directly then the job was working fine.

 

Q: what is your backend DB for Hue?

Default

 

 

Here is my configuration

Hue™ 3.9.0 - The Hadoop UI

Learn more about Hue and Hadoop on http://gethue.com. Hue and the Hue logo are trademarks of Cloudera, Inc.

 

 

Here is what i have done: 

 

When i go to Workflows > Dashboards > Coordinators in web-UI

i can see many of our developer's jobs.

when i run one job (parallel jobs) the job getting failed.

after that i can't able to ssh into that node for that particular user by "ssh user@hadoop_en".

 

when i try to "ssh user@hadoop_en", i am getting this error

packet_write_wait: Connection to UNKNOWN port 0: Broken pipe

 

but i can able to "ssh any_other_user@hadoop_en".

 

Any help ?

 

Re: running parallel jobs in HUE raise packet_write_wait: Connection to UNKNOWN port 0: Broken pipe

Champion
It sound like this is an issue specifically with SSH. I don't recognize the specific error. Try digging through the sshd man page or the various settings in sshd_config. Something in it may be limiting the number of connections per user.