Running a Sqoop script in the background appears to 'hang' but works when running in the foreground

[#1]

Consdering the following bash script for Sqoop:

#!/bin/sh

connection_string="jdbc:sqlserver://remoteserver.somehwere.location-asp.com:1433;database=idistrict"

user_name="OPSUser"

db_password="OPSReader"

sqoop_cmd="list-databases"

sqoop $sqoop_cmd --connect $connection_string --username $user_name --password $db_password

We can run this just fine in the foreground, i.e.:

./sqoop_test.sh

But running it in the background like so:

./sqoop_test.sh &

The script appears to 'hang' when kicking off the actual sqoop command...i.e. nothing happens at all.

Using -x on the #!/bin/sh line shows that we end up at the last line of the script and then nothing...

We have tried all kinds of iterations of different commands like:

nohup bash sqoop.sh > results.txt 2>&1 &

./sqoop.sh &> /dev/null &

switched to #!/bin/bash

Any ideas? The odd thing is that the same exact script works fine both foregrounded and backgrounded on a different cluster. /etc/profile, and .bash_profile don't look to have any major differences.

[#2 · Accepted solution]

@Alex Miller I was able to reproduce and personally had luck with screen as well as placing the "&" inside my test script itself at the end of the sqoop command rather than trying to background the script at invocation time (i.e. ./sqoop.sh &).

The /dev/null thing was also successful for me as well with Accumulo in place.

The customer apparently had gone ahead and removed the Accumulo bits before they had a chance to test my suggestions since any further they weren't using it, anyway.

So I really think there isn't a bug and we are hitting some bash-isms here more than anything else.

Thanks, all, for the tips.

[#3]

@Kent Baxley Thanks for sharing this.

[#4]

I too had this problem. My bash script worked fine in my DEV environment as a background job and as a foreground job. However, in my TEST environment the job would only run as a foreground job. In TEST, running as a nohup job would seem to stop at the point where my Sqoop step was called. Ultimately I came across this thread which pointed me in the right direction. Essentially you can emulate nohup by "daemonizing" your script.

setsid ./sqoop.sh </dev/null &>myLog.out &

[#5]

@Kent Baxley

Check what is the $PATH variable used in the script and in the CMD line. try adding the PATH to the script to be same as you run in the cmdline.