Running a Sqoop script in the background appears to 'hang' but works when running in the foreground

[#1]

Consdering the following bash script for Sqoop:

#!/bin/sh

connection_string="jdbc:sqlserver://remoteserver.somehwere.location-asp.com:1433;database=idistrict"

user_name="OPSUser"

db_password="OPSReader"

sqoop_cmd="list-databases"

sqoop $sqoop_cmd --connect $connection_string --username $user_name --password $db_password

We can run this just fine in the foreground, i.e.:

./sqoop_test.sh

But running it in the background like so:

./sqoop_test.sh &

The script appears to 'hang' when kicking off the actual sqoop command...i.e. nothing happens at all.

Using -x on the #!/bin/sh line shows that we end up at the last line of the script and then nothing...

We have tried all kinds of iterations of different commands like:

nohup bash sqoop.sh > results.txt 2>&1 &

./sqoop.sh &> /dev/null &

switched to #!/bin/bash

Any ideas? The odd thing is that the same exact script works fine both foregrounded and backgrounded on a different cluster. /etc/profile, and .bash_profile don't look to have any major differences.

[#2]

@Kent Baxley

Is this cluster setup with queues?

Can you check if sqoop is waiting on other jobs to finish ?

[#3]

@Neeraj - No other jobs are waiting to finish and we can run this pretty much at-will in the foreground without things getting seemingly stuck.

[#4]

add "`whoami'" and "`hostname`" to the script, see what prints out. I'd also add "2>&1 | tee -a log", i.e. redirect the output of the console to a file to see the output in foreground and background. It should give you some insight to what's happening. What specifically is the reason to running it in the background @Kent Baxley?

[#5]

check limits on the user in the background and in the foreground. There may be an OS limit on background processes.

[#6]

I would recommend using screen. It gives you all of the benefits of a background job will all of the benefits of a job running in the foreground as well: https://wiki.archlinux.org/index.php/GNU_Screen

[#7]

@Artem Ervits The reason behind the backgrounding is there are quite a few tables with 2+ million records and they would like to run start a sqoop job in the background after hours (there has to be a better way to do this, in my opinion).

Foregrounded:

-bash-4.1$ ./sample_sqoop.sh

whoami sboddu

hostname node2.example.com

2015-11-05 19:15:03,027 INFO - [main:] ~ Running Sqoop version: 1.4.6.2.3.0.0-2557 (Sqoop:92) 2015-11-05 19:15:03,043 WARN - [main:] ~ Setting your password on the command-line is insecure. Consider using -P instead. (BaseSqoopTool:1021)

2015-11-05 19:15:03,248 INFO - [main:] ~ Using default fetchSize of 1000 (SqlManager:98) SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.3.0.0-2557/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.3.0.0-2557/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.3.0.0-2557/accumulo/lib/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

master

tempdb

model msdb

ReportServer

Idistrict_Distributer

idistrict

iDistrict_Audit

iDistrict_SlimDB

iDistrict_Reports

ReportServerTempDB

Idistrict_Replication

iDistrict_Attachment FTL

Backgrounded:

-bash-4.1$ ./sample_sqoop.sh& [2] 33320 -bash-4.1$

whoami sboddu

hostname node2.example.com

[2]+ Stopped ./sample_sqoop.sh

[#8]

you're sqooping master, model, reportdb, not sure if you need to do that, I would limit the tables just to the ones you need. Other than that, please check ulimit on the user executing the job in foreground and background, http://www.commandlinefu.com/commands/view/9893/find-ulimit-values-of-currently-running-process.

[#9]

@Artem Ervits Turns out that the factor was having the Accumulo Client installed on the machine alongside sqoop.

With Accumulo client in the mix, the sqoop script, if invoked to run in the background, would go into a Stopped state and could only resume if the script were foregrounded using the "fg" command.

Uninstalling the Accumulo client was what ultimately worked-around / fixed the issue.

Not sure if this is a bug or due to the fact that sqoop is self is a bash script that calls another script that sources the configure-sqoop script.

Thanks for your help.

[#10]

@Kent Baxley did you have a chance to try using screen before uninstalling Accumulo client? Based on your discovery that redirecting output (sqoop.sh &> /dev/null &) was successful, I would think using screen would also work.