I am ingesting data into Hive using Sqoop. I'm connecting to DB2 for iSeries using the jt400.jar jdbc driver. Here is the log of the error I'm getting. I appreciate help on how to solve it. Warning: /usr/hdp/2.4.2.0-258/accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. /usr/hdp/2.4.2.0-258/hbase/conf/hbase-env.sh: line 15: export: ` -Djava.net.preferIPv4Stack=true -XX:+PrintGCTimeStamps -verbose:gc -XX:CMSInitiatingOccupancyFraction=70 -XX:+PerfDisableSharedMem -XX:+PrintGCDetails -Xloggc:/var/log/hbase/gc.log-201803161634 -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=8 -XX:+PrintGCDateStamps -XX:+UseCMSInitiatingOccupancyOnly': not a valid identifier 18/03/16 16:34:53 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.2.0-258 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 18/03/16 16:34:55 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override 18/03/16 16:34:55 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc. 18/03/16 16:34:56 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time. 18/03/16 16:34:56 INFO tool.CodeGenTool: Beginning code generation 18/03/16 16:34:57 ERROR sqoop.Sqoop: Got exception running Sqoop: java.awt.HeadlessException: No X11 DISPLAY variable was set, but this program performed an operation which requires it. java.awt.HeadlessException: No X11 DISPLAY variable was set, but this program performed an operation which requires it. at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:207) at java.awt.Window.<init>(Window.java:535) at java.awt.Frame.<init>(Frame.java:420) at java.awt.Frame.<init>(Frame.java:385) at com.ibm.as400.access.ToolboxSignonHandler.displayMessage(ToolboxSignonHandler.java:336) at com.ibm.as400.access.ToolboxSignonHandler.displayMessage(ToolboxSignonHandler.java:323) at com.ibm.as400.access.ToolboxSignonHandler.userIdUnknown(ToolboxSignonHandler.java:302) at com.ibm.as400.access.AS400.promptSignon(AS400.java:2997) at com.ibm.as400.access.AS400.signon(AS400.java:4252) at com.ibm.as400.access.AS400.connectService(AS400.java:1342) at com.ibm.as400.access.AS400JDBCConnection.setProperties(AS400JDBCConnection.java:3383) at com.ibm.as400.access.AS400JDBCDriver.prepareConnection(AS400JDBCDriver.java:1407) at com.ibm.as400.access.AS400JDBCDriver.initializeConnection(AS400JDBCDriver.java:1244) at com.ibm.as400.access.AS400JDBCDriver.connect(AS400JDBCDriver.java:395) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:215) at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:885) at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:744) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:767) at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:270) at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241) at org.apache.sqoop.manager.SqlManager.getColumnTypesForQuery(SqlManager.java:234) at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:304) at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1845) at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) at org.apache.sqoop.Sqoop.run(Sqoop.java:148) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235) at org.apache.sqoop.Sqoop.main(Sqoop.java:244) ... View more

My Sqoop import appears to be failing at the step where it copies the data from the hdfs INPATH to the Hive table the data is being imported to. I pasted the last snippet of the output below. When I test the status code from the Sqoop command it returns a 1. I suspect it is something with my Linux shell script, because I have other versions of the script that work fine. I have not been able to debug this even though I have turned on --verbose option in Sqoop and examined log files from YARN. I am pretty sure the error has something to do with the data not being correctly transferred from the hdfs directory where the files are imported to the hdfs directory associated with the managed table (name shown below) but I can't find any error messages that point me to the solution. Any ideas how to debug this? 18/01/12 00:54:53 DEBUG hive.TableDefWriter: Load statement: LOAD DATA INPATH 'hdfs://surus-nameservice/data/groups/hdp_ground/sqoop/offload_scan_detail_staging_test' OVERWRITE INTO TABLE `hdp_ground.offload_scan_detail_staging_test` 18/01/12 00:54:53 INFO hive.HiveImport: Loading uploaded data into Hive 18/01/12 00:54:53 DEBUG hive.HiveImport: Using in-process Hive instance. 18/01/12 00:54:53 DEBUG util.SubprocessSecurityManager: Installing subprocess security manager Logging initialized using configuration in jar:file:/usr/hdp/2.4.2.0-258/hive/lib/hive-common-1.2.1000.2.4.2.0-258.jar!/hive-log4j.properties ... View more

I am trying to run beeline script called from a linux shell script using nohup. When I do this, the script "hangs" until I issue a Linux kill command for the process associated with the script launched with nohup. Can someone help me get this to work so that I can launch the script in the background and have it finish without having to issue the Linux kill command? #Code for beeline script simple_query.hql ---------------------------------------------- use hdp_ground; select scan_create_dt_part from offload_scan_detail_orc where scan_create_dt_part = '20171001' limit 5; ---------------------------------------------- #Code for Linux shell script run_ beeline_hql.sh ---------------------------------------------- #run environment script THISFILE='run_beeline_hql' EXT1=$(date +%y%m%d) EXT2=$(date +%H%M%S) . $(dirname $0)/srv.env exec > $output_dir/${THISFILE}_$EXT1.$EXT2.log 2>&1 chmod 666 $output_dir/${THISFILE}_$EXT1.$EXT2.log beeline -f simple_query.hql exit ---------------------------------------------- What I type to launch the Linux script: ---------------------------------------------- nohup ./run_beeline_hql.sh & ---------------------------------------------- What I see on my screen: nohup ./run_beeline_hql.sh & [1] 58486 [xxxx/home/xxxx/xxx/xxx]$ nohup: ignoring input and appending output to `nohup.out' [1] + Stopped (SIGTTOU) nohup ./run_beeline_hql.sh & -------------------------- when I do ps -ef | grep 58486 it appears the script is still running and never finishes: -------------------------------------------------------- [xxx/home/xxx]$ ps -ef | grep 58486 xxx 45124 31863 0 17:35 pts/0 00:00:00 grep 58486 xxx 58486 31863 0 17:13 pts/0 00:00:00 /bin/sh ./run_beeline_hql.sh xxx 58493 58486 0 17:13 pts/0 00:00:00 /opt/java/hotspot/7/64_bit/jdk1.7.0_79/bin/java -Xmx12288m -Dhdp.version=2.4.2.0-258 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.4.2.0-258 -Dhadoop.log.dir=/var/hadoop/log/hadoop/f5057708 -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.2.0-258/hadoop -Dhadoop.id.str=f5057708 -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx12288m -XX:MaxPermSize=512m -Dlog4j.configuration=beeline-log4j.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.4.2.0-258/hive/lib/hive-beeline-1.2.1000.2.4.2.0-258.jar org.apache.hive.beeline.BeeLine -u jdbc:hive2://xxx;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 -f simple_query.hql ------------------------------- The output of the script looks like this: ----------------------------------------- WARNING: Use "yarn jar" to launch YARN applications. ----------------------------------------- when I issue the kill command for the process: kill 58486 the script finishes and the output gets written ---------------------------------------------- Connecting to jdbc:hive2://xxx;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 Connected to: Apache Hive (version 1.2.1000.2.4.2.0-258) Driver: Hive JDBC (version 1.2.1000.2.4.2.0-258) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://xxx,> use hdp_ground; No rows affected (0.062 seconds) 0: jdbc:hive2://xxx,> select scan_create_dt_part from offload_scan_detail_orc where scan_create_dt_part = '201 71001' limit 5; +----------------------+--+ | scan_create_dt_part | +----------------------+--+ | 20171001 | | 20171001 | | 20171001 | | 20171001 | | 20171001 | +----------------------+--+ 5 rows selected (0.127 seconds) 0: jdbc:hive2://xxx,> Closing: 0: jdbc:hive2://xxx/xxx;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 ... View more