Reply
Explorer
Posts: 14
Registered: ‎12-03-2014

sqoop user

I tried to transfer data from posgresql database to Hbase table, using the following command:

sqoop import --connect jdbc:postgresql://database:port/table --table tablename --hbase-table tableFromSqoop --column-family cf --hbase-row-key id --hbase-create-table --username username --password password

 however, I get the following stack trace of the exception:

 

Spoiler
14/12/09 13:56:26 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/cloudera/parcels/CDH-5.0.2-1.cdh5.0.2.p0.13/lib/hadoop/lib/native
14/12/09 13:56:26 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
14/12/09 13:56:26 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
14/12/09 13:56:26 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
14/12/09 13:56:26 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
14/12/09 13:56:26 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-431.20.3.el6.x86_64
14/12/09 13:56:26 INFO zookeeper.ZooKeeper: Client environment:user.name=root
14/12/09 13:56:26 INFO zookeeper.ZooKeeper: Client environment:user.home=/root
14/12/09 13:56:26 INFO zookeeper.ZooKeeper: Client environment:user.dir=/root
14/12/09 13:56:26 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=trafficdata1.sis.uta.fi:2181 sessionTimeout=60000 watcher=hconnection-0x3d1bd556, quorum=trafficdata1.sis.uta.fi:2181, baseZNode=/hbase
14/12/09 13:56:26 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x3d1bd556 connecting to ZooKeeper ensemble=trafficdata1.sis.uta.fi:2181
14/12/09 13:56:26 INFO zookeeper.ClientCnxn: Opening socket connection to server trafficdata1.sis.uta.fi/153.1.62.167:2181. Will not attempt to authenticate using SASL (unknown error)
14/12/09 13:56:26 INFO zookeeper.ClientCnxn: Socket connection established to trafficdata1.sis.uta.fi/153.1.62.167:2181, initiating session
14/12/09 13:56:26 INFO zookeeper.ClientCnxn: Session establishment complete on server trafficdata1.sis.uta.fi/153.1.62.167:2181, sessionid = 0x14a116fc00c50c8, negotiated timeout = 60000
14/12/09 13:56:27 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
14/12/09 13:56:27 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=trafficdata1.sis.uta.fi:2181 sessionTimeout=60000 watcher=catalogtracker-on-hconnection-0x3d1bd556, quorum=trafficdata1.sis.uta.fi:2181, baseZNode=/hbase
14/12/09 13:56:27 INFO zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x3d1bd556 connecting to ZooKeeper ensemble=trafficdata1.sis.uta.fi:2181
14/12/09 13:56:27 INFO zookeeper.ClientCnxn: Opening socket connection to server trafficdata1.sis.uta.fi/153.1.62.167:2181. Will not attempt to authenticate using SASL (unknown error)
14/12/09 13:56:27 INFO zookeeper.ClientCnxn: Socket connection established to trafficdata1.sis.uta.fi/153.1.62.167:2181, initiating session
14/12/09 13:56:27 INFO zookeeper.ClientCnxn: Session establishment complete on server trafficdata1.sis.uta.fi/153.1.62.167:2181, sessionid = 0x14a116fc00c50c9, negotiated timeout = 60000
14/12/09 13:56:27 INFO zookeeper.ZooKeeper: Session: 0x14a116fc00c50c9 closed
14/12/09 13:56:27 INFO zookeeper.ClientCnxn: EventThread shut down
14/12/09 13:56:27 INFO client.RMProxy: Connecting to ResourceManager at trafficdata1.sis.uta.fi/153.1.62.167:8032
14/12/09 13:56:29 INFO db.DBInputFormat: Using read commited transaction isolation
14/12/09 13:56:29 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN("id"), MAX("id") FROM "traffic" WHERE ( id<10000 )
14/12/09 13:56:29 INFO mapreduce.JobSubmitter: number of splits:4
14/12/09 13:56:29 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1417632191545_0007
14/12/09 13:56:30 INFO impl.YarnClientImpl: Submitted application application_1417632191545_0007
14/12/09 13:56:30 INFO mapreduce.Job: The url to track the job: http://trafficdata1.sis.uta.fi:8088/proxy/application_1417632191545_0007/
14/12/09 13:56:30 INFO mapreduce.Job: Running job: job_1417632191545_0007
14/12/09 13:56:46 INFO mapreduce.Job: Job job_1417632191545_0007 running in uber mode : false
14/12/09 13:56:46 INFO mapreduce.Job:  map 0% reduce 0%
14/12/09 13:56:56 INFO mapreduce.Job: Task Id : attempt_1417632191545_0007_m_000003_0, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

14/12/09 13:56:56 INFO mapreduce.Job: Task Id : attempt_1417632191545_0007_m_000001_0, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 13:56:58 INFO mapreduce.Job: Task Id : attempt_1417632191545_0007_m_000002_0, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 13:56:58 INFO mapreduce.Job: Task Id : attempt_1417632191545_0007_m_000000_0, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 13:57:05 INFO mapreduce.Job: Task Id : attempt_1417632191545_0007_m_000003_1, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 13:57:05 INFO mapreduce.Job: Task Id : attempt_1417632191545_0007_m_000002_1, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 13:57:05 INFO mapreduce.Job: Task Id : attempt_1417632191545_0007_m_000001_1, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 13:57:05 INFO mapreduce.Job: Task Id : attempt_1417632191545_0007_m_000000_1, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 13:57:12 INFO mapreduce.Job: Task Id : attempt_1417632191545_0007_m_000001_2, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 13:57:12 INFO mapreduce.Job: Task Id : attempt_1417632191545_0007_m_000003_2, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 13:57:12 INFO mapreduce.Job: Task Id : attempt_1417632191545_0007_m_000000_2, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 13:57:12 INFO mapreduce.Job: Task Id : attempt_1417632191545_0007_m_000002_2, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 13:57:19 INFO mapreduce.Job:  map 100% reduce 0%
14/12/09 13:57:20 INFO mapreduce.Job: Job job_1417632191545_0007 failed with state FAILED due to: Task failed task_1417632191545_0007_m_000001
Job failed as tasks failed. failedMaps:1 failedReduces:0

14/12/09 13:57:20 INFO mapreduce.Job: Counters: 12
    Job Counters
        Failed map tasks=13
        Killed map tasks=3
        Launched map tasks=16
        Other local map tasks=16
        Total time spent by all maps in occupied slots (ms)=80948
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=80948
        Total vcore-seconds taken by all map tasks=80948
        Total megabyte-seconds taken by all map tasks=82890752
    Map-Reduce Framework
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0
14/12/09 13:57:20 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
14/12/09 13:57:20 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 52.4248 seconds (0 bytes/sec)
14/12/09 13:57:20 INFO mapreduce.ImportJobBase: Retrieved 0 records.
14/12/09 13:57:20 ERROR tool.ImportTool: Error during import: Import job failed!

Firstly, I saw from "zookeeper.ZooKeeper: Client environment:java.library.path=/opt/cloudera/parcels/CDH-5.0.2-1.cdh5.0.2.p0.13/lib/hadoop/lib/native", which mean it connected to hadoop zookeeper, should I change it to hbase's zookeeper? if yes, how?

It seems there is timeout error and mainly the error: Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z

Could anyone please help me about this question?

Thanks in advance.

Cloudera Employee abe
Cloudera Employee
Posts: 109
Registered: ‎08-08-2013

Re: sqoop user

Could you run your command with --verbose right after "import"? Then copy paste the contents to this thread?
Highlighted
Explorer
Posts: 14
Registered: ‎12-03-2014

Re: sqoop user

 

I ran it as

sqoop import --verbose --connect jdbc:postgresql://dbtabase:port/table --table table --hbase-table tableFromSqoop --column-family cf --hbase-row-key id --hbase-create-table --username username --password passwaord

It seems no much differnence happens, here is the stack trace:

 

Spoiler
14/12/09 19:15:03 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/cloudera/parcels/CDH-5.0.2-1.cdh5.0.2.p0.13/lib/hadoop/lib/native
14/12/09 19:15:03 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
14/12/09 19:15:03 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
14/12/09 19:15:03 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
14/12/09 19:15:03 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
14/12/09 19:15:03 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-431.20.3.el6.x86_64
14/12/09 19:15:03 INFO zookeeper.ZooKeeper: Client environment:user.name=root
14/12/09 19:15:03 INFO zookeeper.ZooKeeper: Client environment:user.home=/root
14/12/09 19:15:03 INFO zookeeper.ZooKeeper: Client environment:user.dir=/root
14/12/09 19:15:03 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=trafficdata1.sis.uta.fi:2181 sessionTimeout=60000 watcher=hconnection-0x2aaf0838, quorum=trafficdata1.sis.uta.fi:2181, baseZNode=/hbase
14/12/09 19:15:03 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2aaf0838 connecting to ZooKeeper ensemble=trafficdata1.sis.uta.fi:2181
14/12/09 19:15:03 INFO zookeeper.ClientCnxn: Opening socket connection to server trafficdata1.sis.uta.fi/153.1.62.167:2181. Will not attempt to authenticate using SASL (unknown error)
14/12/09 19:15:03 INFO zookeeper.ClientCnxn: Socket connection established to trafficdata1.sis.uta.fi/153.1.62.167:2181, initiating session
14/12/09 19:15:03 INFO zookeeper.ClientCnxn: Session establishment complete on server trafficdata1.sis.uta.fi/153.1.62.167:2181, sessionid = 0x14a116fc00c5401, negotiated timeout = 60000
14/12/09 19:15:04 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
14/12/09 19:15:04 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=trafficdata1.sis.uta.fi:2181 sessionTimeout=60000 watcher=catalogtracker-on-hconnection-0x2aaf0838, quorum=trafficdata1.sis.uta.fi:2181, baseZNode=/hbase
14/12/09 19:15:04 INFO zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x2aaf0838 connecting to ZooKeeper ensemble=trafficdata1.sis.uta.fi:2181
14/12/09 19:15:04 INFO zookeeper.ClientCnxn: Opening socket connection to server trafficdata1.sis.uta.fi/153.1.62.167:2181. Will not attempt to authenticate using SASL (unknown error)
14/12/09 19:15:04 INFO zookeeper.ClientCnxn: Socket connection established to trafficdata1.sis.uta.fi/153.1.62.167:2181, initiating session
14/12/09 19:15:04 INFO zookeeper.ClientCnxn: Session establishment complete on server trafficdata1.sis.uta.fi/153.1.62.167:2181, sessionid = 0x14a116fc00c5402, negotiated timeout = 60000
14/12/09 19:15:04 INFO zookeeper.ZooKeeper: Session: 0x14a116fc00c5402 closed
14/12/09 19:15:04 INFO zookeeper.ClientCnxn: EventThread shut down
14/12/09 19:15:04 INFO client.RMProxy: Connecting to ResourceManager at trafficdata1.sis.uta.fi/153.1.62.167:8032
14/12/09 19:15:06 DEBUG db.DBConfiguration: Fetching password from job credentials store
14/12/09 19:15:06 INFO db.DBInputFormat: Using read commited transaction isolation
14/12/09 19:15:06 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN("id"), MAX("id") FROM "traffic" WHERE ( id<10000 )
14/12/09 19:15:06 DEBUG db.IntegerSplitter: Splits: [                           1 to                        9,971] into 4 parts
14/12/09 19:15:06 DEBUG db.IntegerSplitter:                            1
14/12/09 19:15:06 DEBUG db.IntegerSplitter:                        2,494
14/12/09 19:15:06 DEBUG db.IntegerSplitter:                        4,987
14/12/09 19:15:06 DEBUG db.IntegerSplitter:                        7,479
14/12/09 19:15:06 DEBUG db.IntegerSplitter:                        9,971
14/12/09 19:15:06 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound '"id" >= 1' and upper bound '"id" < 2494'
14/12/09 19:15:06 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound '"id" >= 2494' and upper bound '"id" < 4987'
14/12/09 19:15:06 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound '"id" >= 4987' and upper bound '"id" < 7479'
14/12/09 19:15:06 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound '"id" >= 7479' and upper bound '"id" <= 9971'
14/12/09 19:15:06 INFO mapreduce.JobSubmitter: number of splits:4
14/12/09 19:15:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1417632191545_0009
14/12/09 19:15:07 INFO impl.YarnClientImpl: Submitted application application_1417632191545_0009
14/12/09 19:15:07 INFO mapreduce.Job: The url to track the job: http://trafficdata1.sis.uta.fi:8088/proxy/application_1417632191545_0009/
14/12/09 19:15:07 INFO mapreduce.Job: Running job: job_1417632191545_0009
14/12/09 19:15:23 INFO mapreduce.Job: Job job_1417632191545_0009 running in uber mode : false
14/12/09 19:15:23 INFO mapreduce.Job:  map 0% reduce 0%
14/12/09 19:15:33 INFO mapreduce.Job: Task Id : attempt_1417632191545_0009_m_000001_0, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 19:15:33 INFO mapreduce.Job: Task Id : attempt_1417632191545_0009_m_000003_0, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 19:15:35 INFO mapreduce.Job: Task Id : attempt_1417632191545_0009_m_000002_0, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

14/12/09 19:15:35 INFO mapreduce.Job: Task Id : attempt_1417632191545_0009_m_000000_0, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 19:15:43 INFO mapreduce.Job: Task Id : attempt_1417632191545_0009_m_000001_1, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 19:15:43 INFO mapreduce.Job: Task Id : attempt_1417632191545_0009_m_000002_1, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 19:15:43 INFO mapreduce.Job: Task Id : attempt_1417632191545_0009_m_000003_1, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 19:15:43 INFO mapreduce.Job: Task Id : attempt_1417632191545_0009_m_000000_1, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 19:15:50 INFO mapreduce.Job: Task Id : attempt_1417632191545_0009_m_000003_2, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 19:15:50 INFO mapreduce.Job: Task Id : attempt_1417632191545_0009_m_000001_2, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 19:15:50 INFO mapreduce.Job: Task Id : attempt_1417632191545_0009_m_000000_2, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 19:15:50 INFO mapreduce.Job: Task Id : attempt_1417632191545_0009_m_000002_2, Status : FAILED
Error: org.postgresql.jdbc3g.Jdbc3gPreparedStatement.isClosed()Z
14/12/09 19:15:56 INFO mapreduce.Job:  map 100% reduce 0%
14/12/09 19:15:57 INFO mapreduce.Job: Job job_1417632191545_0009 failed with state FAILED due to: Task failed task_1417632191545_0009_m_000003
Job failed as tasks failed. failedMaps:1 failedReduces:0

14/12/09 19:15:57 INFO mapreduce.Job: Counters: 12
    Job Counters
        Failed map tasks=13
        Killed map tasks=3
        Launched map tasks=16
        Other local map tasks=16
        Total time spent by all maps in occupied slots (ms)=80655
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=80655
        Total vcore-seconds taken by all map tasks=80655
        Total megabyte-seconds taken by all map tasks=82590720
    Map-Reduce Framework
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0
14/12/09 19:15:57 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
14/12/09 19:15:57 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 52.465 seconds (0 bytes/sec)
14/12/09 19:15:57 INFO mapreduce.ImportJobBase: Retrieved 0 records.
14/12/09 19:15:57 DEBUG util.ClassLoaderStack: Restoring classloader: sun.misc.Launcher$AppClassLoader@45a8123b
14/12/09 19:15:57 ERROR tool.ImportTool: Error during import: Import job failed!

 

do you get any new ideas?

 

 

 

 

 

Announcements
New solutions