Member since
10-20-2017
26
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3486 | 10-27-2017 03:59 PM |
11-06-2017
11:39 PM
@Jay Kumar SenSharma Thanks for your help. I solved this problem as follows: Create a sqoop job for extract data and load in HDFS Using a hive job for execute a "load data inpath" for load data inside hive
... View more
11-06-2017
08:15 PM
@Jay Kumar SenSharma I find this error in log 2017-11-06 18:09:51,792 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for JobFinishedEvent
2017-11-06 18:09:51,792 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1509998820430_0003Job Transitioned from COMMITTING to SUCCEEDED
2017-11-06 18:09:51,796 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry
2017-11-06 18:09:51,796 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
2017-11-06 18:09:51,796 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified that shouldUnregistered is: true
2017-11-06 18:09:51,796 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
2017-11-06 18:09:51,796 INFO [Thread-68] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
2017-11-06 18:09:51,796 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
2017-11-06 18:09:51,810 INFO [Thread-68] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0
2017-11-06 18:09:52,093 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://ambari.4linux.com.br:8020/user/admin/.staging/job_1509998820430_0003/job_1509998820430_0003_1.jhist to hdfs://ambari.4linux.com.br:8020/mr-history/tmp/admin/job_1509998820430_0003-1509998963150-admin-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DFluxo_BigDatinh-1509998991790-1-0-SUCCEEDED-default-1509998978047.jhist_tmp
2017-11-06 18:09:52,227 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2017-11-06 18:09:52,235 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e29_1509998820430_0003_01_000002
2017-11-06 18:09:52,235 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2017-11-06 18:09:52,237 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1509998820430_0003_m_000000_0: Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
... View more
11-06-2017
06:49 PM
@Jay Kumar SenSharma All commands works fine, but when i try to execute a sqoop job in workflow manager i get a error] My Job Configuration:
... View more
11-06-2017
06:29 PM
@Jay Kumar SenSharma In my path "/usr/hdp/current/sqoop-client/lib/" i hasn't the driver postgres, i put inside the path and run the job but the same failed.
... View more
11-06-2017
06:22 PM
@Jay Kumar SenSharma I create a sqoop job in workflow manager and works, now i try to execute this command and the job failed, you can help me ? sqoop import --connect jdbc:postgresql://10.0.0.78:5433/cr4prd --username ambari --password xxxxxx --table tb_con_contatos --hive-database cr4prd --hive-import -m1
... View more
11-06-2017
06:15 PM
@Jay Kumar SenSharma My "job.properties" hasn't included the lib, i put the lib and the job failed. 😞
... View more
11-06-2017
06:08 PM
@Jay Kumar SenSharma yes, after i put jar i run the sharelibupdate
... View more
11-06-2017
05:59 PM
@Jay Kumar SenSharma I have the postgres jar inside te /user/oozie/share/lib/*/sqoop hdfs dfs -ls /user/oozie/share/lib/*/sqoop | grep sql
-rw-r--r-- 3 root hdfs 1208 2017-11-01 13:54 /user/oozie/share/lib/lib_20171030154340/sqoop/db.hsqldb.properties
-rw-r--r-- 3 root hdfs 1081 2017-11-01 13:55 /user/oozie/share/lib/lib_20171030154340/sqoop/db.hsqldb.script
-rw-r--r-- 3 oozie hdfs 643727 2017-10-30 15:46 /user/oozie/share/lib/lib_20171030154340/sqoop/hsqldb-1.8.0.7.jar
-rw-r--r-- 3 root hdfs 883898 2017-11-01 12:56 /user/oozie/share/lib/lib_20171030154340/sqoop/mysql-connector-java.jar
-rw-r--r-- 3 root hdfs 579785 2017-10-31 15:40 /user/oozie/share/lib/lib_20171030154340/sqoop/postgresql-9.2-1002.jdbc4.jar
-rw-r--r-- 3 root hdfs 446067 2017-11-06 16:01 /user/oozie/share/lib/lib_20171030154340/sqoop/postgresql-jdbc.jar
... View more
11-06-2017
05:44 PM
When i try to execute anyone command with user oozie i have thist error: ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: org.postgresql.Driver
java.lang.RuntimeException: Could not load db driver class: org.postgresql.Driver
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:875)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:763)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:246)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:328)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1853)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1653)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:225)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.main(Sqoop.java:243) But when i execute anyone command with root, the command finish with success.
... View more
11-06-2017
04:20 PM
@Jay Kumar SenSharma
my sharelib is: oozie admin -oozie http://ambari.4linux.com.br:11000/oozie -shareliblist[Available ShareLib]
hive
distcp
mapreduce-streaming
spark
oozie
hcatalog
hive2
sqoop
pig
spark_orig<br> but i have the postgres.jar inside sqoop folder
... View more