Member since
10-20-2017
26
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2304 | 10-27-2017 03:59 PM |
11-09-2017
06:47 PM
@Geoffrey Shelton Okot I try to execute a 200 jobs, but if one fail i want it pass condition for the next My file:arq.xml Thanks
... View more
11-09-2017
04:27 PM
@Jay Kumar SenSharma You can help me ? Thanks
... View more
11-09-2017
02:13 PM
Ok, thanks
... View more
11-09-2017
02:05 PM
@Aditya Sirna You can help me ? Thanks
... View more
11-09-2017
01:28 PM
@Agneesh Priyadarshi I don't know man, but i create a topic teaching a install ambari and yours services, but the topic is in portuguese Install Ambari Install Services
... View more
11-09-2017
12:33 PM
@Agneesh Priyadarshi You disabled your iptables(firewall) and disabled your selinux ?
... View more
11-09-2017
12:23 PM
@Agneesh Priyadarshi When you try to connect ssh root@hdpnn.capgemini.com you log in without password ? If yes you copy your id_rsa into configuration keys ?
... View more
11-09-2017
12:15 PM
@Agneesh Priyadarshi You configure your "SSH KEY" or your "HOSTNAME" correctly ?
... View more
11-09-2017
12:10 PM
@Anurag Mishra I solved this problem as follows: First i create a sharelib, after i just update sharelib and works usr/hdp/current/oozie-server/bin/oozie-setup.sh sharelib create -fs hdfs://ambari.4linux.com.br:8020 -locallib /usr/hdp/current/oozie-server/oozie-sharelib.tar.gz
oozie admin -oozie http://ambari.4linux.com.br:11000/oozie -sharelibupdate
oozie admin -oozie http://ambari.4linux.com.br:11000/oozie -shareliblist
... View more
11-09-2017
12:03 PM
I have a error when i try to send error that's the same path job ok
... View more
Labels:
- Labels:
-
Apache Oozie
11-06-2017
11:39 PM
@Jay Kumar SenSharma Thanks for your help. I solved this problem as follows: Create a sqoop job for extract data and load in HDFS Using a hive job for execute a "load data inpath" for load data inside hive
... View more
11-06-2017
08:15 PM
@Jay Kumar SenSharma I find this error in log 2017-11-06 18:09:51,792 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for JobFinishedEvent
2017-11-06 18:09:51,792 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1509998820430_0003Job Transitioned from COMMITTING to SUCCEEDED
2017-11-06 18:09:51,796 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry
2017-11-06 18:09:51,796 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
2017-11-06 18:09:51,796 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified that shouldUnregistered is: true
2017-11-06 18:09:51,796 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
2017-11-06 18:09:51,796 INFO [Thread-68] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
2017-11-06 18:09:51,796 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
2017-11-06 18:09:51,810 INFO [Thread-68] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0
2017-11-06 18:09:52,093 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://ambari.4linux.com.br:8020/user/admin/.staging/job_1509998820430_0003/job_1509998820430_0003_1.jhist to hdfs://ambari.4linux.com.br:8020/mr-history/tmp/admin/job_1509998820430_0003-1509998963150-admin-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DFluxo_BigDatinh-1509998991790-1-0-SUCCEEDED-default-1509998978047.jhist_tmp
2017-11-06 18:09:52,227 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2017-11-06 18:09:52,235 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e29_1509998820430_0003_01_000002
2017-11-06 18:09:52,235 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2017-11-06 18:09:52,237 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1509998820430_0003_m_000000_0: Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
... View more
11-06-2017
06:49 PM
@Jay Kumar SenSharma All commands works fine, but when i try to execute a sqoop job in workflow manager i get a error] My Job Configuration:
... View more
11-06-2017
06:29 PM
@Jay Kumar SenSharma In my path "/usr/hdp/current/sqoop-client/lib/" i hasn't the driver postgres, i put inside the path and run the job but the same failed.
... View more
11-06-2017
06:22 PM
@Jay Kumar SenSharma I create a sqoop job in workflow manager and works, now i try to execute this command and the job failed, you can help me ? sqoop import --connect jdbc:postgresql://10.0.0.78:5433/cr4prd --username ambari --password xxxxxx --table tb_con_contatos --hive-database cr4prd --hive-import -m1
... View more
11-06-2017
06:15 PM
@Jay Kumar SenSharma My "job.properties" hasn't included the lib, i put the lib and the job failed. 😞
... View more
11-06-2017
06:08 PM
@Jay Kumar SenSharma yes, after i put jar i run the sharelibupdate
... View more
11-06-2017
05:59 PM
@Jay Kumar SenSharma I have the postgres jar inside te /user/oozie/share/lib/*/sqoop hdfs dfs -ls /user/oozie/share/lib/*/sqoop | grep sql
-rw-r--r-- 3 root hdfs 1208 2017-11-01 13:54 /user/oozie/share/lib/lib_20171030154340/sqoop/db.hsqldb.properties
-rw-r--r-- 3 root hdfs 1081 2017-11-01 13:55 /user/oozie/share/lib/lib_20171030154340/sqoop/db.hsqldb.script
-rw-r--r-- 3 oozie hdfs 643727 2017-10-30 15:46 /user/oozie/share/lib/lib_20171030154340/sqoop/hsqldb-1.8.0.7.jar
-rw-r--r-- 3 root hdfs 883898 2017-11-01 12:56 /user/oozie/share/lib/lib_20171030154340/sqoop/mysql-connector-java.jar
-rw-r--r-- 3 root hdfs 579785 2017-10-31 15:40 /user/oozie/share/lib/lib_20171030154340/sqoop/postgresql-9.2-1002.jdbc4.jar
-rw-r--r-- 3 root hdfs 446067 2017-11-06 16:01 /user/oozie/share/lib/lib_20171030154340/sqoop/postgresql-jdbc.jar
... View more
11-06-2017
05:44 PM
When i try to execute anyone command with user oozie i have thist error: ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: org.postgresql.Driver
java.lang.RuntimeException: Could not load db driver class: org.postgresql.Driver
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:875)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:763)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:246)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:328)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1853)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1653)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:225)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.main(Sqoop.java:243) But when i execute anyone command with root, the command finish with success.
... View more
11-06-2017
04:20 PM
@Jay Kumar SenSharma
my sharelib is: oozie admin -oozie http://ambari.4linux.com.br:11000/oozie -shareliblist[Available ShareLib]
hive
distcp
mapreduce-streaming
spark
oozie
hcatalog
hive2
sqoop
pig
spark_orig<br> but i have the postgres.jar inside sqoop folder
... View more
11-06-2017
04:07 PM
When i try to execute a shell job with sqoop or sqoop job in workflow manager i have a error: I am try to import tables from postgres to hive 2017-11-06 13:58:46,622 WARN ShellActionExecutor:523 - SERVER[ambari.4linux.com.br] USER[admin] GROUP[-] TOKEN[] APP[Fluxo_BigDatinha] JOB[0000003-171106133302301-oozie-oozi-W] ACTION[0000003-171106133302301-oozie-oozi-W@Verifica_Sujeira_HDFS] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1] My Shell Script: sqoop import --connect jdbc:postgresql://10.0.0.78:5433/cr4prd --username ambari --password xxxxxxx --table tb_con_contatos --target-dir /user/admin/teste2 My Cluster Configuration: I am running in /user/admin/sqoop and inside this directory have a paste with lib:
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Sqoop
10-27-2017
03:59 PM
I delete my paste /user/oozie/share/lib/*/ and recreate te sharelib again
... View more
10-26-2017
07:00 PM
When i try to execute anyone type of job in workflow manager i have the same error: 017-10-26 16:49:46,682 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1509041364763_0004_000002
2017-10-26 16:49:47,060 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-10-26 16:49:47,098 INFO [main] org.apache.hadoop.security.SecurityUtil: Updating Configuration
2017-10-26 16:49:47,110 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2017-10-26 16:49:47,110 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 4 cluster_timestamp: 1509041364763 } attemptId: 2 } keyId: 218153851)
2017-10-26 16:49:47,418 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN, Service: 10.0.0.73:8050, Ident: (owner=admin, renewer=oozie mr token, realUser=oozie, issueDate=1509043779741, maxDate=1509648579741, sequenceNumber=13, masterKeyId=36)
2017-10-26 16:49:47,439 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter
2017-10-26 16:49:47,441 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter not found
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2273)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:524)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:501)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1640)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:501)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:287)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1598)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1595)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1526)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2241)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2265)
... 12 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2147)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2239)
... 13 more
... View more
Labels:
- Labels:
-
Apache Oozie
10-20-2017
01:59 PM
@Sandeep Nemuri In the host my two datanodes is ok, but in the summary the ambari doesn't recognize the datanodes are lives
... View more
10-20-2017
01:46 PM
Hello, When i install HDFS by ambari my capacity of HDFS is zero. hdfs dfsadmin -report Configured Capacity: 0 (0 B) Present Capacity: 0 (0 B) DFS Remaining: 0 (0 B) DFS Used: 0 (0 B) DFS Used%: NaN% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0
... View more
Labels:
- Labels:
-
Apache Hadoop