About it_duanxiong

it_duanxiong · ‎10-10-2018

I want this function too.Current version do not have this function to finish this.

it_duanxiong · ‎10-08-2018

it_duanxiong · ‎09-25-2018

how to remote debug oozie on spark2?

it_duanxiong · ‎09-17-2018

oozie can successed schedule the other mission , i.e. mapreduce?

it_duanxiong · ‎09-16-2018

I agree with you!you are right about this problem,First,have to try use spark-sumit to run this App,Then can use oozie go to schedule.

it_duanxiong · ‎09-14-2018

Just need to add this tag in wf.xml <property> <name>oozie.launcher.mapred.child.java.opts</name> <value>-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8000</value> </property>

it_duanxiong · ‎09-13-2018

Hi,friend.I want to remote debug oozie workflow,but I don't know how to do it.I need some help.Thank you very much. I use oozie.sh jpda start,and use my IDEA connected,but i can't submit wf and the oozie process is on ,so somebody can tell me how to do next?

it_duanxiong · ‎09-06-2018

run：oozie admin -oozie http://localhost:11000/oozie -status have the question

it_duanxiong · ‎09-06-2018

wrong log: Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Authentication failed, status: 302, message: Found oozie-site.xml: <property> <name>oozie.service.HadoopAccessorService.kerberos.enabled</name> <value>true</value> </property> <property> <name>local.realm</name> <value>HADOOP.COM</value> </property> <property> <name>oozie.service.HadoopAccessorService.keytab.file</name> <value>/usr/lib/cluster001/oozie-4.3.0/conf/oozie-http.keytab</value> </property> <property> <name>oozie.service.HadoopAccessorService.kerberos.principal</name> <value>oozie/ds1@HADOOP.COM</value> </property> <property> <name>oozie.authentication.type</name> <value>kerberos</value> </property> <property> <name>oozie.authentication.kerberos.principal</name> <value>HTTP/ds1@HADOOP.COM</value> </property> <property> <name>oozie.authentication.kerberos.name.rules</name> <value>DEFAULT</value> </property> keytab:(ds1上运行oozie-server，I have test this tap,It is successd) [root@ds1 conf]# klist -e -k -t oozie-http.keytab Keytab name: FILE:oozie-http.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 1 09/05/18 14:56:16 oozie/ds1@HADOOP.COM (aes256-cts-hmac-sha1-96) 1 09/05/18 14:56:16 oozie/ds1@HADOOP.COM (aes128-cts-hmac-sha1-96) 2 09/05/18 14:56:16 HTTP/ds1@HADOOP.COM (aes256-cts-hmac-sha1-96) 2 09/05/18 14:56:16 HTTP/ds1@HADOOP.COM (aes128-cts-hmac-sha1-96) 1 09/06/18 19:43:58 oozie/ds1@HADOOP.COM (aes256-cts-hmac-sha1-96) 1 09/06/18 19:43:58 oozie/ds1@HADOOP.COM (aes128-cts-hmac-sha1-96) 2 09/06/18 19:43:58 HTTP/ds1@HADOOP.COM (aes256-cts-hmac-sha1-96) 2 09/06/18 19:43:58 HTTP/ds1@HADOOP.COM (aes128-cts-hmac-sha1-96)

it_duanxiong · ‎08-24-2018

This problem have to add spark.yarn.jars in sparkopts.and It have to start with"hdfs://"

it_duanxiong · ‎08-24-2018

It's my linux problem.

it_duanxiong · ‎08-23-2018

Hi,I have the same question,same error with one 1 action,I try your method,but it didn't work!

it_duanxiong · ‎08-20-2018

version: oozie:4.3 spark2.1 2018-08-20 19:33:23,998 [uber-SubtaskRunner] ERROR org.apache.spark.SparkContext - Error initializing SparkContext. java.lang.NullPointerException at scala.collection.mutable.ArrayOps$ofRef$.newBuilder$extension(ArrayOps.scala:190) at scala.collection.mutable.ArrayOps$ofRef.newBuilder(ArrayOps.scala:186) at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:246) at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259) at scala.collection.mutable.ArrayOps$ofRef.filter(ArrayOps.scala:186) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6.apply(Client.scala:531) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6.apply(Client.scala:527) at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74) at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:527) at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:868) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:170)

it_duanxiong · ‎08-20-2018

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] log4j:ERROR Could not find value for key log4j.appender.CLA log4j:ERROR Could not instantiate appender named "CLA". log4j:ERROR Could not find value for key log4j.appender.CLA log4j:ERROR Could not instantiate appender named "CLA". Using properties file: null When I use oozie on spark2,I run example "SparkFileCopy",It have some problem.Thank you. version: oozie 4.3 spark 2.1

it_duanxiong · ‎08-18-2018

Yaeh,I don't konw why,I change another cluster,every thing is fine.so thank you very much!

it_duanxiong · ‎08-18-2018

Thank you very much! It is a problem abour jar conflict.

it_duanxiong · ‎08-18-2018

When use spark2.1,We have to add oozie-sharelib-spark-4.2.0.jar in spark2

it_duanxiong · ‎08-18-2018

When I use oozie on spark21.It hava one error: Library directory '/data/hadoop/usercache/root/appcache/application_1533889422422_0065/container_1533889422422_0065_01_000001/assembly/target/scala-2.11/jars' does not exist; make sure Spark is built. Thanks!

it_duanxiong · ‎08-17-2018

when I use oozie to schedule spark2 on yarn.I have e pproblem : Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, null java.lang.NullPointerException at scala.collection.mutable.ArrayOps$ofRef$.newBuilder$extension(ArrayOps.scala:190) at scala.collection.mutable.ArrayOps$ofRef.newBuilder(ArrayOps.scala:186) at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:246) at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259) at scala.collection.mutable.ArrayOps$ofRef.filter(ArrayOps.scala:186) at org.apache.spark.deploy.yarn.Client$anonfun$prepareLocalResources$6.apply(Client.scala:531) at org.apache.spark.deploy.yarn.Client$anonfun$prepareLocalResources$6.apply(Client.scala:527) at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74) at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:527) at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:868) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:170) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156) at org.apache.spark.SparkContext.<init>(SparkContext.scala:509) at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)

it_duanxiong · ‎08-16-2018

@Sandeep Nemuri,Thank you very much!when I first install oozie,I use mysql.I think it caused by mysql.So i try Derby,But it stay slow!

it_duanxiong · ‎08-16-2018

When I use oozie，I hava install oozie success，But this web loading is very slow，I don‘t know why .Oozie's other function can work,and this web can work too,but web is very slow .Anyone know why this happening?Thank you very much.(or tell me where has Installation Steps and I will go to read(no ambari,just oozie and hadoop))

it_duanxiong · ‎08-16-2018

When I run oozie's example map-reduce.In yarn application,It shows this mission had successd,But in oozie ,this mission always running？I don't know how to resolve it.Anyonr has some idea about this question,Thank you very much!

it_duanxiong · ‎07-25-2017

When I do this,I found many source files are synthesized into a single file，I don't want like this,I just want to realize one file to another file,one to one.So anyone have function to realize this,Thank you very much!

it_duanxiong · ‎06-07-2017

Can you describe this question more clear?I can't understand this.

it_duanxiong · ‎06-07-2017

Hi,when I register a Hive udf in FunctionRegistry.java,I fund many UDF has been registed in this file,So how i can get the registry function's name list?Like this.So in the fact have the command to realize this? thank you very much.

it_duanxiong · ‎06-07-2017

thank you !I will try this.

it_duanxiong · ‎06-07-2017

Yeah，maybe because the network problem.so i summit the same question twice,sorry!

it_duanxiong · ‎06-06-2017

Ok,thank you very much.I thought that in the fact there is a way to remove it,just I don't know。

it_duanxiong · ‎06-06-2017

Thank you very much,I just want to change the replications of files that have upload to HDFS.So yesterdy,I found that one command can realize that.So thank you very much!

it_duanxiong · ‎06-04-2017

I just want to remove the record!thank you!

Online	Offline
Last Visited	‎10-12-2018 02:45 AM

Date Registered	‎10-03-2016 07:23 AM
Last Visited	‎10-12-2018 02:45 AM
Total Messages Posted	51
Total Kudos Received	1

Re: Hi,friend.I want to remote debug oozie workflo...

Re: oozie and spark2 ERROR

Re: spark2 job on oozie

Re: Limit Oozie workflow or node execution time

Hi,How to use IDEA local debug oozie resourceCode...

how to remote debug oozie on spark2?

Re: oozie workflow is stuck at running without err...

Re: oozie workflow is stuck at running without err...

Re: Hi,friend.I want to remote debug oozie workflo...

Hi,friend.I want to remote debug oozie workflow,bu...

Re: oozie4.3 with kerberos: I use oozie in kerbero...

oozie4.3 with kerberos: I use oozie in kerberos h...

Re: oozie and spark2 ERROR

Re: Oozie can't display the graph cause number of ...

Re: Oozie can't display the graph cause number of ...

oozie and spark2 ERROR

Error in oozie on spark2

Re: One question about Oozie:Web show

Re: spark2 job on oozie

Re: spark2 job on oozie

oozie on spark2.1 Error: make sure Spark is built.

spark2 job on oozie

Re: One question about Oozie:Web show

One question about Oozie:Web show

Oozie have a wrong info :Error : E0303: Invalid pa...

About flume,I want to use flume to archieve copy f...

Re: Webhdfs REST API - Search for a filename or di...

Hi,when I register a Hive udf in FunctionRegistry....

Re: Hi,How to make hadoop to read the configuratio...

Re: Hi,How to make hadoop to read the configuratio...

Re: Hi,I want to remove hdfs dead datanode in WebU...

Re: Hi,How to make hadoop to read the configuratio...

Hi,I want to remove hdfs dead datanode in WebUi,bu...