Member since
10-01-2015
3933
Posts
1150
Kudos Received
374
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3362 | 05-03-2017 05:13 PM | |
2792 | 05-02-2017 08:38 AM | |
3068 | 05-02-2017 08:13 AM | |
3002 | 04-10-2017 10:51 PM | |
1510 | 03-28-2017 02:27 AM |
02-27-2017
02:16 AM
1 Kudo
@Osoeng Kwon I tested your imports and it does indeed look like ContentExchange is no longer in the package org.eclipse.jetty.client. My maven search prompted me to the following version that has both classes. <dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-client</artifactId>
<version>7.6.13.v20130916</version>
<type>jar</type>
</dependency>
you might want to reconsider using it as it's from 2013, there are many version of jetty-client available since then. Also there's something wrong with your code, I ran it in Zeppelin and it didn't work, most likely I'm missing some of the imports you did not copy/paste. I loaded the dependency using the interpreter configuration page, under Spark, click edit and add the dependency in the dependencies section (the screenshot below shows how it looks in my configuration), using z.load as I've shown earlier is deprecated. I ran all of your code in Zeppelin after that and I received the following: import org.eclipse.jetty.client.ContentExchange
import org.eclipse.jetty.client.HttpClient
createExchangeForSign: (client: org.eclipse.jetty.client.HttpClient, sign: String)(String, org.eclipse.jetty.client.ContentExchange)
<console>:29: error: not found: type CallLog
def readExchangeCallLog(mapper: ObjectMapper, exchange: ContentExchange): Array[CallLog] = {
^
<console>:29: error: not found: type ObjectMapper
def readExchangeCallLog(mapper: ObjectMapper, exchange: ContentExchange): Array[CallLog] = {
... View more
02-26-2017
04:26 PM
1 Kudo
@Osoeng Kwon I think the issue here is that ContentExchange is not in org.eclipse.jetty I see a similar class in org.mortbay.jetty http://grepcode.com/file/repo1.maven.org/maven2/org.eclipse.jetty/jetty-client/8.1.14.v20131031/org/eclipse/jetty/client/ContentExchange.java confirm that's what you need, then you can load it in Zeppelin with command below and whatever the new maven repo for the class ContentExchange you're looking for. %spark.dep
z.reset()
z.load("org.eclipse.jetty:jetty-client:9.4.2.v20170220")
%spark
import org.eclipse.jetty.client.ContentExchange
import org.eclipse.jetty.client.HttpClient
... View more
02-26-2017
03:45 PM
@Ali Mohammadi Shanghoshabad it is important that you click the checkbox for capture-output, your xml when you preview should look like so Here is that checkbox below, look for it in the shell action.
... View more
02-26-2017
03:37 PM
1 Kudo
@Param NC here's how I got it to work on my cluster export HADOOP_CONF_DIR=/etc/hadoop/conf
/usr/hdp/current/spark-client/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --executor-memory 1G --num-executors 3 /usr/hdp/current/spark-client/lib/spark-examples*.jar 100
... View more
02-26-2017
03:12 PM
2 Kudos
@Param NC please take a look at our documentation http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_spark-component-guide/content/ch_developing-spark-apps.html for general knowledge here's an example of doing it in YARN mode, from: http://spark.apache.org/docs/1.6.2/submitting-applications.html usually HADOOP_CONF_DIR points to /etc/hadoop/conf on HDP distribution. That directory contains core-site.xml, yarn-site.xml, hdfs-site.xml etc. export HADOOP_CONF_DIR=XXX
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode cluster \ # can be client for client mode
--executor-memory 20G \
--num-executors 50 \
/path/to/examples.jar \
1000
... View more
02-26-2017
01:16 PM
@Ali Mohammadi Shanghoshabad Sandbox 2.5 has an older version of the tool I'm using. Click on create workflow on your screen to the right and see if creating coordinators is possible. If you have a specific coordinator issue, open it as a new question and I'll try to help. Just wait for Apache Ambari 2.5 to get this version released.
... View more
02-25-2017
10:34 PM
You can use with filter df2 = df1.filter($"Status" === 2 || $"Status" === 3)
... View more
02-25-2017
06:20 PM
Excellent, please mark the answer as accepted to close the thread.
... View more
02-25-2017
05:15 PM
You can certainly achieve that, or contribute a patch but it defeats the purpose. If you need more explanation, ping me on hipchat. Don't forget to accept the answer!
... View more
02-25-2017
05:04 PM
1 Kudo
though you can achieve what you're planning it really goes against what both these products are designed to do. Oozie uses coordinator and bundle to schedule their workflows, Nifi is about always flowing data without start and finish. Oozie does have rest API you can invoke to start a workflow if you intend to go your route but I would first ask what user is trying to do. If you need a way to check whether data landed in HDFS before executing Oozie workflow, look at the following coordinator examples by Yahoo, specifically https://github.com/yahoo/oozie/wiki/Oozie-Coord-Use-Cases#triggering-coordinator-jobs-when-data-directory-is-available Also suggest you read my articles https://community.hortonworks.com/articles/85354/apache-ambari-workflow-manager-view-for-apache-ooz-5.html and https://community.hortonworks.com/articles/85361/apache-ambari-workflow-manager-view-for-apache-ooz-6.html
... View more