About pradhumankumar_

DanieleConsalvo · ‎02-02-2022

Sorry, there is a mode to disable polling option? If i want to load some files in example 3 files and only them and after i want load this to a next rerun of the job as i can to disable the polling function to prevent that the processor is always in waiting or in listening!? Regards, Daniele

pradhumankumar_ · ‎04-17-2017

Thank you Jay for your help !!

cstanca · ‎04-07-2017

@Pradhuman Gupta Apache Spark cannot do that out of the box. But you might be looking for is somemiddlewarethat can interface with Apache Spark and run, submit and manage jobs for you. Livy - REST server with extensive language support (Python, R, Scala), ability to maintain interactive sessions and object sharing. spark-jobserver - A simple Spark as a Service which supports objects sharing using so called named objects. JVM only. Mist - A service for exposing Spark analytical jobs and machine learning models as realtime, batch or reactive web services. Apache Toree - IPython protocol based middleware for interactive applications. Hortonworks recommends Livy. Also, read the last comment at: https://issues.apache.org/jira/browse/SPARK-2243 allowMultipleContexts has a very limited use for test only and can lead to the error you see.

deepesh1 · ‎04-06-2017

A simple Spark1 Java application to show a list of tables in Hive Metastore is as follows: import org.apache.spark.SparkContext; import org.apache.spark.SparkConf; import org.apache.spark.sql.hive.HiveContext; import org.apache.spark.sql.DataFrame; public class SparkHiveExample { public static void main(String[] args) { SparkConf conf = new SparkConf().setAppName("SparkHive Example"); SparkContext sc = new SparkContext(conf); HiveContext hiveContext = new org.apache.spark.sql.hive.HiveContext(sc); DataFrame df = hiveContext.sql("show tables"); df.show(); } } Note that Spark pulls metadata from Hive metastore and also uses hiveql for parsing queries but the execution of queries as such happens in the Spark execution engine.

rchoksi · ‎03-30-2017

Similar to https://community.hortonworks.com/questions/19897/apache-spark-error-unable-to-load-native-hadoop-li.html

pradhumankumar_ · ‎03-03-2017

@Matt Clarke Hi Matt, Thank you I understood it now, I will go through documentation to learn more on this. Thanks

bbende · ‎02-24-2017

Can you take a thread dump and provide the output here? ./bin/nifi.sh dump /path/to/output/dump.txt

pradhumankumar_ · ‎02-24-2017

Thanks Arun, you are right. I did it using TailFile. @Arun A K

MattWho · ‎02-22-2017

@Pradhuman Gupta Which "Protocol" are you using in your PutSplunk processor. There is no assurance or delivery if you are using UDP. With TCP protocol there is confirmed delivery. You could use NiFi's provenance to track the FlowFile's processed by the PutSplunk processor. This will allow you to get the details on FlowFiles that have "SEND" provenance events associated to them. Thanks, Matt

pradhumankumar_ · ‎02-23-2017

Thanks @Bryan Bende @Timothy Spann, @ozhurakousky foryour reply. It was some configuration issue. While trying to put file into Splunk, I was using web-port (8081 in my case) of splunk in configuration of PutSplunk. When I pointed my PutSplunk configuration to TCP port of Splunk(In Splunk setting go to Data Inputs -> Click on TCP and enter details as instructed to create a new TCP input port of Splunk) it started working properly.

Online	Offline
Last Visited	‎08-10-2018 05:19 AM

Member Since	‎01-30-2017 08:22 AM
Last Visited	‎08-10-2018 05:19 AM
Posts	49
Kudos received	3

Cloudera Community

Re: While running Apache NiFi and using its PutSpl...

Re: What is Polling Interval, Max Select , Run Sch...

Re: Unable to run HDP in VMWare

Re: Does Spark allow multiple context to run in

Re: How to connect and run Hive query from Apache ...

Re: Why I am getting : The short-circuit local re...

Re: Can we add custom attribute in Apache NiFi on ...

Re: Why PutSplunk stopped picking the data from Qu...

Re: How to put running log of Apahce NiFi into Spl...

Re: How to track if NiFi PutSplunk processor is se...

Re: While running Apache NiFi and using its PutSpl...