https://community.cloudera.com/t5/Support-Questions/Integration-between-Apache-Pig-Apache-Nifi-and-Apache-Spark/m-p/145081#M107653 <P>hello Timothy</P><P>There are mutilple ways to integrate these 3 services. As a starting point Nifi will probably be your ingestion flow. During this flow you could</P><P>- put your data to kafka and have spark read from it</P><P>- push your nifi data to spark: <A href="https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark">https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark</A></P><P>- you could use and execute script processor and start a pig job</P><P>In summary you can have a push and forget connection, you can have a push to service and pick in next flow approach, or even execute in processor as corner case maybe</P><P>hope this shares some insight</P> Fri, 17 Jun 2016 22:30:35 GMT nmaillard1 2016-06-17T22:30:35Z https://community.cloudera.com/t5/Support-Questions/Integration-between-Apache-Pig-Apache-Nifi-and-Apache-Spark/m-p/145080#M107652 <P>What are the various ways to integrate Apache Pig, Nifi and Spark?</P><P>I know I can connect some with Kafka or via files.</P> Fri, 17 Jun 2016 22:23:48 GMT https://community.cloudera.com/t5/Support-Questions/Integration-between-Apache-Pig-Apache-Nifi-and-Apache-Spark/m-p/145080#M107652 TimothySpann 2016-06-17T22:23:48Z https://community.cloudera.com/t5/Support-Questions/Integration-between-Apache-Pig-Apache-Nifi-and-Apache-Spark/m-p/145081#M107653 <P>hello Timothy</P><P>There are mutilple ways to integrate these 3 services. As a starting point Nifi will probably be your ingestion flow. During this flow you could</P><P>- put your data to kafka and have spark read from it</P><P>- push your nifi data to spark: <A href="https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark">https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark</A></P><P>- you could use and execute script processor and start a pig job</P><P>In summary you can have a push and forget connection, you can have a push to service and pick in next flow approach, or even execute in processor as corner case maybe</P><P>hope this shares some insight</P> Fri, 17 Jun 2016 22:30:35 GMT https://community.cloudera.com/t5/Support-Questions/Integration-between-Apache-Pig-Apache-Nifi-and-Apache-Spark/m-p/145081#M107653 nmaillard1 2016-06-17T22:30:35Z