@dconnolly My pleasure. Would you mind accepting the answer and up voting? ... View more

There is certainly no reason why you could not use Nifi for one off, movement of small amounts of data or for repeated enrichment of events from an RDBMS. But just as you would not use a hand axe to chop down a Redwood, you generally would not use an event/message stream processing tool to handle a large static ETL job such as a database export. It's simply a matter of using the tool for the purpose that it was designed for. ... View more

@Sridhar Babu M Glad it worked out. Would you mind accepting this answer and the one from the other thread? https://community.hortonworks.com/questions/24518/spark-sql-query-to-modify-values.html ... View more

<code>groupId: com.databricks artifactId: spark-csv_2.10 version: 1.4.0 @Sridhar Babu M Make sure that you add the dependency in Zeppelin %dep z.load("com.databricks:spark-csv_2.10:1.4.0") or spark-shell --packages com.databricks:spark-csv_2.10:1.4.0 ... View more

@dconnolly The simplest approach is to build a REST web service on a servlet container like Tomcat or Jetty. Strictly speaking, you could publish that web service anywhere, however, I would recommend that you leverage the existing resources available in the Hadoop cluster use Slider to run the web service on Yarn. Try this: Create a web service and build it as a runnable jar. Put that jar into a linux docker container and then create the configuration files need to run the docker container with slider. https://slider.incubator.apache.org/docs/slider_specs/application_pkg_docker.html http://www.slideshare.net/hortonworks/docker-on-slider-45493303 Use Slider to start the docker container on Yarn and secure the listener endpoint with Knox. This would allow you to leverage the resources of the cluster, manage the synch service resources with Yarn, and provide security for the API endpoint with Knox. ... View more

@Sridhar Babu M If you actually need to change the value in the file then you will need to export the resulting Data Frame to file. The save function that is part of DF class creates a files for each partition. If you need a single file you convert back to an RDD and use coalesce(1) to get everything down to a single partition so you get one file. Make sure that you add the dependency in Zeppelin %dep z.load("com.databricks:spark-csv_2.10:1.4.0") or spark-shell --packages com.databricks:spark-csv_2.10:1.4.0 import org.apache.spark.sql.SQLContext import org.apache.spark.sql.SaveMode case class Person(name: String, age: Int) var personRDD = sc.textFile("/user/spark/people.txt") var personDF = personRDD.map(x=>x.split(",")).map(x=>Person(x(0),(x(1).trim.toInt))).toDF() personDF.registerTempTable("people") var personeDF = sqlContext.sql("SELECT * FROM people") var agedPerson = personDF.map(x=>if(x.getAs[String]("name")=="Justin"){Person(x.getAs[String]("name"), x.getAs[Int]("age")+2)}else{Person(x.getAs[String]("name"), x.getAs[Int]("age"))}).toDF() agedPerson.registerTempTable("people") var agedPeopleDF = sqlContext.sql("SELECT * FROM people") agedPeopleDF.show agedPeopleDF.select("name", "age").write.format("com.databricks.spark.csv").mode(SaveMode.Overwrite).save("agedPeople") var agedPeopleRDD = agedPeopleDF.rdd agedPeopleRDD.coalesce(1).saveAsTextFile("agedPeopleSingleFile") ... View more