Hi All , In my project , where I am trying to read log files and process it in spark , I am using NiFi to read the file from tomcat log folder location and copy it to my Edge node in my hadoop cluster. But the problem is that my Application (for which I am processing log files) is in cluster environment and in all 4 tomcat cluster log file names are same. So what I want to do , getFTP will get the log file from app server location , then data will flow into a updateAttribite processor , which will append the server and cluster identification(just something like server1Cluste1 or server2Cluster1) with the file name and then putFile will store the log file in local file system with new name. Which I will process in spark job. Can any one help me out for configuration of updateAttribute in my case? is there anything in updateAttribute by which I can identify from which server I am getting this file and depending on that can I change the file name to putFile? Any help will be highly appreciated Thanks in advance ... View more

Hi All, My rolling log file pattern is something like this /my/path/directory/my-app-2017-09-06.log /my/path/directory/my-app-2017-09-07.log /my/path/directory/my-app-2017-09-08.log Can you one suggest what can set for property in NIFI for a tailFile processor to read out those. Please note I have old file also and some different file also , but I want to read file wit this specific file name and today onward only , not the old file. I read the doc available for NiFi in website , but not clear me ... Can any one please help me out to configure tailFile with this file pattern. Any help will be highly appreciated actually I stuck on this issue for last 5 days .... ... View more

Hi , In my project I am using Nifi to read log file from tomcat and process those data in a spark application , after that insert those process data in DB. But my problem is that , at app server level , I have 4 tomcat cluster(4 different log file) in 2 different box and I have mark out which data is from which cluster at spark level. In my present set up I have 2 tailFile processor in Nifi which pointing to single outport , in per box but not able to identify which data is from which cluster at spark level. Is there any option in tailFile processor to add some suffix or prefix or file name(or any attribute) in each record ? so that I can identify each record is coming from which cluster and persist in db in that way? Any help will be highly appreciated .... Thanks in advance ... View more

Hi All, I have a sample table(stuends1) in HIVE which I want to connect from Spark using JDBC (as Hive is not in same cluster). I was just trying with following code... def main(args: Array[String]): Unit = { //Class.forName("org.apache.hive.jdbc.HiveDriver").newInstance() val conf = new SparkConf().setAppName("SOME APP NAME").setMaster("local[*]") val sc = new SparkContext(conf) val spark = SparkSession .builder() .appName("Spark Hive Example") .getOrCreate() val jdbcDF = spark.read .format("jdbc") .option("url", "jdbc:hive2://34.223.237.55:10000") .option("dbtable", "students1") .option("user", "hduser") .option("password", "hadoop") //.option("driver", "org.apache.hadoop.hive.jdbc.HiveDriver") .load() println("able to connect------------------") jdbcDF.show jdbcDF.printSchema() jdbcDF.createOrReplaceTempView("std") val sqlDF = spark.sql("select * from std") println("Start println-----") spark.sqlContext.sql("select * from std").collect().foreach(println) println("end println-----") sqlDF.show(false) } I tried in multiple ways but all the time its showing table structure with column name only. Like ... +--------------+-------------+-------------+ |students1.name|students1.age|students1.gpa| +--------------+-------------+-------------+ +--------------+-------------+-------------+ But not data, but able to get data when trying to with dbeaver from my local with SQL query. From spark, jdbcDF.printSchema() also showing proper schema , so I guess no issue with connection. I am using spark 2.1.1 with HIVE 1.2.1. My sbt.build file is like this .... libraryDependencies ++= Seq( "log4j" % "log4j" % "1.2.17", "org.apache.spark" % "spark-core_2.11" % "2.1.1" , "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.2", "org.apache.spark" % "spark-hivecontext-compatibility_2.10" % "2.0.0-preview", "org.apache.spark" % "spark-sql_2.11" % "2.1.1" , "org.apache.spark" % "spark-hive_2.10" % "2.1.1", "org.apache.hive" % "hive-jdbc" % "1.2.1" } can any one suggest why I am not getting any output of show(). Thanks in advance... ... View more