I have an FTP location where I must grab specific files, *.tar, from within named sub directories and only those sub directories. The layout is like so: path/conf1/conf1.tar path/conf2/conf2.tar path/conf3/conf3.tar path/support/support.tar   I only want tar files from path/conf*/. Is this possible using Path Filter Regex or some combo of properties? I do not want to look into support/ at all. In fact some directories I do not have permissions to look at so I get those permission exceptions. How to limit to only conf*/ folders? Thanks ... View more

I am trying to simply take the content from a file I break up with SplitProcessor and send the line of text to a text FlowFile and convert that line to JSON and send to another flowFile in onTrigger(). I have tried cloning/creating new session but I am missing something. // Snippets onTrigger() { ... // read flowFile text line and send over to "TEXT" relationship (works fine) String content = new String(buffer,...); ff = session.write(ff, new OutputStream...); session.transfer(ff, TEXT); session.remove(ff); // now convert that string to JSON and send to JSON ff2 = session.create(); String jsonString = toJson(content); // write to ff2 ff2 = session.write(ff2, ... session.transfer(ff2, JSON); session.remove(ff2); I do get errors depending on if I remove or not (NullPointer, already marked for transfer...) Is there an example here that show how to do what I want to do? Thanks MK ... View more

Is there documentation on how to specify time formats for example in Polling Interval for the human readable string? Ie: 60 sec, 1 hour, 10 hours? Does Nifi understand 1sec and "1 sec" and which words are available? sec, hour, hours?? Basically the "Allowed Values" column on the docs pages dont say. ... View more

Is there instructions on how to install spark2 on an HDP 2.5.6 cluster? We are currently running 1.6.3 and we need to use the Magellan spatial libs but this does not function under 1.6.3 spark. Can you point me to spark 2 installation instructions? ... View more

First time running teragen/sort/validate test went ok on 100 Gig test size. After re-running test, teragen fails and log says following: I see onyl 26 maps are processed of the total of 92. Any ideas...stack below. 2017-07-19 14:01:12,719 INFO [Thread-145] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STOPPED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /mr-history/tmp/user1/job_1499796473254_0009.summary_tmp could only be replicated to 0 nodes instead of minReplication (=1). There are 4 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1620) at ... org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2211) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2207) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2205) org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /mr-history/tmp/user1/job_1499796473254_0009.summary_tmp could only be replicated to 0 nodes instead of minReplication (=1). There are 4 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1620) Other info: hdfs dfsadmin -report Configured Capacity: 1074290753536 (1000.51 GB) Present Capacity: 601786240099 (560.46 GB) DFS Remaining: 172010277124 (160.20 GB) DFS Used: 429775962975 (400.26 GB) DFS Used%: 71.42% Under replicated blocks: 307 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 ... View more

I run HDP spark 1.4.1 and 1.6.1 I have to process the input of rapidly arriving XML from a Kafka topic with spark strmng. I am able to do the .print function and see my data is indeed coming into spark. I have them batched now at 10 s. Now I need to know: 1) Is there a way to delimit each XML message? 2) How can I apply a JAXB like schema function to each message? I have a process already doing this in plain java and it works fine using the standard kafka APIs and JAXB. Sample output where I write the data with saveAsTextFiles() shows broken messages, seemed to be split on space and large XML messages are spread across more than one file. Thanks, M ... View more

Trying to get some data written over to an elastic search node from spark-shell before creating scala app. Running Horton with 1.4.1 Spark. Whenever I run following: val conf = new SparkConf() conf.set("es.index.auto.create", "true") conf.set("es.nodes", "127.0.0.1") conf.set("es.port", "9200") sc.makeRDD(Seq(numbers, airports)).saveToEs("my_index") <console>:27: error: value saveToEs is not a member of org.apache.spark.rdd.RDD[scala.collection.immutable.Map[String,Any]] sc.makeRDD(Seq(numbers, airports)).saveToEs("my_index") Where is the saveToEs() method defined? I have tried this other way: import org.elasticsearch.spark.rdd.EsSpark EsSpark.saveToEs( sc.makeRDD(Seq(numbers, airports)),"my_index") And get: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only' Any ideas for those who have tried? MAK ... View more

Running spark-submit job with Spark 1.4.1 on HDP 2.3.2. Jobs show no logs when I click on the Executors tab for my spark job. Jobs and Stages tabs are also empty. Where can I find the stderr/out logs for my job? Also, job still says "Incomplete" when it has finished. This same app works fine in Spark 1.6.1. I can see everything and can refresh the page as it runs with updates. ... View more

I have a few external jars such as elasticsearch-spark_2.10-2.4.0.jar. Currently I use --jars option to load it for spark-shell. Is there a way to get this or other jars to load with spark for my cluster? I see through Ambari there is the spark-defaults but was wondering if I could just copy X.jar to /usr/hdp<Ver>/spark/lib and it would get picked up. A side question somewhat related, its in the same command line, is I use the following: "--packages com.databricks:spark-avro_2.10:2.0.1". I notice the first time this is done, spark goes out and grabs the jars like maven would. But I could not find these and wonder if I need this argument as well or can I get databrick libs installed permanently as with elasticsearch? Thanks, Mike ... View more

Running HDP-2.4.2, Spark 1.6.1, Scala 2.10.5 I am trying to read avro files on HDFS from spark shell or code. First trying to pull in the schema file. If I use: val schema = sc.textFile("/user/test/ciws.avsc") This loads file and I can do : schema.take(100).foreach(println) and see contents. If I do (using Avro Schema parser): val schema = new Schema.Parser().parse(new File("/home/test/ciws.avsc")) or val schema = scala.io.Source.fromFile("/user/test/ciws.avsc").mkString I get: java.io.FileNotFoundException: /user/test/ciws.avsc (No such file or directory) My core-site.xml specifies defaulFS as our namenode. I have tried adding "hdfs:/" to filepath and "hdfs://<defaultFS>/..." and still no dice. Any ideas how to reference file in HDFS with the Schema.Parser class or this scala.io.Source class? Mike ... View more