Is Hadoop/Hive loaded on the HDF sandbox? It does not appear to be. Do I need to download the HDP sandbox? If so, how do I configure both virtual guests to successfully process a NiFi dataflow from the HDF virtual sandbox to the HDP sandbox? ... View more

Fellow Techies-- I've set up my local open source development environment under the Eclipse Mars 4.5 IDE bench with Git, PyDev, Ivy, Gradle, Scala, Oozie, Pig and HDF/MapReduce plugins all accessing a VMWare Player Hortonworks 2.4 sandbox. This works for general small-scale development before deployment to other similar installations. I will soon be working on an Azure/Microsoft installation. This development environment is centered around Visual Studio with NuGet plugin equivalents. Can anyone tell me what the recommended development/deployment environment is for this ecosystem? ... View more

I am primarily a .NET/C# and JBOSS/J2EE database and applications integration developer beginning to work with NiFi. Since my previous work has been primarily on the back end, I find myself at a disadvantage when working with 3rd party API examples which reference javascript/web page access. I am interested in an available public API from DOT: https://vpic.nhtsa.dot.gov/api/ . Here is an example of a method for decoding an array of vehicle id numbers (VINs) called DecodeVINValuesBatch. DOT offers the javascript coding example below. How do I express or re-interpret this example in the context of NiFi? Assuming InvokeHttp is selected for the processor used for calling the service, how would I pass InvokeHttp the collection of VIN#s ( perhaps a different processor which passes the collection before InvokeHttp)? How would I address System.Threading? Would it be necessary? I am looking for some guidance on how I should approach this with a practical example. string text = "3GNDA13D76S000000;5XYKT3A12CG000000;"; string url = @"https://vpic.nhtsa.dot.gov/api/vehicles/DecodeVINValuesBatch/"; var nameValues = new Dictionary<string, string>(); nameValues.Add("data", text); nameValues.Add("format", "json"); HttpClient client = new HttpClient(); client.BaseAddress = new Uri(url); // using FormUrlEncodedContent var name = new FormUrlEncodedContent(nameValues); client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json")); System.Threading.CancellationToken token = new System.Threading.CancellationToken(); try { var tmp = client.PostAsync(client.BaseAddress, name, token).Result; var result = tmp.Content.ReadAsStringAsync(); } catch (Exception err) { // error handling } ... View more

Techies-- I am working on a POC to replace some of the traditional ETL processes we have in place using NiFi to import data to HDFS/Hive -- where the data can then later be accessed by Microsoft's Polybase and visualization tools. As an adjunct, some SQL-power users have an interest in accessing the preliminary data streams imported to HDFS/Hive. In our production environment there are about 40,000 such files daily. Each file arrives with a fixed-length header and multiple fixed-length line items. These files include the datetime stamp in the naming convention. The files are sourced from multiple locations, from multiple partners, so an immediate example of a SQL-like query in HDFS/Hive would be (1) partner identification [derived from header]; (2) location [derived from header], (3) date/time [derived from filename] and (4) products [derived from line items]. From a guidance perspective, how should I approach this project? Is NiFi robust enough to import from so many active streams ? Is the data arriving with putHDFS inherently "Hive" access ready, or is there another step to consider? For audit purposes, how should I approach capturing things like line item counts, file names the data arrived on, etc.? Which examples from here: https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates are the best to follow? I am looking forward to your advice. ... View more