Re: How to get Nifi invokehttp SSL works in apache...

opreaadrian1 · ‎10-26-2018

Hi, I had the same issue and after i created the SSLContextService, i had to change the property in the InvokeHttp "Always Output Response" to true and this will give you an output, in the output look for the invokehttp.remote.dn, since is a 403 error "Forbidden" it means that the dn does not have access to make this request but your SSLContextService is working. Next step is to add the Identity that will make the https request(invokehttp.remote.dn) in NiFi User UI and run again the InvokeHTTP. Hope this helps

opreaadrian1 · ‎06-12-2018

Can you post you json file content ? Test your json path definition at http://jsonpath.com/? , is easy and debug any issues with the json format.

opreaadrian1 · ‎05-08-2018

Hi, You want to extract those entire 280 millions in a single call ? not a good idea ! I suggest you loop thru it using the table partition + bucket keys(for better query response on Hive). What is you table construct ? Based on this you can run SelectHiveQL in parallel or sequentially thru a loop. A once extract won`t do. Thx

opreaadrian1 · ‎05-04-2018

Hi all, Just upgraded to the latest and grates NiFi 1.6.0 and i am getting a strange error on a ExecuteSQL Processor. error: cannot insert multiple commands into a prepared statement So the ExecuteSQL uses a Database Connection Pooling Service that runs some queries against a database in this case Vertica(with the latest jar), and there are multiple sequential queries. I managed to fix the issue by rolling back my Apache NiFi Install to NiFi 1.5.0 and is all normal now. I can also replicate the error. I actually made a short Video on the behavior of the ExecuteSQL on 1.5.0 vs 1.6.0. Demo of Error Also Link: https://youtu.be/FJPf7PDmTSE Any ideas ? maybe a bug with new release ? Thx

opreaadrian1 · ‎04-28-2018

Hi, Your bottle neck is the flowfile repo configuration parameters. Things to look for: 1 - is your repos sharing the same disk volume ? I have for each major repo their own disk so i wont have to fight for IO and they are not on the same disk as the nifi install. 2 - what is the value of the nifi.queue.swap.threshold parameter, default is 20000 after this your NiFi will swap even with RAM available Eg: if your max queue would be 4000000 lines make sure you have this set in the Note: You JVM memory settings in the bootstrap.conf have to copmply with the volume allowed in your nifi.queue.swap.threshold, so if you have multiple data flows with 400 mb + what ever else you need to sum this to fit in your JVM Heap 3 - what is the Concurrent Tasks value on the SplitText by 10 ? (make sure this is 25% of Split by 1) eg:2 4 - what is the Concurrent Tasks value on the SplitText by 1 ? eg:8 I do a similar data flow and i can do 5000000 rows in 40 sec. This does not consider Kafka Push. i hope this helps

opreaadrian1 · ‎04-26-2018

Hi, So is the getfile & splitfile processors taking 5 min ? What is the use pattern of the data after the file is split. Also it took me a secound to get a 400 mb split it in 5 files(split by lines number)

opreaadrian1 · ‎02-27-2018

Hi, I manage to get to the bottom of it at https://stackoverflow.com/questions/48985818/apache-nifi-executestreamcommand-wrong-output , @daggett he asked to set Ignore STDIN to true This fixed my issue. How ever this was not very self explanatory becouse i don`t know how and why would the output be changed?!

opreaadrian1 · ‎02-26-2018

I have NiFi flow that runs some shell scripts using the ExecuteStreamCommand processor and the output of the ExecuteStreamCommand is not correct. The Shell i run is: if (( $(ps -ef | grep -v grep | grep kibana | wc -l) > 0 )); then echo "1"; else echo "0"; fi; if the service is up then 1 if is down then 0, simple but the output is wrong, not matter is the service is up or down the output is always 1. Here is a demo if the flow: https://youtu.be/4e00rzerjSQ

opreaadrian1 · ‎02-01-2018

Are you able to ping the S3 or access the S3 endpoint from you NiFi server ? If your NiFi server is on private subnet with no EIP attached you need to create an VPC endpoint to your S3 service If this you case follow this link :https://aws.amazon.com/blogs/aws/new-vpc-endpoint-for-amazon-s3/

opreaadrian1 · ‎01-23-2018

I have a Apache NiFi Server i use to collect data from multiple source (Windows/Linux) where i have minifi on them setup. On each edge minifi flow configuration i have a self-control PG that collect minifi internal statistics such as instance:health,stats,bulletins. Where the call look like this Eg #/bin/bash /opt/minifi/bin/minifi.sh flowStatus instance:health,stats,bulletins | tail -2 | head -1 So far so good - i get this data extract the json -> create an Influx Call and push it into a InfluxDb that is then displayed into Grafana, this has alerts when one Egde goes off or there is a high queue, or even the Garbage Collector on the minifi edge stats bla bla. Question: How do i do this in Windowzzzzz ? i am going crazy. The .bat executables are not responding to any of the flowStatus options I managed to do it thru Cygwin but i don`t want to setup Cygwin in all egdes. Any ideas here on how i can achieve this ? thanks

opreaadrian1 · ‎12-11-2017

Hi, I am having a problem/issue for the last 4 days and i am stuck. So i have a Processor that runs as PowerShell Script : ((Get-Counter -Counter '\Process(MSTRSvr2_64)\Private Bytes').CounterSamples.CookedValue) The output at the power-shell cmd line is different than the response i get in NiFi the Full Cmd is : $MSTRPrivateBytes = ((Get-Counter -Counter '\Process(MSTRSvr2_64)\Private Bytes').CounterSamples.CookedValue) $HostName = $env:computername write-host "MSTRPrivateBytes,HostName=$HostName $MSTRPrivateBytes" My Execute Stream CMD look like this : Is there ant truncate involved ? If i run manually it gives me the correct result, but in NiFi it is different !!! I am using the same cmds to get other pointers but they work great only this one is giving me problems. Any ideas ?

opreaadrian1 · ‎11-23-2017

Your question is bad formulated , can you post the error or what is this execute script processor json ? There is no executescriptjson processor

opreaadrian1 · ‎11-23-2017

Just my Two Cents on it, If you are experiencing a lot of queued flow-files and you want to clear them without having to reload the config.yml, a very quick and dirty way would be to clear all the repos on the minifi edge. Here how i do it - assuming your minifi sits in /opt - you will lose all queued data by doing this, but you no longer have queues.(not ideal) -- before cleanup /opt/minifi/bin/minifi.sh flowStatus connection:all:health,stats | tail -2 | head -1| jq '.' /opt/minifi/bin/minifi.sh stop rm -rf /opt/minifi/flowfile_repository/* rm -rf /opt/minifi/content_repository/* rm -rf /opt/minifi/provenance_repository/* /opt/minifi/bin/minifi.sh start -- after cleanup /opt/minifi/bin/minifi.sh flowStatus connection:all:health,stats | tail -2 | head -1| jq '.' Ideal solution: Use the minifi.sh flowStatus cmd to push data into the NiFi server and monitor this types of stuff. From here you can develop nifi flows to act on what needs to be done to those edge minifi. you can even build a dashboard from this data and actually see how to edges are doing. No doubt Horton has this on the to do list.

opreaadrian1 · ‎11-23-2017

Hi, I managed to setup minifi in Linux and is running great, but in windows not so great :(. I am actually trying to get flowStatus of the minifi on the windows host using the run-minifi.bat . I can actually go around this issue using cygwin or mobaxterm(linux like cmd lines in windows), but was wandering if is there a way to do it using the .bat scripts ? Thx

opreaadrian1 · ‎11-09-2017

yeah i know is not very intuitive 🙂

opreaadrian1 · ‎11-08-2017

Hi, You almost got it right 🙂 In the Replacement Value: type "Shift-Enter" -- this will add a new line as replacement value. the search value will remove all empty lines and prefix spaces. Also i have attached a sample template : replace.xml

opreaadrian1 · ‎11-08-2017

Hi all, I am trying to understand the output of the minifi.sh flowfile <options> commands. I wanna be able to monitor protectively all the "edge points" created by minifi. Example: The is the output of this cmd /minifi.sh flowStatus systemdiagnostics:heap,processorstats,contentrepositoryusage,flowfilerepositoryusage,garbagecollection Output { "controllerServiceStatusList": null, "processorStatusList": null, "connectionStatusList": null, "remoteProcessGroupStatusList": null, "instanceStatus": null, "systemDiagnosticsStatus": { "garbageCollectionStatusList": [{ "name": null, "collectionCount": 3067, "collectionTime": 26764 }, { "name": null, "collectionCount": 6, "collectionTime": 861 }], "heapStatus": { "totalHeap": 260571136, "maxHeap": 260571136, "freeHeap": 100187704, "usedHeap": 160383432, "heapUtilization": 62, "totalNonHeap": 76414976, "maxNonHeap": -1, "freeNonHeap": 2070192, "usedNonHeap": 74344784, "nonHeapUtilization": -1 }, "contentRepositoryUsageList": [{ "name": "default", "freeSpace": 11783532544, "totalSpace": 29994987520, "usedSpace": 18211454976, "diskUtilization": 61 }], "flowfileRepositoryUsage": { "freeSpace": 11783532544, "totalSpace": 29994987520, "usedSpace": 18211454976, "diskUtilization": 61 }, "processorStatus": { "loadAverage": 10.52, "availableProcessors": 14 } }, "reportingTaskStatusList": null, "errorsGeneratingReport": [] } Some of the metrics listed are self explanatory but not in all cases and i would hate to make wrong assumptions and build a process around wrong interpreted metrics. Where do i get stuck ?! - trying to capture connection metrics/values /minifi.sh flowStatus connection:all:health,stats And i have Questions like : connextionstatus.inputCount: 25 <--- this value is this since begging ? is this based on 5 min interval ? How to i interpret this ? { "controllerServiceStatusList": null, "processorStatusList": null, "connectionStatusList": [{ "name": "OUT Monitor - Clean and Extract//015f11b2-ba5c-14f0-d0dd-f17188d247d8", "connectionHealth": { "queuedCount": 0, "queuedBytes": 0 }, "connectionStats": { "inputCount": 25, "inputBytes": 3584, "outputCount": 25, "outputBytes": 3584 } }, { "name": "OUT Monitor - Initiate //IN Monitor - Clean and Extract", "connectionHealth": { "queuedCount": 0, "queuedBytes": 0 }, "connectionStats": { "inputCount": 6, "inputBytes": 828, "outputCount": 6, "outputBytes": 828 } }], "remoteProcessGroupStatusList": null, "instanceStatus": null, "systemDiagnosticsStatus": null, "reportingTaskStatusList": null, "errorsGeneratingReport": [] } I know minifi is still a small project i am so very grateful for the work the community is putting into it ! But is there a more detailed documentation around this ? Thanks all

opreaadrian1 · ‎11-07-2017

Thanks @Hellmar Becker

opreaadrian1 · ‎11-05-2017

Hi, Very interesting article. In what day to day scenario would this be useful ? Thx

opreaadrian1 · ‎11-02-2017

Hi, Use This - this lists the resources in the server and formats the templates ids into a api call. note: jq -- formats the output to be readable curl -v -X GET http://NiFiServerIP:Port/nifi-api/resources | jq '.' | grep '"/templates/' | awk '{print $2}' | sed 's/\"//g'| sed 's/\/templates//g' | awk '{print "curl -v -X GET http://NiFiServerIP:Port/nifi-api/templates"$1"/download"}' The output should be a Rest Api call: curl -v -X GET http://NiFiServerIP:Port/nifi-api/templates/885dc02e-cb13-429c-bdvd-9a4e0cbe1212b/download In the future if you don`t see how is done in the docs you can always SPY on the nifi-app.log, it gives you what you need to know. Hope this helped.

opreaadrian1 · ‎11-02-2017

Use port 587 - it work for me. See link here: https://support.google.com/a/answer/176600?hl=en Note: Before you start the configuration, make sure that Less secure apps is enabled for the desired account.

opreaadrian1 · ‎10-24-2017

Why don`t you use a nifi-app_${now():format('yyyy-MM-dd')}_* Where the output will be nifi-app_2017-10-25_*. You can also use attributes for you log prefix and then you route on attribute based on the log type.

opreaadrian1 · ‎10-20-2017

Hi, Is there a way to query the H2 Database in Apache NiFi ? Scope: 1 - i want to be able to list all Queues in a Process Group. 2 - i want to be able to associate Process Group/Processors with their IDs and Given names. This way i can build a dynamic mechanism to better integrate and administer the NiFi work. Thx

opreaadrian1 · ‎10-19-2017

Hi, I was wandering if it is possible to capture the Process Group ID into a Flow-file Attribute. I am trying to actually create a dynamic Template in NiFi that will manage it self using this attributes and generating RestApi calls against it self. Scenario would be : 1 - Process Group ID #1 is done - subtask generate restapi to stop this PG and initiate the next PG. - i currently do this using "Monitor Activity" and "ExecuteStreamCommand" - has the hard coded RestApi calls in a shell script. It work perfect- but i am looking to actually make this more reusable but having this Attributes shared and used in a template. Also is there a way to actually Query the H2 NiFi database to extract this values from the raw db ? Thanks all

opreaadrian1 · ‎09-27-2017

Hi, I have the following JSON { "ARTGEntryJsonResult": { "AnnualChargeExemptWaverDate": null "Conditions": [ "" ], "ConsumerInformation": { "DocumentLink": "" }, "EntryType": "Medicine", "LicenceClass": "", "LicenceId": "152567" }, "Products": [ { "AdditionalInformation": [], "Components": [ { "DosageForm": "Drug delivery system, transdermal", "RouteOfAdministration": "Transdermal", "VisualIdentification": "Dull, homogenous" } ], "Containers": [ { "Closure": "", "Conditions": [ "Store at room temperature" ], "LifeTime": "2 Years", "Material": null, "Temperature": "Store below 25 degrees Celsius", "Type": "Sachet" } ], "EffectiveDate": "2017-09-18", "GMDNCode": "", "GMDNTerm": "", "Ingredients": [ { "Name": "Fentanyl", "Strength": "6.3000 mg" } ], "Name": "FENTANYL SANDOZ ", "Packs": [ { "PoisonSchedule": "(S8) Controlled Drug", "Size": "1" }, { "PoisonSchedule": "(S8) Controlled Drug", "Size": "10" }, { "PoisonSchedule": "(S8) Controlled Drug", "Size": "2" }, { "PoisonSchedule": "(S8) Controlled Drug", "Size": "3" }, { "PoisonSchedule": "(S8) Controlled Drug", "Size": "4" }, { "PoisonSchedule": "(S8) Controlled Drug", "Size": "5" }, { "PoisonSchedule": "(S8) Controlled Drug", "Size": "7" }, { "PoisonSchedule": "(S8) Controlled Drug", "Size": "8" } ], "SpecificIndications": [ "Management of chronic pain requiring opioid analgesia." ], "StandardIndications": [], "Type": "Single Medicine Product", "Warnings": [] } ], } } I am able to extract it using the splitJson + EvaluateJsonPath but i get the Array Fields Data in a single row. Example of what i managed to get considering this JSON file: LicenceId Name PoisonSchedule Size 152567 FENTANYL SANDOZ "(S8) Controlled Drug","(S8) Controlled Drug","(S8) Controlled Drug" 1,10,2 What i actually want is : LicenceId Name PoisonSchedule Size 152567 FENTANYL SANDOZ (S8) Controlled Drug 1 152567 FENTANYL SANDOZ (S8) Controlled Drug 10 152567 FENTANYL SANDOZ (S8) Controlled Drug 2 Any ideas are much appreciated !!! Thanks

opreaadrian1 · ‎09-26-2017

You don`t need to Extract the text . Your flow should look like this GenerateTableFetch --> ExecuteSQL -->ConvertAvroToJson(optional) --> StoreResults Where Your GenerateTableFetch conf would be: And the ExecuteSQL is: You would get a Json/Avro Output Format: Then you can work on you Json file to extract it

opreaadrian1 · ‎08-08-2017

Ok , I need to give you a big hug !!! I was going out of my mind with this !!! Could not imagine why was not working ! thank you so much ! you made my day 🙂

opreaadrian1 · ‎08-07-2017

Hi all, I am getting my arse kicked by the EvaluateJsonPath. So the task is to be able to extract some json attribute values into a CSV format or a text format that will be used for inserting into file, db ,etc.. So here is what i got so far: GetFile (it reads a json file) --> SplitJson --> EvaluateJsonPath --> PutFile (i will throw a merge in between after i get after the EvaluateJsonPath to work "understand how it work"). The json used is: - this is just a dummy json data { "results": [ { "gender": "female", "name": { "title": "ms", "first": "juliette", "last": "schmitt" }, "location": { "street": "7260 esplanade du 9 novembre 1989", "city": "le havre", "state": "lot", "postcode": 58491 }, "email": "juliette.schmitt@example.com", "login": { "username": "smallcat938", "password": "horizon", "salt": "fvaSIrep", "md5": "3d1a182db3002dc88cd54b258838aa89", "sha1": "71600ad7a6f65ec00fed7521bcf3d943a87a8172", "sha256": "cd9bfc086b5b83a935c99b496c27fc58716521a43c2630e286f87147e4b4dd8a" }, "dob": "1959-08-16 23:11:26", "registered": "2010-07-16 20:27:45", "phone": "04-09-99-35-16", "cell": "06-87-17-31-45", "id": { "name": "INSEE", "value": "259755098491 56" }, "picture": { "large": "https://randomuser.me/api/portraits/women/71.jpg", "medium": "https://randomuser.me/api/portraits/med/women/71.jpg", "thumbnail": "https://randomuser.me/api/portraits/thumb/women/71.jpg" }, "nat": "FR" } ], "info": { "seed": "ca021f036c48503a", "results": 1, "page": 1, "version": "1.1" } } 1- GetFile setup: Getfile 2- SplitJson setup: SplitJson 3- EvaluateJsonPath EvaluateJsonPath Note: this one is working! Now adding a new property to the EvaluateJsonPath gets me stuck ErrorJsonPaths The documentation is not very illustrative on how this should be used and i am not an json expert. I am trying to understand how the does the twitter template actually does it ? twitt.png -where multiple paths are declared. thanks all

Online	Offline
Last Visited	‎06-18-2019 11:40 PM

Date Registered	‎07-22-2016 03:57 AM
Last Visited	‎06-18-2019 11:40 PM
Total Messages Posted	28
Total Kudos Received	5

Re: How to get Nifi invokehttp SSL works in apache...

Re: Replacetextprocessor remove blank lines

Re: List existing Nifi templates using REST API

Re: How to get Nifi invokehttp SSL works in apache...

Re: How to use Apache NiFi EvaluateJsonPath for JS...

Re: import a large table from hive in nifi using S...

Apache NiFi 1.6.0 Error Post Upgrade

Re: How to process Huge files from FTP arriving in...

Re: How to process Huge files from FTP arriving in...

Re: Apache NiFI ExecuteStreamCommand Wrong Output

Apache NiFI ExecuteStreamCommand Wrong Output

Re: transfer On premises file to AWS S3 using nifi

How to get the flowstatus on a Windows Apache Mini...

Apache NiFi Strange OutPut on ExecuteProccessor Po...

Re: Excute script processor check json contain att...

Re: How to clear MiniFi queues?

How to Get Minifi flowStatus in Windows

Re: Replacetextprocessor remove blank lines

Re: Replacetextprocessor remove blank lines

Apache minifi Monitoring

Re: Counting lines in text files with NiFi - part ...

Re: Counting lines in text files with NiFi - part ...

Re: List existing Nifi templates using REST API

Re: Nifi PutEmail failure to send flowfile to gmai...

Re: how to use File filter in GetFile processor at...

How to Read from H2 datanase in NiFi

How to capture Apache NiFI Process Group ID as Flo...

Extract Data from JSON Array and Merge with Parent...

Re: how to extract text from generateTablefetcha a...

Re: How to use Apache NiFi EvaluateJsonPath for JS...

How to use Apache NiFi EvaluateJsonPath for JSON t...