Member since
07-19-2018
613
Posts
101
Kudos Received
117
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 5690 | 01-11-2021 05:54 AM | |
| 3812 | 01-11-2021 05:52 AM | |
| 9493 | 01-08-2021 05:23 AM | |
| 9296 | 01-04-2021 04:08 AM | |
| 38644 | 12-18-2020 05:42 AM |
05-26-2020
05:26 AM
@renuu The solution you are looking for is: ${tags:unescapeJson()} If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven @ DFHZ
... View more
05-20-2020
11:40 AM
1 Kudo
@vikrant_kumar24 1. Yes, sorry for some reason they didnt get saved in that version of the template. I will update. You should see or set it like this: 2. Yes, you will need to enter the Remote Url, and maybe more properties. You also need to confirm that the json content in flow file is correct. Mine is just an example. I do not know what service you are connect too. That service may require username, passwords, tokens, headers, etc. The configuration for InvokeHttp will depend on these. If you prefer to share more details about the service you are connecting in PM we can discuss in private. Once you have the response from InvokeHttp, if the response is json, you will again do EvaluateJson, grab the values you want for the new Long/Lat, and replace or overwrite the existing attributes and continue downstream with your flow. 3. Converting CSV to JSON is pretty typical. The CSV->JSON Method you are a working on is just fine. You should be able to link the flowfile from that, to EvaluateJson in the flow. My example is just a demo to show you the concepts of EvaluateJson, UpdateAttribute, and AttributesToJson.
... View more
05-20-2020
07:30 AM
@vikrant_kumar24 To add new attributes to a flow file you should use UpdateAttribute. To get attributes from json you should use EvaluateJsonPath. Once you have all the values you want as attributes you should use AttributesToJson and then finally send the json to invokeHttp. Your flow will look like this: And the Queued Flowfile waiting on InvokeHttp is: Please note I did not have your original Json, just the schema above, so I created a simple JSON of the Longitute & Latitude values you gave me. I also did not touch InvokeHttp, its default. Once you get this far, and want to work on handling the response from InvokeHttp let me know here, in another post, or a private message. To see my entire template, check out my GitHub for the Invoke Http Demo 2 Template: https://github.com/steven-dfheinz/NiFi-Templates If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven @ DFHZ
... View more
05-20-2020
07:02 AM
1 Kudo
@keithg If you change the 1 sec to 15 sec or 30 sec does it become more stable? With the schedule set to 1 sec, invokeHttp will execute 1600 connections as fast as possible. This opens up quite a bit of room for either exceeding the capabilities of your environment (not enough nodes, ram or cores) or the upstream service you are connecting too (actual network issues with the workload). Since you only have 2 cores, you need to sensitive to tuning the flow, and most likely need to go slower (ie not 1 sec). You should also investigate documented NiFi Performance Tuning Steps, adding a lot more cores, adding more ram to each node including min ram: 2-6 gb max ram: 4-8-16 gb, and/or adjusting Garbage Collection.
... View more
05-19-2020
04:52 AM
1 Kudo
@keithg Can you update the post with the scheduling settings for InvokeHttp? Also would nice to see some specific details about the nifi environment (min/max memory settings, nifi node spec, disk config, etc) and setting for min/max threads? Without taking those things in consideration, I would assume you are overloading capability resulting in instability you are experiencing. If you slow down the scheduling and consume the incoming queue at a slower rate, does the timeout persist?
... View more
05-16-2020
05:54 AM
@gbukovszki The behavior you are describing is just how nifi escapes the string representation of the JSON inside of the schema. It is required in order to send to different avro processors. Assuming you have the schema in an attribute JSONAttribute, when you need to unescape, use the expression language below in UpdateAttribute : ${JSONAttribute:unescapeJson()} You can also do similar action if the escaped values are in a FlowFiles content with ReplaceText in Replacement Value: ${'$1':unescapeJson()} If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven @ DFHZ
... View more
05-16-2020
05:33 AM
@johndcal A namespace is not required within the avro schema source in Schema Registry. In the context of avro spec. In order to create an avro schema in the Schema Registry, you have to send the first call to create the schema entity. The next call is then to add the actual avro schema to the existing entity. This is just the behavior of the Schema Registry. You can find some lessons I created in how to use the registry: https://community.cloudera.com/t5/Community-Articles/Using-the-Schema-Registry-API/ta-p/286194 I also have an article showing how to fully automate the creation of Avro Schemas from CSV file (column name and data type) using the Schema Registry, Hive, and NiFI: https://community.cloudera.com/t5/Community-Articles/How-to-automate-creation-of-Avro-and-Hive-Schemas-using-NiFi/ta-p/293183 If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven @ DFHZ
... View more
05-14-2020
04:34 AM
@Regis Previously open source, the new versions of ambari (2.7.5) and HDP (3.1.5) have moved behind a paywall in an open core strategy. You cannot access them without cloudera subscription. I recommend to use the last free versions: Ambari 2.7.4 and HDP 3.1.4. You can find these repos here: https://docs.cloudera.com/HDPDocuments/Ambari-2.7.4.0/bk_ambari-installation/content/ambari_repositories.html https://docs.cloudera.com/HDPDocuments/Ambari-2.7.4.0/bk_ambari-installation/content/hdp_314_repositories.html Also below is more information about the Authorization needed for Pay Walled Repos: https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/access_ambari_paywall.html If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven @ DFHZ
... View more
05-13-2020
08:27 AM
@satishjan1 The initial question is asking about setting the hostname. The information you reference is telling you to do that, but for a different operating system. My first response was telling you how to do it for RHEL. For your next question, you do not have to set the hostname in /etc/sysconfig/network, you have to do it the way required for your operating system. See Above. The hostname must be set, and persist after reboot. If you do not set the hostname before installing the cluster you will have unmentionable problems with services and components later on down the road.
... View more
05-12-2020
06:08 AM
@satishjan1 the command to set the hostname is:
hostnamectl set-hostname host.name.com
Depending on your OS config, you may also need to update items that may manage that hostname and the host files. You can confirm by doing above, and then reboot. The hostname should persist after reboot.
If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.
Thanks,
... View more