Member since
07-19-2018
613
Posts
101
Kudos Received
117
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
4903 | 01-11-2021 05:54 AM | |
3340 | 01-11-2021 05:52 AM | |
8645 | 01-08-2021 05:23 AM | |
8161 | 01-04-2021 04:08 AM | |
36044 | 12-18-2020 05:42 AM |
08-27-2020
06:08 AM
1 Kudo
@P_Rat98 Creating an API with NiFi using HandleHttpRequest and HandleHttpResponse is something I have done quite a few times for Hortonworks and NiFI Customers. This is a great use case for NiFi and sending receiving JSON to NiFi, processing JSON, and completing actions downstream is super easy. I have created a basic template for you which includes the HandleHttpRequest (inbound port 80 call) a process group for doing something with the JSON, and HandleHttpResponse (provides 200 response code) to respond to inbound call. This is an API in the simplest form with NiFi. Depending on your use case you can build out Process Api Request Process Group to suit your needs. Out of the box you should be able to send to import template, add/start the StandHttpContextMap Controller Service, Start the flow, send a call to http://yourhost:80 and have JSON sitting in the bottom of the flow Success Queue. You can find the template here: https://github.com/steven-matison/NiFi-Templates/blob/master/NiFi_API_with_HandleHttpRequest_Demo.xml Some API suggestions: Be sure to take a look at both HandleHttp Processors for the properties you can configure. Ports, hostname, acceptable methods, ssl, authentication, and more. If your API call does not care if the Process API Request finishes, you can put HandleHttpResponse right after HandleHttpRequest, and let all the downstream work happen after the request/response is completed. This is common when I expect my API to be only giving inbound data, and doesn't care what the response is (other than just 200 to know it was received). In this case I accept the payload, return 200, and rest of the flow is decoupled from the connection. If my processing time is lengthy I usually do this so the system initiating the api call is not left waiting. Once you have the basic framework built, consider handling errors, and or returning different status codes as a variable (created before the response) in the Status Code for Handle Http Response. Sometimes I even have different HandleHttpResponse at end of different flow branches. For example: if someone sends invalid JSON, I return maybe 302 or 404 with the invalid error as the content body. Have fun with it. If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven @ DFHZ
... View more
08-24-2020
09:28 AM
@Koffi If you have a nifi flow created and tuned at a very large spec, and you downgrade that spec, you are going to have all kinds of problems like you are experiencing. You are going to need to go into the flow and reduce concurrency and min/max thread pool settings and completely re-tune the flow for the new environment since you reduce the ram and per core of the nodes. Another suggestion is that nifi 1.7 is very dated. You should consider an upgrade to nifi 1.12 and use at least 3 nodes. If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven @ DFHZ
... View more
08-24-2020
05:55 AM
@K_K The error I notice in here is: Caused by: java.net.BindException: Address already in use Some Suggestions: You need to check your DNS, networking (/etc/hosts) and make sure that is all correct. If make adjustments make sure to restart networking or reboot nodes then restart ambari server & agent. You need to make sure ambari-server and ambari-agent do not already have something running on the port which says "address already in use". Make sure that the host(s) for yarn are the correct hosts. I see "0.0.0.0 port 53" in the error log. You will want to make sure yarn is using the right ip/address/host etc and not some form of localhost or 0.0.0.0.0 I hope some of these help you arrive at the solution. If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven @ DFHZ
... View more
08-18-2020
05:24 AM
I suppose nowadays you would use ConvertRecord instead of ConvertCSVToAvro and then ConvertAvroToJSON.
... View more
08-17-2020
04:44 AM
@sppandita85BLR You should update your post with the issues you are having. You should be looking to add exact error messages found in ambari/agent logs. That process should be pretty easy: https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/configuring-wire-encryption/content/set_up_two-way_ssl_between_ambari_server_and_ambari_agents.html To confirm this works, using an ambari cluster I have ready for testing, I edited the ambari-server properties files, restarted ambari-server, and restarted ambari-agent. If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven @ DFHZ
... View more
08-17-2020
04:34 AM
@vikrant_kumar24 I believe the solution you are looking for is to use ExtractText to check for string matching the country you want in the first row. This uses regex to match the entire file which you only need 1 match to know what country it is. Using ExtractText to get an attribute called "country" you would when use RouteOnAttribute to create different country routes. For example: usa => ${country.equals("usa"). Once your routes are defined you can pull them off RouteOnAttribute and send them down separate flows you create for each country. You also should know that you can achieve the same logic of checking/defining/routing country by using QueryRecord. Either method is suitable, but the latter method is more standard in the newest versions of nifi. If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven @ DFHZ
... View more
08-15-2020
02:46 PM
1 Kudo
Thanks Steven
... View more
08-14-2020
05:55 AM
1 Kudo
Please accept the solution as answer. Doin this helps complete the solution.
... View more
08-13-2020
08:37 AM
@elfz I believe the functions you are working in only return unix timestamp to the seconds. It is completely ignoring .SSS which should be the correct indication for 3 millisecond decimals. Hive timestamps are able to be down to milliseconds with to 9 decimals. Here is a great post on this topic: https://community.cloudera.com/t5/Support-Questions/HIVE-datatype-Timestamp-with-miliseconds/td-p/150944
... View more