@Muffex    My recommendation for ingestion process is to always use staging/temporary tables which are managed separately from the master table the data needs to arrive in.   This allows you operate on the staging tables before or after those results are added to the master table w/o effecting the master table.    In your use case, your ingestion process would sqoop to temp,  insert from temp to master table, then drop temp location.    In some of my past implementations of this manner, the temp tables were organized hourly, and they stay active for at least 7 days before a decoupled cleanup job removes anything 7 days old.   This idea was done for auditing purposes, but normally I would create and destroy the data during the ingestion procedure.     If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

@P_Rat98 Creating an API with NiFi using HandleHttpRequest and HandleHttpResponse is something I have done quite a few times for Hortonworks and NiFI Customers.  This is a great use case for NiFi and sending receiving JSON to NiFi, processing JSON, and completing actions downstream is super easy.   I have created a basic template for you which includes the HandleHttpRequest (inbound port 80 call) a process group for doing something with the JSON, and HandleHttpResponse (provides 200 response code) to respond to inbound call.    This is an API in the simplest form with NiFi.   Depending on your use case you can build out Process Api Request Process Group to suit your needs.   Out of the box you should be able to send to import template, add/start the StandHttpContextMap Controller Service, Start the flow, send a call to http://yourhost:80 and have JSON sitting in the bottom of the flow Success Queue.   You can find the template here: https://github.com/steven-matison/NiFi-Templates/blob/master/NiFi_API_with_HandleHttpRequest_Demo.xml     Some API suggestions: Be sure to take a look at both HandleHttp Processors for the properties you can configure.  Ports, hostname, acceptable methods, ssl, authentication, and more. If your API call does not care if the Process API Request finishes,  you can put HandleHttpResponse right after HandleHttpRequest, and let all the downstream work happen after the request/response is completed.  This is common when I expect my API to be only giving inbound data, and doesn't care what the response is (other than just 200 to know it was received).  In this case I accept the payload, return 200, and rest of the flow is decoupled from the connection.     If my processing time is lengthy I usually do this so the system initiating the api call is not left waiting. Once you have the basic framework built,  consider handling errors, and or returning different status codes as a variable (created before the response) in the Status Code for Handle Http Response.  Sometimes I even have different HandleHttpResponse at end of different flow branches.  For example:  if someone sends invalid JSON,  I return maybe 302 or 404 with the invalid error as the content body. Have fun with it.       If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

@Koffi   If you have a nifi flow created and tuned at a very large spec, and you downgrade that spec, you are going to have all kinds of problems like you are experiencing.   You are going to need to go into the flow and reduce concurrency and min/max thread pool settings and completely re-tune the flow for the new environment since you reduce the ram and per core of the nodes.   Another suggestion is that nifi 1.7 is very dated.  You should consider an upgrade to nifi 1.12 and use at least 3 nodes.     If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

@vikrant_kumar24 I believe the solution you are looking for is to use ExtractText to check for string matching the country you want in the first row.  This uses regex to match the entire file which you only need 1 match to know what country it is.   Using ExtractText to get an attribute called "country" you would when use RouteOnAttribute to create different country routes.  For example: usa => ${country.equals("usa").  Once your routes are defined you can pull them off RouteOnAttribute and send them down separate flows you create for each country.   You also should know that you can achieve the same logic of checking/defining/routing country by using QueryRecord.  Either method is suitable, but the latter method is more standard in the newest versions of nifi.     If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

@Jarinek    Some infofmation about your nifi configuration will help be more accurate.  For example: min/max ram, number of cores, disk configuration, etc..  Information about your flow is important too. The processor/queue back pressure, concurrency, and run time all effect the performance you need.    Without this information, it sounds like your testing solution is exceeding the inbound capabilities of the flow tuning (nifi config, processor/queue config).   You should look to increase concurrency, increase queue size and back pressure based on # of flowfiles moving through. your data flow.  You should also inspect the min/max thread counts as these have a major impact on performance.   All of these items will be seriously limited with a single node, so be mindful of your expectations.   If you can I would recommend a small 3 node nifi cluster to evaluate nifi performance in a better test environment where you can really turn up the performance and distribute the work load across 3 nodes.   With 3 times as many cores & ram you can make better use of min/max thread count, increase concurrency much higher, and you should see the stability you are expecting.     If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more