Member since
02-01-2022
292
Posts
105
Kudos Received
61
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 125 | 07-28-2026 12:03 PM | |
| 1371 | 05-15-2025 05:45 AM | |
| 5482 | 06-12-2024 06:43 AM | |
| 8529 | 04-12-2024 06:05 AM | |
| 6407 | 12-07-2023 04:50 AM |
07-26-2022
12:31 AM
My problem was solved after using a different email id. As far as gmail settings are concerned: access to less secure apps must be turned on. IMAP access must be turned on in settings of gmail.
... View more
07-20-2022
08:23 AM
I think you will also need to do the transfer/commit in the each list logic
... View more
07-20-2022
05:14 AM
@AbhishekSingh I believe you can accomplish this with some very complicated Expression Language chaining. It is possible to have many Expression Languages combined into one expression. Reference: https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html I do not have time to prove this out, but in theory, you would combine lesson learned in your other post (expression language to change the database.table with Replace, ifElse and maybe Equals to get your desired outcome.
... View more
07-20-2022
05:02 AM
@AbhishekSingh It looks like the issue with the Replace Expression Language was just the ` around the database and table name. That works for me as follows: ${query:replace('`nrpuserorgdb2306`.`status`', '`nrpreportdb`.`user_org_status`')} Here is my flow to test: Notice the use of a simple flow to test the concept. I also set query in GenerateFlowfile and define a separate query2 in UpdateAttribute. I always work in ways that prove functionality, then when the concept is working, take the lesson learned into real flows. Some additional Screen Shots: Template and Flow Definition File on my GitHub: https://github.com/cldr-steven-matison/NiFi-Templates
... View more
06-29-2022
06:20 AM
@Tryfan You mention this file comes in daily. You also mention that this file arrives through a load-balancer so you don't know which node will receive it. This means you can't configure your source processor for "primary node" only execution as you have done in your shared sample flow with the ListFile. As Primary Node only, the elected primary node will be the only node that executes that processor. So if the source file lands on any other node, it would not get listed. You could handle this flow in the following manor: GetFile ---> (7 success relationships) PostHTTP or InvokeHTTP (7 of these with one configured for each node in your cluster cluster) ListenHTTP --> UpdateAttribute --> PutFile So in this flow, no matter which node receives your source file, the GetFile will consume it. It will then get cloned 6 times (7 copies then exist) with one copy of the FlowFile getting routed to 7 unique PostHttp processors. Each of these targets the ListenHTTP processor listening on each node in your cluster. That ListenHTTP processor will receive all 7 copies (one copy per node) of the original source file. Then use the UpdateAttribute to set your username and location info before the putFile which place each copy in the desired location on the source node. If you add or remove nodes from your cluster, you would need to modify this flow accordingly which is a major downside to such a design. Thus the best solution is still one where the source file is placed somewhere all nodes can retrieve it from so it scales automatically. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
06-10-2022
12:17 AM
2 Kudos
Follow the steps to install, lack _psycopg psycopg2 package. Py files The solution 1:wget https://github.com/psycopg/psycopg2/archive/refs/tags/2_8_5.tar.gz 2:tar -xf 2_8_5.tar.gz 3:python setup.py build python setup.py install 4:cp psycopg2-2.8.5-py3.7-linux-x86_64.egg/psycopg2 /usr/share/python3
... View more
06-01-2022
10:59 AM
1 Kudo
@steven-matison thank you so much.
... View more
05-24-2022
06:02 AM
1 Kudo
@FediMannoubi Below is a basic approach to solve. Assuming both postgres tables are populated with rows per your example, your nifi flow would need to get the CSV (various ways to do that), once the contents of the csv are in a flowfile (i use GenerateFlowFile processor), you can use a RecordReader based processor to read the csv. This will allow you to write SQL against the flowfile with QueryRecord to get a single value. For example: SELECT city_name FROM FLOWFILE Next, in your flow you will need to get the city_name value into an attribute, i use EvaluateJsonPath. After that a ExecuteSQL processor and associated DBCP Connection pool to postgres. Then in ExecuteSQL your query is SELECT city_id FROM CITY WHERE city_name=${city_name} At the end of this flow you will have the city_name from csv, and city_id from postgres. You can now combine or use the further downstream to suit your needs. INSERT is done similarly, once you have the data in flowfiles, or attributes, using the same ExecuteSQL you write an insert instead. My test flow looks like this, but forgive the end, as I did not actually have a postgres database setup. You can find this sample flow [here]. I hope this gets you pointed in the right direction for reading csv and querying data from database.
... View more
05-17-2022
02:43 PM
@joshtheflame
I just wanted to provide a bit more context. The partial page shot you've included above appears to show Cloudera Manager running against a CDH 6.1.0 cluster. CDH 6.1.0 was released in December of 2018. As you no doubt are aware, that was quite a while ago, especially in terms of "internet time". Hopefully you are aware that CDH 6.1.x has reached its End of Support (EoS). You can find the most recent official reminder of the previous announcement that Cloudera Enterprise 6.x reached End of Support (EoS) in 2021 here:
March 2021 Customer Advisory - 2: End of Support for Cloudera Products (CDH/CM 6.x & HDP 3.x).
Cloudera's lifecycle support policies are documented here:
https://www.cloudera.com/legal/policies/support-lifecycle-policy.html
My understanding is that organizations with a valid Cloudera subscription for legacy products such as CDH would have been sent this announcement directly.
If that screenshot represents what your bank is running in production, I would recommend that you reach out to your Cloudera Account team and discuss your upgrade options as soon as possible. You are going to have to upgrade in order to take advantage of any of the offerings mentioned in @steven-matison 's reply earlier.
... View more
- « Previous
- Next »