1973
Posts
1225
Kudos Received
124
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
779 | 04-03-2024 06:39 AM | |
1502 | 01-12-2024 08:19 AM | |
772 | 12-07-2023 01:49 PM | |
1331 | 08-02-2023 07:30 AM | |
1923 | 03-29-2023 01:22 PM |
02-09-2021
08:46 AM
Cloudera Schema Registry is rarely on 8081 It is usually on port 9090. https://docs.cloudera.com/csp/2.0.1/schema-registry-overview/topics/csp-examples_of_interacting_with_schema_registry.html Which version are you using? Which form factor? Public Cloud? On-Premise? https://www.datainmotion.dev/2020/10/running-flink-sql-against-kafka-using.html https://www.datainmotion.dev/2020/05/commonly-used-tcpip-ports-in-streaming.html https://www.datainmotion.dev/2020/06/using-apache-kafka-using-cloudera-data.html https://www.datainmotion.dev/2020/08/deleting-schemas-from-cloudera-schema.html Check out the swagger rest docs https://www.datainmotion.dev/2020/11/flank-smart-weather-websocket.html
... View more
02-08-2021
07:52 AM
Variables are deprecated. Parameters are new and easy to externalize. I can use them in devops processes via REST, NiFi CLI and Python. You can programatically build parameter contexts and parameters and assign them to process groups. Parameters are getting some upgrades to do some of the more advanced things you mentioned. https://www.datainmotion.dev/2020/09/devops-working-with-parameter-contexts.html And they help power stateless nifi., https://www.datainmotion.dev/2019/11/exploring-apache-nifi-110-parameters.html https://www.datainmotion.dev/2021/01/automating-starting-services-in-apache.html
... View more
02-08-2021
07:49 AM
1 Kudo
Grok is Grok. Find Grok expressions that work for you and use a grok tester https://stackoverflow.com/questions/38462630/logstash-grok-filter-key-value-pairs http://grokconstructor.appspot.com/do/match#result https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.grok.GrokReader/additionalDetails.html
... View more
01-28-2021
07:34 AM
QueryRecord is the way to go. You can compare with a sql SELECT * FROM FLOWFILE WHERE timestamp > ${event_time} and value = ${id}
... View more
01-26-2021
11:40 AM
Automating Starting Services in Apache NiFi and Applying Parameters Automate all the things! You can call these commands interactively or script all of them with awesome DevOps tools. @Andre Araujo and @dchaffey can tell you more about that. Enable All NiFi Services on the Canvas By running this three times, I get any stubborn ones or ones that needed something previously running. This could be put into a loop; check the status before trying again. nifi pg-list
nifi pg-status
nifi pg-get-services The NiFi CLI has interactive help available and also some good documentation: NiFi CLI Toolkit Guide /opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-enable-services -u http://edge2ai-1.dim.local:8080 --processGroupId root
/opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-enable-services -u http://edge2ai-1.dim.local:8080 --processGroupId root
/opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-enable-services -u http://edge2ai-1.dim.local:8080 --processGroupId root We could then start a process group if we wanted: nifi pg-start -u http://edge2ai-1.dim.local:8080 -pgid 2c1860b3-7f21-36f4-a0b8-b415c652fc62 List all process groups /opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-list -u http://edge2ai-1.dim.local:8080 List Parameters /opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi list-param-contexts -u http://edge2ai-1.dim.local:8080 -verbose Set parameters to set parameter context for a process group; you can loop to do all. pgid => parameter group id pcid => parameter context id I need to put this in a shell or Python script: /opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-set-param-context -u http://edge2ai-1.dim.local:8080 -verbose -pgid 2c1860b3-7f21-36f4-a0b8-b415c652fc62 -pcid 39f0f296-0177-1000-ffff-ffffdccb6d90 Example setupnifi.sh (Github Link) You could also use the NiFi REST API or Dan's awesome Python API NiPyApi: A Python Client SDK for Apache NiFi References DevOps: Working with Parameter Contexts NiFi Toolkit CLI No More Spaghetti Flows Report on this Apache NiFi Everything Apache Nifi Cloudera Data Platform - Using Apache NiFi REST API in the Public Cloud Using NiFi CLI to Restore NiFi Flows From Backups Automating the Building, Migration, Backup, Restore and Testing of Streaming Applications Apache NiFi Toolkit Guide An overview of Apache NiFi and Toolkit CLI deployments Automate workflow deployment in Apache NiFi with the NiFi Registry DevOps for Apache NiFi 1.7 and More
... View more
Labels:
01-21-2021
11:44 AM
https://nipyapi.readthedocs.io/en/latest/ Can build flows with Python Code. Or you can write a custom nifi processor https://www.nifi.dev/2019/03/custom-processors.html
... View more
01-21-2021
08:25 AM
1 Kudo
use updaterecord to add fields
... View more
01-15-2021
06:32 AM
1 Kudo
Install Cloudera Data Platform Public Cloud Azure edition and there is a one button click to run a NiFi cluster as a datahub. Does all setup, configuration, optimization and security. https://docs.cloudera.com/cdf-datahub/7.2.1/nifi-azure-ingest/topics/cdf-datahub-fm-adls-ingest-overview.html
... View more
01-13-2021
07:10 AM
Here are a few examples of moving Flume flows to NiFi. https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html
... View more
01-13-2021
06:38 AM
Just a note for future that Flume is deprected and Cloudera Flow Management with Apache NiFi is the path forward for such workloads. https://docs.cloudera.com/runtime/7.0.3/release-notes/topics/cdpdc-rt-updated-cdh-components.html Versions are here: https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_63_packaging.html
... View more