1958
Posts
1211
Kudos Received
121
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
340 | 08-02-2023 07:30 AM | |
689 | 03-29-2023 01:22 PM | |
2581 | 06-03-2021 07:11 AM | |
1425 | 06-01-2021 10:05 AM | |
1412 | 05-24-2021 11:58 AM |
08-24-2023
06:41 AM
1 Kudo
I wonder if this was fixed 1.23.2. That seems like it shouldn't be too big, but it may be close. https://calcite.apache.org/docs/reference.html INTEGER, INT 4 byte signed integer Range is -2147483648 to 2147483647 try BIGINT
... View more
08-06-2023
06:07 PM
1 Kudo
You need to look at CDC tools like Cloudera Kafka Connect with Debezium or Cloudera SQL Streaming Builder with Debezium. Kafka Connect https://docs.cloudera.com/runtime/7.2.17/kafka-connect/topics/kafka-connect-connector-debezium-oracle.html SSB https://docs.cloudera.com/csa/1.10.0/how-to-ssb/topics/csa-ssb-cdc-connectors.html Or you can do Oracle GoldenGate with Kafka. I have some examples in this article https://medium.com/cloudera-inc/cdc-not-cat-data-capture-e43713879c03
... View more
08-02-2023
07:30 AM
1 Kudo
Under the cover this is just using Apache FTP Server Depends on passive ports https://mina.apache.org/ftpserver-project/configuration_passive_ports.html Send is over passive An example is here https://gist.github.com/tspannhw/43dbc81b1634a7acb61fada1a3c19a4e
... View more
07-12-2023
08:38 AM
Definitely upgrade to the latest NiFI which is 1.22.
... View more
06-29-2023
11:47 AM
https://medium.com/@tspann/ingesting-events-into-dockerized-ibm-db2-jdbc-with-apache-nifi-f0ca452d1351 It works fine with QueryDatabaseTableRecord and PutDatabaseRecord
... View more
06-06-2023
11:33 AM
1 Kudo
Check with your IBM documentation as this is not standard character sets that non-mainframe platforms deal with. You need this converted to UTF-8 or other modern formats. JMS_IBM_Character_Set: IBM424 JMS_IBM_Encoding: 785 Check with IBM support. If you can convert to Unicode before extracting https://www.ibm.com/docs/en/ibm-mq/7.5?topic=conversion-jms-client-message-encoding You may need another driver https://www.ibm.com/support/pages/apar/IT18078
... View more
05-30-2023
06:13 AM
https://dev.to/tspannhw/simple-change-data-capture-cdc-with-sql-selects-via-apache-nifi-flank-19m4 You can use the metadata database processors to list all tables in a database and then read all values from tables
... View more
04-27-2023
10:03 AM
Also I had one with a Kudu cache for calling Daily Med https://github.com/tspannhw/ApacheConAtHome2020/tree/main/flows/DailyMed https://www.datainmotion.dev/2021/01/flank-using-apache-kudu-as-cache-for.html
... View more
04-27-2023
09:53 AM
I call the OpenAQ rest service and increment the pages https://github.com/tspannhw/flank-airquality/blob/main/flows/AirQuality.json
... View more
04-19-2023
01:10 PM
3 Kudos
Cloudera DataFlow Connectors We would love to have you and your fellow colleagues attend this session. Please feel free to share this internally. We look forward to having you join us! Reading from Iceberg https://github.com/tspannhw/FLaNK-DataFlows/blob/main/jdbc/README.md Tip: if you have missing core-site / CDP Environmental settings then paste these in to deploy. /home/nifi/additional/secret/env_config/core-site.xml,/home/nifi/additional/secret/env_config/hive-site.xml,/home/nifi/additional/secret/env_config/ssl-client.xml May 12, 2023 Attend the Webinar . Rewatch on demand. Register for the Best in Flow Data Flow Sandbox Environment during the event or after via email you received when signing up. Work on the the tutorials to get familiar. Join my Zoom Sessions on May 4 @1pm EST, May 5 @ 9am EST, May 8 @11am EST, and May 9 @ 9am and more . I will start more on demand if there is need. Sign up for the slack group and post if you need help. Build your own flows and experiment. You can use one of the available data streams or bring your own public datasets. Submit your favorite two flows for the competition. Tutorials To Get You Started https://github.com/tspannhw/FLaNK-TravelAdvisory/blob/main/steps.md https://github.com/tspannhw/FLaNK-DataFlows/blob/main/finaltutorials/BestInFlowCompetitionTutorials03May2023.pdf https://github.com/tspannhw/FLaNK-DataFlows/blob/main/finaltutorials/CloudToolGuidance03May2023.pdf How to Start Video Additional Resources Best Practices Guide Meet the Committers Lab Prep Daily Zoom Guidelines Developer Home Page April 25, 2023 SF DataFlow Meetup Video ExamplesCloudera DataFlow Designer: The Key to Agile Data Pipeline Development Streaming Data Ingestion into an Open Data Lakehouse Made Easy with DataFlow Example Cloudera DataFlow Designer: Kafka to Iceberg in Cloudera Data Warehouse Serverless NiFi Flows with DataFlow Functions DataFlow Functions Technical Demo DataFlow Documentation https://www.slideshare.net/bunkertor/best-practices-for-workflow https://www.slideshare.net/bunkertor/meet-the-committers-webinar-lab-preparation https://www.slideshare.net/bunkertor/cloudera-sandbox-event-guidelines-for-workflow * https://www.datainmotion.dev/2023/05/best-in-flow-competition-tutorials-part.html * https://www.datainmotion.dev/2023/05/best-in-flow-competition-tutorials-part_2.html * https://www.datainmotion.dev/2023/05/best-in-flow-competition-tutorials-part_43.html * https://www.datainmotion.dev/2023/05/best-in-flow-competition-tutorials-part_45.html * https://github.com/tspannhw/FLaNK-DataFlows/tree/main/tutorials * https://github.com/tspannhw/FLaNK-DataFlows/blob/main/tutorials/BestInFlowCompetitionTutorials.epub * https://github.com/tspannhw/FLaNK-DataFlows/blob/main/tutorials/BestInFlowCompetitionTutorials.docx * https://github.com/tspannhw/FLaNK-DataFlows/blob/main/tutorials/BestInFlowCompetitionTutorials.zip * https://github.com/tspannhw/FLaNK-DataFlows/blob/main/tutorials/Cloud%20Tools%20Guidance.epub * https://github.com/tspannhw/FLaNK-DataFlows/blob/main/tutorials/Cloud%20Tools%20Guidance.pdf * https://github.com/tspannhw/FLaNK-DataFlows/blob/main/tutorials/Cloud%20Tools%20Guidance.zip https://www.datainmotion.dev/2023/05/cloud-tools-guidance-how-to-build-data.html https://www.datainmotion.dev/2023/05/cloud-tools-guidance-how-to-build-data_2.html https://www.datainmotion.dev/2023/05/cloud-tools-guidance-how-to-build-data_18.html https://www.datainmotion.dev/2023/05/best-in-flow-competition-bring-your-own.html https://www.datainmotion.dev/2023/05/best-in-flow-competition-streaming-data.html https://www.youtube.com/@nifinotes5127 Quick Tips DNS-based ad-blocker prevented some JavaScript from loading Use a Chrome Browser Turn off VPNs and ad-blockers Use a fast enough internet connection Login if you are idle for more than 15 minutes Stop, terminate and delete things once you aren't using them Download and document your flows Prefix your work with your userid like this tim_.
... View more
Labels: