1973
Posts
1225
Kudos Received
124
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
776 | 04-03-2024 06:39 AM | |
1425 | 01-12-2024 08:19 AM | |
771 | 12-07-2023 01:49 PM | |
1327 | 08-02-2023 07:30 AM | |
1922 | 03-29-2023 01:22 PM |
06-06-2023
11:33 AM
1 Kudo
Check with your IBM documentation as this is not standard character sets that non-mainframe platforms deal with. You need this converted to UTF-8 or other modern formats. JMS_IBM_Character_Set: IBM424 JMS_IBM_Encoding: 785 Check with IBM support. If you can convert to Unicode before extracting https://www.ibm.com/docs/en/ibm-mq/7.5?topic=conversion-jms-client-message-encoding You may need another driver https://www.ibm.com/support/pages/apar/IT18078
... View more
05-30-2023
06:13 AM
https://dev.to/tspannhw/simple-change-data-capture-cdc-with-sql-selects-via-apache-nifi-flank-19m4 You can use the metadata database processors to list all tables in a database and then read all values from tables
... View more
04-27-2023
10:03 AM
Also I had one with a Kudu cache for calling Daily Med https://github.com/tspannhw/ApacheConAtHome2020/tree/main/flows/DailyMed https://www.datainmotion.dev/2021/01/flank-using-apache-kudu-as-cache-for.html
... View more
04-27-2023
09:53 AM
I call the OpenAQ rest service and increment the pages https://github.com/tspannhw/flank-airquality/blob/main/flows/AirQuality.json
... View more
04-19-2023
01:10 PM
3 Kudos
Cloudera DataFlow Connectors We would love to have you and your fellow colleagues attend this session. Please feel free to share this internally. We look forward to having you join us! Reading from Iceberg https://github.com/tspannhw/FLaNK-DataFlows/blob/main/jdbc/README.md Tip: if you have missing core-site / CDP Environmental settings then paste these in to deploy. /home/nifi/additional/secret/env_config/core-site.xml,/home/nifi/additional/secret/env_config/hive-site.xml,/home/nifi/additional/secret/env_config/ssl-client.xml May 12, 2023 Attend the Webinar . Rewatch on demand. Register for the Best in Flow Data Flow Sandbox Environment during the event or after via email you received when signing up. Work on the the tutorials to get familiar. Join my Zoom Sessions on May 4 @1pm EST, May 5 @ 9am EST, May 8 @11am EST, and May 9 @ 9am and more . I will start more on demand if there is need. Sign up for the slack group and post if you need help. Build your own flows and experiment. You can use one of the available data streams or bring your own public datasets. Submit your favorite two flows for the competition. Tutorials To Get You Started https://github.com/tspannhw/FLaNK-TravelAdvisory/blob/main/steps.md https://github.com/tspannhw/FLaNK-DataFlows/blob/main/finaltutorials/BestInFlowCompetitionTutorials03May2023.pdf https://github.com/tspannhw/FLaNK-DataFlows/blob/main/finaltutorials/CloudToolGuidance03May2023.pdf How to Start Video Additional Resources Best Practices Guide Meet the Committers Lab Prep Daily Zoom Guidelines Developer Home Page April 25, 2023 SF DataFlow Meetup Video ExamplesCloudera DataFlow Designer: The Key to Agile Data Pipeline Development Streaming Data Ingestion into an Open Data Lakehouse Made Easy with DataFlow Example Cloudera DataFlow Designer: Kafka to Iceberg in Cloudera Data Warehouse Serverless NiFi Flows with DataFlow Functions DataFlow Functions Technical Demo DataFlow Documentation https://www.slideshare.net/bunkertor/best-practices-for-workflow https://www.slideshare.net/bunkertor/meet-the-committers-webinar-lab-preparation https://www.slideshare.net/bunkertor/cloudera-sandbox-event-guidelines-for-workflow * https://www.datainmotion.dev/2023/05/best-in-flow-competition-tutorials-part.html * https://www.datainmotion.dev/2023/05/best-in-flow-competition-tutorials-part_2.html * https://www.datainmotion.dev/2023/05/best-in-flow-competition-tutorials-part_43.html * https://www.datainmotion.dev/2023/05/best-in-flow-competition-tutorials-part_45.html * https://github.com/tspannhw/FLaNK-DataFlows/tree/main/tutorials * https://github.com/tspannhw/FLaNK-DataFlows/blob/main/tutorials/BestInFlowCompetitionTutorials.epub * https://github.com/tspannhw/FLaNK-DataFlows/blob/main/tutorials/BestInFlowCompetitionTutorials.docx * https://github.com/tspannhw/FLaNK-DataFlows/blob/main/tutorials/BestInFlowCompetitionTutorials.zip * https://github.com/tspannhw/FLaNK-DataFlows/blob/main/tutorials/Cloud%20Tools%20Guidance.epub * https://github.com/tspannhw/FLaNK-DataFlows/blob/main/tutorials/Cloud%20Tools%20Guidance.pdf * https://github.com/tspannhw/FLaNK-DataFlows/blob/main/tutorials/Cloud%20Tools%20Guidance.zip https://www.datainmotion.dev/2023/05/cloud-tools-guidance-how-to-build-data.html https://www.datainmotion.dev/2023/05/cloud-tools-guidance-how-to-build-data_2.html https://www.datainmotion.dev/2023/05/cloud-tools-guidance-how-to-build-data_18.html https://www.datainmotion.dev/2023/05/best-in-flow-competition-bring-your-own.html https://www.datainmotion.dev/2023/05/best-in-flow-competition-streaming-data.html https://www.youtube.com/@nifinotes5127 Quick Tips DNS-based ad-blocker prevented some JavaScript from loading Use a Chrome Browser Turn off VPNs and ad-blockers Use a fast enough internet connection Login if you are idle for more than 15 minutes Stop, terminate and delete things once you aren't using them Download and document your flows Prefix your work with your userid like this tim_.
... View more
Labels:
04-03-2023
08:12 AM
1 Kudo
CODE + COMMUNITY This week in FLaNK Stack Weekly, we have some events going on, a meetup in preparation and a lot of interesting new tools to explore. Please Join my meetup groups. We will be hybrid so if you are remote you can still see via zoom or Youtube. For those in the Princeton, New York City or Philadelphia are we will be in person as well. https://www.meetup.com/futureofdata-princeton/ https://www.meetup.com/futureofdata-newyork/ https://www.meetup.com/futureofdata-philadelphia I have a meetup in person at our San Francisco office. I will also be speaking at the Real-Time Analytics Summit that week. https://www.meetup.com/futureofdata-sanfrancisco/events/292453316/ This is Issue #77 and if you wish to look at all of our back issues, check them out in github. They are in a few different formats. https://github.com/tspannhw/FLiPStackWeekly I travel the world spreading the word of streaming, please join us. https://www.linkedin.com/pulse/schedule-2023-tim-spann-/ Videos These were the most interesting streaming videos of the week, check them out on Youtube. https://www.youtube.com/watch?v=iT60STl-Wuk https://www.youtube.com/watch?v=4X5Yky3CT6I&t=13s https://www.youtube.com/watch?v=V_DpqTo4bQ0 https://www.youtube.com/watch?v=p9-Y1PRYDn4&t=2s https://www.youtube.com/watch?v=s80sz3NWwHo Articles These were the most interesting articles of the week. https://community.cloudera.com/t5/What-s-New-Cloudera/Cloudera-DataFlow-Designer-for-self-service-data-flow/ba-p/366039 https://posthog.com/blog/dev-marketing-for-startups#its-ok-for-other-companies-to-be-much-better-than-you-at-social-media https://developerrelations.com/devrel-roundtable/looking-ahead-to-conference-season https://ossinsight.io/collections/chat-gpt-alternatives/ https://robertsahlin.substack.com/p/the-data-engineer-is-dead-long-live https://blogs.oracle.com/javamagazine/ https://www.infoq.com/articles/billions-messages-minute/? https://technology.amis.nl/big-data-database/apache-nifi-automating-tasks-using-nipyapi/ https://blog.twitter.com/engineering/en_us/topics/open-source/2023/twitter-recommendation-algorithm https://www.linkedin.com/posts/michael-kohs-27a17525_snowpipe-snowflake-nifi-activity-7047694779786084352-ArdL/ https://thenewstack.io/linkedin-unifies-stream-and-batch-processing-with-apache-beam/ Recent Talks This is my most recent talk at the Trenton Computer Festival, I spoke on Streaming. Trenton Computer Festival Pro https://www.slideshare.net/bunkertor/itpc-building-modern-data-streaming-apps https://www.youtube.com/watch?v=iT60STl-Wuk&list=PLIJGKvnQWB-u0SPXIwozegOWCG2V85WGe&index=12 Events I have a number of events coming up soon, check them out if you can. https://www.cloudera.com/about/events/evolve.html https://web.cvent.com/event/7598f981-2f7e-4915-b662-bd7be9b5f48d/summary?RefId=homepage_impact24 April 4-6, 2023: DevNexus: Atlanta, GA. In-Person. https://devnexus.com/ April 24-26, 2023: Real-Time Analytics Summit: San Francisco, CA. In-Person. https://rtasummit.com/ April 25, 2023: Future of Data Meetup: San Francisco, CA. In-Person. https://www.meetup.com/futureofdata-princeton/ https://www.meetup.com/futureofdata-sanfrancisco/events/292453316/ May 9, 2023: Garden State Java User Group. In-Person. New Jersey https://gsjug.org/ May 10-12, 2023: Open Source Summit North America. Virtual https://events.linuxfoundation.org/open-source-summit-north-america/ May 23, 2023: Pulsar Summit Europe. Virtual https://pulsar-summit.org/ Cloudera Events https://www.cloudera.com/about/events.html More Events: https://www.linkedin.com/pulse/schedule-2023-tim-spann-/ Code There are a couple of good demos with source code available, check them out. https://github.com/pdefusco/Oozie2CDE_Migration https://github.com/SuperEllipse/edge2ai_pred_maint https://github.com/tspannhw/FLaNK-AllTheStreams https://github.com/tspannhw/CloudDemo2023 Tools There are a lot of tools I have found in the open source to be very helpful. https://github.com/bencgreenberg/stackexchange-tutorial-themes https://github.com/jaymody/picoGPT https://regex.ai/ https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apache.nifi.processors.standard.JoinEnrichment/additionalDetails.html https://clickhouse.com/docs/en/integrations/nifi https://github.com/TheoKanning/openai-java https://pyscript.net/ https://tmate.io/ https://github.com/dylanaraps/neofetch https://github.com/jesseduffield/lazydocker https://github.com/httpie/httpie https://github.com/GothenburgBitFactory/taskwarrior https://github.com/newsboat/newsboat https://github.com/jarun/ddgr https://github.com/cointop-sh/cointop https://github.com/Byron/dua-cli https://nicolargo.github.io/glances/ https://github.com/aristocratos/bpytop https://github.com/hacker1024/coretemp https://github.com/bcicen/ctop https://github.com/imsnif/bandwhich https://github.com/jbruchon/jdupes https://exiftool.org/ https://github.com/aria2/aria2 https://github.com/muesli/duf https://github.com/ajeetdsouza/zoxide https://github.com/PrefectHQ/marvin https://github.com/libAudioFlux/audioFlux https://github.com/jamesturk/scrapeghost/ https://gut-cli.dev/ https://yakgpt.vercel.app/ https://github.com/HamburgChimps/apple-notes-liberator https://www.cursor.so/ https://orbstack.dev/ https://a16z.com/2023/03/30/b2b-generative-ai-synthai/ https://github.com/twitter/the-algorithm https://github.com/twitter/the-algorithm-ml https://github.com/fipso/ccurl.sh https://donuts-are-good.github.io/shhhbb/ Thanks for reading, same time next week! © 2023 Tim Spann
... View more
Labels:
03-30-2023
05:13 AM
If it does come up again, post here and we can put in a JIRA.
... View more
03-29-2023
01:22 PM
I haven't seen anyone do a key in avro, generally you want a simple key. Why avro as a key? Is this common for some use cases?
... View more
07-23-2021
05:42 AM
if you read a binary file it should be passed into NiFi with no issue.
... View more
07-22-2021
10:00 AM
You can have a QueryDataTableRecord to watch when changes happen and have that trigger your process. You may want to try Debezium with Cloudera Kafka You may want to try Debezium with Cloudera Flink SQL https://dev.to/tspannhw/simple-change-data-capture-cdc-with-sql-selects-via-apache-nifi-flank-19m4 See: https://github.com/tspannhw/EverythingApacheNiFi https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/replicate-track-change-data-capture-always-on-availability?view=sql-server-ver15 https://debezium.io/documentation/reference/connectors/sqlserver.html https://sandeepkattepogu.medium.com/streaming-data-from-microsoft-sql-server-into-apache-kafka-2fb53282115f https://www.linkedin.com/pulse/achieving-incremental-fetch-change-data-capture-via-apache-rajpal/ https://www.datainmotion.dev/2021/02/using-apache-nifi-in-openshift-and.html
... View more