Cloudera Community
Ibrahem
user rank
Explorer

SQL Stream Builder (SSB) Flink push to Kafka

by Explorer in Support Questions
‎10-19-2022 10:50 PM
‎10-19-2022 10:50 PM
Hi I'm using SSB and Kafka to read from a topic and do some transformation and output the result to another topic.   Here is what I did: 1. Create a Kafka Topic (lets name input_topic) 2. pushed some events, so SSB can detect schema in the next step. 3. Created a Table in SSB Console using the wizard and I detected the schema properly. Table uses Local Kafka option (CDP Kafka) 4. I wrote my query (complex queries with join, windows and CTEs) the query runs perfectlly when I run it I can see the result from SSN console as expected. 5. I created a view from the query i named it (input_topic_v) 6. I created the output topic (named it output_topic) with no data its new topic. 7. I created local-kafka table using the template pointing to output_topic. 8. I wrote INSERT INTO output_topic SELECT * FROM input_topic_v.   and i get the following the error. (I need more logs to debug but this the response i get) [10/20/2022, 8:34:32 AM][INFO] StreamBuilder job awesome_allen is starting. [10/20/2022, 8:34:32 AM][ERROR] Job execution failed: Error while creating streaming job awesome_allen   after a while tried this: (dummy select statement with needed schema) I NSERT INTO output_topic select 1,2,3,4,5, 's'   and the query exacted without any issue and the sink worked and data pushed to Kafka So the sink works and SSB can push data to Kafka. and the query from View works and I can see the result from Console but when I sink the data from the view it does not.   ... View more
Labels:

Streaming SQL Builder - TUMBLING windows and Water...

by Explorer in Support Questions
‎10-15-2022 11:24 PM
‎10-15-2022 11:24 PM
Hi   I'm using SSB to create a table from a Kafka topic (using kafka timestamp as watermark eventTimestamp) and doing a TUMBLING windows function to aggregate the data evert 5 minutes and push the data to another table (Kafka Topic).   The latest event is always delayed until I push another one.   If SSB query is running, and I push an event into Kafka (topic_1) no data will be pushed to topic_2. if push another event to topic_1 then the previous message gets pushed to topic_2 and so on.   Here is a stackoverflow issue of someone who had this exact problem, but I'm unable to do the solution in SSB.   Can you help translate this to SSB? https://stackoverflow.com/questions/58907070/apache-flink-tumbling-window-delayed-result   Here is the query: insert into topic_2 select process_code, window_start, window_end, cast(COUNT(*) as int) as total_events FROM TABLE( TUMBLE( TABLE topic_1, DESCRIPTOR(eventTimestamp), INTERVAL '5' minute)) GROUP BY process_code, window_start, window_end ... View more
Labels:

Re: Jolt Transformation (nested array modification...

by Explorer in Support Questions
‎02-22-2022 11:48 PM
1 Kudo
‎02-22-2022 11:48 PM
1 Kudo
THANK YOU SO MUCH. It makes total sense.     ... View more

Re: Jolt Transformation (nested array modification...

by Explorer in Support Questions
‎02-22-2022 03:44 AM
‎02-22-2022 03:44 AM
Thank you so much @araujo for the help.   Is it possible to have the new flatten array and the original array within one jolt spec?   This is the current structure    can we change it to this:   ... View more

Jolt Transformation (nested array modifications)

by Explorer in Support Questions
‎02-18-2022 07:31 AM
‎02-18-2022 07:31 AM
Hi everyone!   I hope all is well.   I have this json    {     "Data": {         "ID": "09878666",         "DATE": "2022-01-01",         "ARRAY_ONE": [             {                 "NAME": "test_1",                 "details": [                     {                         "address": "123123"                     }                 ]             },             {                 "NAME": "test_2",                 "details": [                     {                         "address": "123123"                     },                     {                         "address": "123123"                     }                 ]             }         ]     } }       and I transformed the structure to this: [     {         "ID": "09878666",         "DATE": "2022-01-01",         "NAME": "test_1",         "details": [             {                 "address": "123123"             }         ]     },     {         "ID": "09878666",         "DATE": "2022-01-01",         "NAME": "test_2",         "details": [             {                 "address": "123123"             },             {                 "address": "123123"             }         ]     } ]   Which is perfect except for the inner arrays, I want to flatten the array  to be like this [     {         "ID": "09878666",         "DATE": "2022-01-01",         "NAME": "test_1",         "details_0_adrress":"123123"             },     {         "ID": "09878666",         "DATE": "2022-01-01",         "NAME": "test_2",         "details_0_adrress":"123123",         "details_1_adrress":"123123"     } ]     Here is my spec: [     {         "operation": "shift",         "spec": {             "Data": {                 "ARRAY_ONE": {                     "*": {                         "@(2,ID)": "[#2].ID",                         "@(2,DATE)": "[#2].DATE",                         "NAME": "[#2].NAME",                         "details": "[#2].details"                     }                 }             }         }     } ]         I know how to flatten an array but I'm not sure how to do it using the current spec:, What I would normally do is something like this: {         "operation": "shift",         "spec": {             "details": {                 "*": {                     "address": "details_&1_address"                 }             }         }     }   But it did not work so well.     Thank you! ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎11-10-2023 03:02 PM
Community Statistics
Member Since ‎08-01-2021 12:10 AM
Last Visited ‎11-10-2023 03:02 PM
Posts 7
Kudos received 1