About sahil0915

sahil0915 · ‎08-19-2023

I am not looking for replication from nifi to different data centers. I wanted to check missing records which are already replicated in data centers Thanks

sahil0915 · ‎08-19-2023

Thanks for the reply @MattWho Cant we acheive this using detectDuplicate processors. I have done a POC first on only 2 tables. Lets say we have only 10 records in table1 and copied 8 records in table 2. I fetch all the records using QueryDatabaseTable processor from both tables and process each flow file in cryptographicHashConent and DetectDuplicate processor. anyway the results are not expected. I have pasted the template file , but please have a look and let me know if we could detect the records which exists only once in flow files or text files(if I fetch records from file). <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <template encoding-version="1.3"> <description></description> <groupId>018a1075-885b-197d-61c4-6910ed948be3</groupId> <name>consistencyCheck approach 1</name> <snippet> <processors> <id>c18e2f53-136e-3e5b-0000-000000000000</id> <parentGroupId>2629c87c-99e6-3b17-0000-000000000000</parentGroupId> <position> <x>0.0</x> <y>0.0</y> </position> <bundle> <artifact>nifi-avro-nar</artifact> <group>org.apache.nifi</group> <version>1.22.0</version> </bundle> <config> <backoffMechanism>PENALIZE_FLOWFILE</backoffMechanism> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>JSON container options</key> <value> <name>JSON container options</name> </value> </entry> <entry> <key>Wrap Single Record</key> <value> <name>Wrap Single Record</name> </value> </entry> <entry> <key>Avro schema</key> <value> <name>Avro schema</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <maxBackoffPeriod>10 mins</maxBackoffPeriod> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>JSON container options</key> <value>array</value> </entry> <entry> <key>Wrap Single Record</key> <value>false</value> </entry> <entry> <key>Avro schema</key> </entry> </properties> <retryCount>10</retryCount> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>ConvertAvroToJSON</name> <relationships> <autoTerminate>true</autoTerminate> <name>failure</name> <retry>false</retry> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>success</name> <retry>false</retry> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.avro.ConvertAvroToJSON</type> </processors> </snippet> <timestamp>08/19/2023 21:23:10 IST</timestamp> </template>

sahil0915 · ‎08-17-2023

Hi, I have 3 data centers dc1,dc2,dc3. There are about 100 million records migrated to dc1 and replicated to both dc2 and dc3. My goal is to find those records which are not replicated to dc2 and dc3 from dc1. Thanks in advance.

sahil0915 · ‎09-18-2022

Hello Everyone, Greetings! I need help in JoltTransformJson processor. I have GALLERY_IDS string in flow attribute as mentioned in the below screenshot: I want to fetch each value from GALLERY_IDS attribute and pass it to JoltTransformJson processor as mentioned in below snapshot In the output queue, I should get flow file for each gallery with personid. {"GALLERY_ID:"1","PERSON_ID":"test1"} {"GALLERY_ID:"2","PERSON_ID":"test2"} {"GALLERY_ID:"3","PERSON_ID":"test3"} {"GALLERY_ID:"4","PERSON_ID":"test4"} {"GALLERY_ID:"5","PERSON_ID":"test5"} {"GALLERY_ID:"6","PERSON_ID":"test6"} {"GALLERY_ID:"7","PERSON_ID":"test7"} Kindly help. Thanks in advance!

sahil0915 · ‎06-30-2022

I want to insert splitted records from splitjson processor in database table using nifi. For eg: Below is splitjson processor output. I want to push all the flow files in database table having same fragment.identifier. table columns are mentioned below: identity_id|json|person_ref where identity_id should be auto sequence number json would flow-file content person_ref is the foreign key having int data type so table will have 13 entries and identity_id column will be unique numbers in sequence, json column will have different flow file contents and person_ref column should be same for all 13 entries lets say 111.

sahil0915 · ‎04-15-2022

field to add "quality" input json : { "json_data": { "id": "1234", "items": [ { "id": "000000", "metadata": { "newId": "82896888047600" } } ] } } output json : { "json_data": { "id": "1234", "items": [ { "id": "000000", "metadata": { "newId": "82896888047600", "quality" :"good" } } ] } }

Online	Offline
Last Visited	‎10-06-2023 05:37 AM

Member Since	‎04-14-2022 01:24 AM
Last Visited	‎10-06-2023 05:37 AM
Posts	9

Cloudera Community

Re: verify data replication in different data cent...

Re: verify data replication in different data cent...

verify data replication in different data centers ...

traverse list of strings of flow attribute in Jol...

insert splitted records in database using apache n...

add node in json array using Apache nifi