Member since
07-07-2022
42
Posts
2
Kudos Received
8
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
4237 | 07-27-2022 12:05 PM | |
1692 | 07-25-2022 05:01 AM | |
916 | 07-21-2022 02:03 AM | |
2188 | 07-21-2022 01:52 AM | |
1353 | 07-19-2022 10:48 AM |
12-05-2022
02:19 AM
Hi All, 1. Is there any documentation to scale up, scale down and create infra for Apache NIFI in production environment?. If yes Please share it with me. 2. What are the minimum system requirements for deploying Superset (e.g. of RAM/storage/CPU etc). If yes Please share it with me. Thanks in advance,
... View more
Labels:
- Labels:
-
Apache NiFi
08-04-2022
03:19 AM
@araujo Changing a datatype of source DB in production environment will not be a good solution. Is there any other approach to handle Mysql JSON datatype in NiFi ready from Mysql binarylog? Thanks,
... View more
08-03-2022
10:39 PM
Hi, I connected the two mysql server and tried to both server DB. Both server data were received by both server captureChangeMySql processor. one server data was send to destination DB by EnforceOrder processor but it is giving error for second server flow file data. Can anyone suggest how to resolve this issue. Error Log: 2022-08-04 10:19:01,530 WARN [Timer-Driven Process Thread-3] o.a.n.processors.standard.EnforceOrder EnforceOrder[id=9c7bbf3f-df40-3d1a-6cc0-087c9d2052ce] Skipped, FlowFile order was 10 but current target is 897. StandardFlowFileRecord[uuid=ac8e8594-17b8-4e2a-872b-44af8dfea43b,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1659588192914-1, container=default, section=1], offset=3466, length=384],offset=0,name=ac8e8594-17b8-4e2a-872b-44af8dfea43b,size=384]
... View more
Labels:
- Labels:
-
Apache NiFi
08-03-2022
04:15 AM
Hi, I want to connect connect two different MySQL database services(mysql 5.7 and Mysql 8.0 ). I am using two two CaptureChangeMySQL Processors for this. Mysql 5.7 is install in my local machine and CaptureChangeMySQL Processors connecting successfully. Mysql 8.0 is install on docker container and on host machine port 3310 like (sudo docker run --name mysql-8.0-container -p 127.0.0.1:3310:3306 -e MYSQL_ROOT_PASSWORD=password -d mysql:8.0). CaptureChangeMySQL connecting with mysql 8.0 is giving error. Can any suggest why i am getting this error and how to resolve it. Note: I am able to connect mysql 8.0 using mysql workbench. Error: 2022-08-03 15:59:58,863 ERROR [Timer-Driven Process Thread-4] o.a.n.c.m.processors.CaptureChangeMySQL CaptureChangeMySQL[id=352daba9-0182-1000-a02a-faf160ab1329] Processing failed org.apache.nifi.processor.exception.ProcessException: Could not connect binlog client to any of the specified hosts due to: Client does not support authentication protocol requested by server; consider upgrading MySQL client at org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.setup(CaptureChangeMySQL.java:664) at org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.onTrigger(CaptureChangeMySQL.java:678) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1283) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) Thanks,
... View more
Labels:
- Labels:
-
Apache NiFi
07-28-2022
12:05 AM
Hi , I want to move source DB data to destination DB. Source db table has 'metadat' column, dataType='JSON'. I am using 'captureChangeMySql' processor to ready the change data from mysql binary-log file and put a data to destination db. 'captureChangeMySql' processor to reading the change data from mysql binary-log file but 'metadat' column, dataType='JSON' , value is showing as some encoded format. Can anyone suggest, how to decode, encoded data read by 'captureChangeMySql' processor flowfile output using any other NiFi processor? Original data in db: {"mappedProducts": ["20-4040", "20-4041", "20-4042", "20-4043"]} Flowfile output: [ { "id" : 18, "name" : "Transitional Licenses - Provider", "external_id" : "101", "product_tag" : null, "tenant_id" : 1, "metadata" : "\u0000\u0001\u0000I\u0000\u000B\u0000\u000E\u0000\u0002\u0019\u0000mappedProducts\u0004\u00000\u0000\f\u0010\u0000\f\u0018\u0000\f \u0000\f(\u0000\u000720-4040\u000720-4041\u000720-4042\u000720-4043", "duration_in_days" : null, "offline_completion_duration" : null, "product_code" : "cps_provider_7_ed", "external_code" : null, "external_url" : null, "short_description" : "This course is for Providers", "long_description" : "This course is for Providers", "meta_keyword" : null, "meta_description" : null, "meta_title" : null, "status" : "Active", "event_type" : "ESSENTIALS", "event_participation_type" : "REGISTER_ILE", "terms" : null, "product_type_id" : 1, "is_perpetual" : null, "sort_order" : "20", "transitional_sort_order" : null, "completion_cert_type" : null, "ecard_cert_type" : null, "ecard_validity" : null, "ecard_name" : null, "is_deleted" : "{}", "created_at" : "Thu Mar 04 13:15:53 IST 2021", "updated_at" : "Thu Mar 04 13:15:53 IST 2021", "created_by" : null, "updated_by" : null, "is_legacy" : "{0}", "has_ce" : "{}", "region" : "CA", "self_registration_override" : "{}", "allow_assignment" : "{0}", "default_license_type" : null, "dispatch_confirmation" : "{}", "multi_package" : "{}" } ]
... View more
Labels:
- Labels:
-
Apache NiFi
07-27-2022
12:05 PM
@cnelson2 Are you specifying the database to use in your DBCP connection pool service? -> yes I solve this issue. I have given Catalog Name : destinationDBName and Make empty schema Name value in putdatabase record processor. Now I am able to perform insert , update and delete action source to destination 'user' table. Thanks,
... View more
07-27-2022
06:59 AM
Hi, I am getting error "Record does not have a value for the PrimaryKey column 'HOST' " when I update a data in source db table. This issue is happening when destination DB table's name: 'user' or 'users'. It is working fine when destination table's name is change to 'report_user' (I mean when we change the table name 'user'or 'users' to something else which is not used as mysql internal table) I am not sure why it is trying to insert or update on internal mysql table as I am giving the destination DBName and tableName. Can anyone suggest how can we fix this issue without changing the destination table name? Error Log: PutDatabaseRecord[id=ff8ca68c-8252-3652-8f55-1044ad3f1bab] Failed to put Records to database for StandardFlowFileRecord[uuid=43e64992-a3d0-4a75-bb85-8d70c1f556d7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1658747398462-4, container=default, section=4], offset=20549, length=1214],offset=0,name=43e64992-a3d0-4a75-bb85-8d70c1f556d7,size=1214]. Routing to failure. org.apache.nifi.serialization.MalformedRecordException: Record does not have a value for the PrimaryKey column 'HOST' at org.apache.nifi.processors.standard.PutDatabaseRecord.normalizeKeyColumnNamesAndCheckForValues(PutDatabaseRecord.java:1367) PutDatabaseRecord processor flow file output: [ { "id" : 1, "first_name" : "john", "middle_name" : "s_test", "last_name" : "asjndjnas", "user_name" : "deleted_8AB-0034", "is_terms_accepted" : "true", "is_retail" : "true", "tenant_id" : 1, "email" : "sreejith.karonnan2@mailinator.com", "alternate_email" : null, "gender" : "M", "dob" : "2008-01-01", "sso_id" : null, "phone_number" : null, "status_id" : 2, "time_zone_id" : 1, "is_activated" : "true", "activated_at" : "2021-01-07 16:29:25", "is_deleted" : "false", "import_id" : null, "created_at" : "2021-01-07 16:29:25", "updated_at" : "2021-11-16 18:38:42", "created_by" : 123, "updated_by" : 3, "last_login_date" : null, "agreement_acceptance_date" : "2021-03-22 20:44:23", "user_ref_id" : "589RngS1", "is_admin" : "false", "is_distributor" : "true", "is_self_register" : "false", "ce_job_id" : null, "region" : "US", "recovery_status" : null, "hlc_user_name" : null, "allow_assignment" : false, "has_ce" : false, "product_launched_at" : null, "is_perpetual" : false, "learning_completed_at" : null, "enrolled_at" : null, "dispatch_confirmation" : false, "subscription_end_at" : null, "ecard_date" : null, "offline_completion_date" : null, "subscription_start_at" : null, "product_completed_at" : null, "online_completion_date" : null, "is_legacy" : false, "last_activity_at" : null, "multi_package" : false, "metadata" : null, "self_registration_override" : false } ] Thanks,
... View more
Labels:
- Labels:
-
Apache NiFi
07-26-2022
10:17 PM
Hi, I have install a NiFi setup and deign the flow file to move data from source mysql db to destination mysql db. Now I want to know how can I setup it in dev or QA environments. I was going to NiFi admin guide and came to know about NiFi Registry and NiFi toolkit feature. I have below doubts: 1. Is it mandatory to install NiFi toolkit along with NiFi to configure clustering and managing the cluster node? or without installing a NiFi toolkit, we can configure cluster and manage it? 2. Is it mandatory to install NiFi Registry along with NiFi to manage the flowfile change restriction?
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
07-25-2022
05:01 AM
@ckumar Are you suggesting to create two flow-file templates with respect to MySQL database service? Or This can be achieve by just using the the two CaptureChangeMySQL Processors in one template which will have the configuration with respect to MySQL database service. Attaching the current flow-file screenshot and also as per you suggestion with two captureChangeMySql processor. Please confirm If my understanding is correct as per you suggestion
... View more
07-25-2022
12:22 AM
Hi, Query 1: How to connect to two different source DB instants using captureChangeMySql processor?. Query 2: is it possible to configure two different source DB instants using captureChangeMySql processor to get data from both db instance and put it in destination DB?, If yes, Please suggest the configuration approach. Below is my current configuration screenshot to read the data from single instant connecting two source DB.
... View more
Labels:
- Labels:
-
Apache NiFi