Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi CaptureChangeMySQL Binlog-Connector not working


Nifi CaptureChangeMySQL Binlog-Connector not working

New Contributor

We use Nifi 1.6.0 running via Docker. For Testing I set up with Laragon a mariadb Server (10.2.3).

This is my setup:

  1. I created a DistributedMapCacheClientService with following Settings (Image also attached)

  2. I created a CaptureChangeMySQL-Processor and routed to a output: (Image also attached)

  • I configured the CaptureChangeMySQL-Processor with following Settings:
  • (Image also attached)

    This is the Error output:

     2019-01-04 08:56:23,347 INFO [blc-NB-ASC:3306] c.g.shyiko.mysql.binlog.BinaryLogClient Connected to NB-ASC:3306 at mysql-bin.000001/2638 (sid:1, cid:29)
    nifi_1           | 2019-01-04 08:56:23,356 ERROR [Timer-Driven Process Thread-7] o.a.n.c.m.processors.CaptureChangeMySQL CaptureChangeMySQL[id=0db5b2fa-0168-1000-7a91-92d0c2a830d3] Binlog connector communications failure: Client requested master to start replication from impossible position; the first event 'mysql-bin.000001' at 2638, the last event read from 'mysql-bin.000001' at 4, the last byte read from 'mysql-bin.000001' at 4.: Client requested master to start replication from impossible position; the first event 'mysql-bin.000001' at 2638, the last event read from 'mysql-bin.000001' at 4, the last byte read from 'mysql-bin.000001' at 4.
    nifi_1           | Client requested master to start replication from impossible position; the first event 'mysql-bin.000001' at 2638, the last event read from 'mysql-bin.000001' at 4, the last byte read from 'mysql-bin.000001' at 4.
    nifi_1           |      at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(
    nifi_1           |      at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(
    nifi_1           |      at com.github.shyiko.mysql.binlog.BinaryLogClient$
    nifi_1           |      at

    This is my MySQL Server Configuration File (my.ini):

    # Example MariaDB config file for large systems.
    # This is for a large system with memory = 512M where the system runs mainly
    # MariaDB.
    # MariaDB programs look for option files in a set of
    # locations which depend on the deployment platform.
    # You can copy this option file to one of those
    # locations. For information about these locations, do:
    # 'my_print_defaults --help' and see what is printed under
    # Default options are read from the following files in the given order:
    # More information at:
    # In this file, you can use all long options that a program supports.
    # If you want to know which options a program supports, run the program
    # with the "--help" option.
    # The following options will be passed to all MariaDB clients
    #password = your_password
    port  = 3306
    socket  = /tmp/mysql.sock
    # Here follows entries for some specific programs
    # The MariaDB server
    port  = 3306
    socket  = /tmp/mysql.sock
    key_buffer_size = 256M
    max_allowed_packet = 512M
    table_open_cache = 256
    sort_buffer_size = 1M
    read_buffer_size = 1M
    read_rnd_buffer_size = 4M
    myisam_sort_buffer_size = 64M
    thread_cache_size = 8
    query_cache_size= 16M
    # Try number of CPU's*2 for thread_concurrency
    # thread_concurrency = 8
    # Point the following paths to different dedicated disks
    #tmpdir  = /tmp/
    # Don't listen on a TCP/IP port at all. This can be a security enhancement,
    # if all processes that need to connect to mysqld run on the same host.
    # All interaction with mysqld must be made via Unix sockets or named pipes.
    # Note that using this option without enabling named pipes on Windows
    # (via the "enable-named-pipe" option) will render mysqld useless!
    # Replication Master Server (default)
    # binary logging is required for replication
    # binary logging format - mixed recommended
    binlog_do_db =test
    # required unique id between 1 and 2^32 - 1
    # defaults to 1 if master-host is not set
    # but will not function as a master if omitted
    server-id = 1
    # Replication Slave (comment out master section to use this)
    # To configure this host as a replication slave, you can choose between
    # two methods :
    # 1) Use the CHANGE MASTER TO command (fully described in our manual) -
    #    the syntax is:
    #    MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
    #    where you replace <host>, <user>, <password> by quoted strings and
    #    <port> by the master's port number (3306 by default).
    #    Example:
    #    CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
    #    MASTER_USER='joe', MASTER_PASSWORD='secret';
    # OR
    # 2) Set the variables below. However, in case you choose this method, then
    #    start replication for the first time (even unsuccessfully, for example
    #    if you mistyped the password in master-password and the slave fails to
    #    connect), the slave will create a file, and any later
    #    change in this file to the variables' values below will be ignored and
    #    overridden by the content of the file, unless you shutdown
    #    the slave server, delete and restart the slaver server.
    #    For that reason, you may want to leave the lines below untouched
    #    (commented) and instead use CHANGE MASTER TO (see above)
    # required unique id between 2 and 2^32 - 1
    # (and different from the master)
    # defaults to 2 if master-host is set
    # but will not function as a slave if omitted
    #server-id       = 2
    # The replication master for this slave - required
    #master-host     =   <hostname>
    # The username the slave will use for authentication when connecting
    # to the master - required
    #master-user     =   <username>
    # The password the slave will authenticate with when connecting to
    # the master - required
    #master-password =   <password>
    # The port the master is listening on.
    # optional - defaults to 3306
    #master-port     =  <port>
    # binary logging - not required for slaves, but recommended
    # Uncomment the following if you are using InnoDB tables
    #innodb_data_home_dir = C:\mysql\data\
    #innodb_data_file_path = ibdata1:10M:autoextend
    #innodb_log_group_home_dir = C:\mysql\data\
    # You can set .._buffer_pool_size up to 50 - 80 %
    # of RAM but beware of setting memory usage too high
    #innodb_buffer_pool_size = 256M
    #innodb_additional_mem_pool_size = 20M
    # Set .._log_file_size to 25 % of buffer pool size
    #innodb_log_file_size = 64M
    #innodb_log_buffer_size = 8M
    #innodb_flush_log_at_trx_commit = 1
    #innodb_lock_wait_timeout = 50
    #show_compatibility_56 = on
    datadir= "C:/laragon/data"
    max_allowed_packet = 512M
    # Remove the next comment character if you are not familiar with SQL
    key_buffer_size = 128M
    sort_buffer_size = 128M
    read_buffer = 2M
    write_buffer = 2M
    default-time-zone= +00:00

    This is the generated mysql-bin.000001 File:

    þbin/\   ü        10.2.3-MariaDB-log                                /\8
    ½ðl«/\£                ‚ëâ/\¡   +   H       mysql-bin.000001='•æ

    I Tried this:

    This generated a file. Here the File:


    I hope I provided all Files necessary.

    Really don´t know why it doesn´t work.

    Thanks in advance.