Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi CaptureChangeMySQL Binlog-Connector not working

Highlighted

Nifi CaptureChangeMySQL Binlog-Connector not working

New Contributor

We use Nifi 1.6.0 running via Docker. For Testing I set up with Laragon a mariadb Server (10.2.3).

This is my setup:

  1. I created a DistributedMapCacheClientService with following Settings

    https://i.imgur.com/aqMCCoS.png (Image also attached)

  2. I created a CaptureChangeMySQL-Processor and routed to a output:

https://i.imgur.com/Hbb4lUt.png (Image also attached)

  • I configured the CaptureChangeMySQL-Processor with following Settings:
  • https://i.imgur.com/ZFeB3rV.png (Image also attached)

    This is the Error output:

    https://paste.ofcode.org

     2019-01-04 08:56:23,347 INFO [blc-NB-ASC:3306] c.g.shyiko.mysql.binlog.BinaryLogClient Connected to NB-ASC:3306 at mysql-bin.000001/2638 (sid:1, cid:29)
    nifi_1           | 2019-01-04 08:56:23,356 ERROR [Timer-Driven Process Thread-7] o.a.n.c.m.processors.CaptureChangeMySQL CaptureChangeMySQL[id=0db5b2fa-0168-1000-7a91-92d0c2a830d3] Binlog connector communications failure: Client requested master to start replication from impossible position; the first event 'mysql-bin.000001' at 2638, the last event read from 'mysql-bin.000001' at 4, the last byte read from 'mysql-bin.000001' at 4.: com.github.shyiko.mysql.binlog.network.ServerException: Client requested master to start replication from impossible position; the first event 'mysql-bin.000001' at 2638, the last event read from 'mysql-bin.000001' at 4, the last byte read from 'mysql-bin.000001' at 4.
    nifi_1           | com.github.shyiko.mysql.binlog.network.ServerException: Client requested master to start replication from impossible position; the first event 'mysql-bin.000001' at 2638, the last event read from 'mysql-bin.000001' at 4, the last byte read from 'mysql-bin.000001' at 4.
    nifi_1           |      at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:882)
    nifi_1           |      at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:559)
    nifi_1           |      at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:793)
    nifi_1           |      at java.lang.Thread.run(Thread.java:748)
    

    This is my MySQL Server Configuration File (my.ini):

    https://paste.ofcode.org/sF6DNwumG8qpAmgygsynKF

    # Example MariaDB config file for large systems.
    #
    # This is for a large system with memory = 512M where the system runs mainly
    # MariaDB.
    #
    # MariaDB programs look for option files in a set of
    # locations which depend on the deployment platform.
    # You can copy this option file to one of those
    # locations. For information about these locations, do:
    # 'my_print_defaults --help' and see what is printed under
    # Default options are read from the following files in the given order:
    # More information at: http://dev.mysql.com/doc/mysql/en/option-files.html
    #
    # In this file, you can use all long options that a program supports.
    # If you want to know which options a program supports, run the program
    # with the "--help" option.
    # The following options will be passed to all MariaDB clients
    [client]
    #password = your_password
    port  = 3306
    socket  = /tmp/mysql.sock
    # Here follows entries for some specific programs
    # The MariaDB server
    [mysqld]
    port  = 3306
    socket  = /tmp/mysql.sock
    skip-external-locking
    key_buffer_size = 256M
    max_allowed_packet = 512M
    table_open_cache = 256
    sort_buffer_size = 1M
    read_buffer_size = 1M
    read_rnd_buffer_size = 4M
    myisam_sort_buffer_size = 64M
    thread_cache_size = 8
    query_cache_size= 16M
    # Try number of CPU's*2 for thread_concurrency
    # thread_concurrency = 8
    # Point the following paths to different dedicated disks
    #tmpdir  = /tmp/
    # Don't listen on a TCP/IP port at all. This can be a security enhancement,
    # if all processes that need to connect to mysqld run on the same host.
    # All interaction with mysqld must be made via Unix sockets or named pipes.
    # Note that using this option without enabling named pipes on Windows
    # (via the "enable-named-pipe" option) will render mysqld useless!
    # 
    #skip-networking
    # Replication Master Server (default)
    # binary logging is required for replication
    log-bin=mysql-bin
    # binary logging format - mixed recommended
    binlog_format=mixed
    binlog_do_db =test
    # required unique id between 1 and 2^32 - 1
    # defaults to 1 if master-host is not set
    # but will not function as a master if omitted
    server-id = 1
    # Replication Slave (comment out master section to use this)
    #
    # To configure this host as a replication slave, you can choose between
    # two methods :
    #
    # 1) Use the CHANGE MASTER TO command (fully described in our manual) -
    #    the syntax is:
    #
    #    CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
    #    MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
    #
    #    where you replace <host>, <user>, <password> by quoted strings and
    #    <port> by the master's port number (3306 by default).
    #
    #    Example:
    #
    #    CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
    #    MASTER_USER='joe', MASTER_PASSWORD='secret';
    #
    # OR
    #
    # 2) Set the variables below. However, in case you choose this method, then
    #    start replication for the first time (even unsuccessfully, for example
    #    if you mistyped the password in master-password and the slave fails to
    #    connect), the slave will create a master.info file, and any later
    #    change in this file to the variables' values below will be ignored and
    #    overridden by the content of the master.info file, unless you shutdown
    #    the slave server, delete master.info and restart the slaver server.
    #    For that reason, you may want to leave the lines below untouched
    #    (commented) and instead use CHANGE MASTER TO (see above)
    #
    # required unique id between 2 and 2^32 - 1
    # (and different from the master)
    # defaults to 2 if master-host is set
    # but will not function as a slave if omitted
    #server-id       = 2
    #
    # The replication master for this slave - required
    #master-host     =   <hostname>
    #
    # The username the slave will use for authentication when connecting
    # to the master - required
    #master-user     =   <username>
    #
    # The password the slave will authenticate with when connecting to
    # the master - required
    #master-password =   <password>
    #
    # The port the master is listening on.
    # optional - defaults to 3306
    #master-port     =  <port>
    #
    # binary logging - not required for slaves, but recommended
    #log-bin=mysql-bin
    # Uncomment the following if you are using InnoDB tables
    #innodb_data_home_dir = C:\mysql\data\
    #innodb_data_file_path = ibdata1:10M:autoextend
    #innodb_log_group_home_dir = C:\mysql\data\
    # You can set .._buffer_pool_size up to 50 - 80 %
    # of RAM but beware of setting memory usage too high
    #innodb_buffer_pool_size = 256M
    #innodb_additional_mem_pool_size = 20M
    # Set .._log_file_size to 25 % of buffer pool size
    #innodb_log_file_size = 64M
    #innodb_log_buffer_size = 8M
    #innodb_flush_log_at_trx_commit = 1
    #innodb_lock_wait_timeout = 50
    sql_mode = "STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
    #show_compatibility_56 = on
    datadir= "C:/laragon/data"
    [mysqldump]
    quick
    max_allowed_packet = 512M
    [mysql]
    no-auto-rehash
    # Remove the next comment character if you are not familiar with SQL
    #safe-updates
    [myisamchk]
    key_buffer_size = 128M
    sort_buffer_size = 128M
    read_buffer = 2M
    write_buffer = 2M
    [mysqlhotcopy]
    interactive-timeout
    [server]
    default-time-zone= +00:00
    


    This is the generated mysql-bin.000001 File:

    https://paste.ofcode.org/8gz22DtnK4fSyhMbmiz926

    þbin/\   ü        10.2.3-MariaDB-log                                /\8
         ä       
                                                                                                                                     
    
    ½ðl«/\£                ‚ëâ/\¡   +   H       mysql-bin.000001='•æ
    

    I Tried this: https://mariadb.com/resources/blog/client-requested-master-to-start-replication-from-impossible-posi...

    This generated a master.info file. Here the master.info File:

    https://paste.ofcode.org/er4b3BytPyDZeyXDBXEMss

    33
    mysql-bin.000001
    4
    3306
    60
    0
    0
    1800.000
    0
    0
    using_gtid=0
    do_domain_ids=0
    ignore_domain_ids=0
    END_MARKER
    

    I hope I provided all Files necessary.

    Really don´t know why it doesn´t work.

    Thanks in advance.

    96617-strokesplus-2019-01-04-09-53-34.png

    96616-opera-2019-01-04-09-46-39.png

    96615-strokesplus-2019-01-04-09-47-17.png