Mafiree logo
  • About
  • Services
  • Blogs
  • Careers
  • Products
    • orbit logo Orbit
    • streamer logo Xstreami
  • Contact
Schedule a Call
Menu
  • About
  • Services
  • Blogs
  • Careers
  • Products
    • orbit logo Orbit
    • streamer logo Xstreami
  • Contact
  • Schedule a Call
Database
Database Managed Database Services
MySQL MySQL
MySQL Consulting
MySQL Migration Services
MySQL Optimization & Query Tuning
MySQL Database Administration
MySQL Backup & Recovery
MySQL Security & Maintenance
MySQL Cloud Services (AWS RDS, Aurora, Google Cloud SQL, Azure)
MySQL for Ecommerce
MySQL High Availability & Replication
MongoDB MongoDB
MongoDB Consulting
MongoDB Migration Services
MongoDB Optimization & Query Tuning
MongoDB Database Administration
MongoDB Backup & Recovery
MongoDB Security & Maintenance
MongoDB Cloud (Atlas)
MongoDB Solutions by Industry
MongoDB High Availability & Replication
PostgreSQL PostgreSQL
PostgreSQL Consulting
PostgreSQL Migration & Upgrades
Performance Tuning & Query Optimization
PostgreSQL Administration & Managed Services
High Availability, Clustering & Replication
PostgreSQL Backup, Recovery & Disaster Planning
PostgreSQL Security, Compliance & Auditing
PostgreSQL for Analytics & Data Warehousing
PostgreSQL on Cloud & Containers
PostgreSQL Extensions & Open-Source Integrations
PostgreSQL for Every Industry
SQL Server MSSQL
MSSQL Consulting
MSSQL Migration Services
MSSQL Optimization & Query Tuning Services
MSSQL Database Administration Services
MSSQL Backup & Recovery Services
MSSQL High Availability & Replication Services
MSSQL Security & Compliance Services
MSSQL Performance Monitoring & Health Checks
MSSQL Solutions by Industry
Aerospike Aerospike
Aerospike Consulting
Aerospike Migration Services
Aerospike Performance Optimization & Tuning
Aerospike Backup & Recovery
Aerospike High Availability
Aerospike Cloud & Hybrid Deployments
Aerospike for Real-Time Applications (AdTech, FinTech, Retail, IoT)
Clickhouse Clickhouse
ClickHouse Consulting
ClickHouse Migration Services
ClickHouse Optimization & Query Tuning
ClickHouse Database Administration
ClickHouse Backup & Recovery
ClickHouse Security & Maintenance
ClickHouse Cloud Services (ClickHouse Cloud, AWS, GCP, Azure)
ClickHouse Solutions by Industry
ClickHouse High Availability & Replication
TiDB TiDB
TiDB Consulting
TiDB Administration & Maintenance
TiDB Security and Privacy Maintenance
TiDB Performance & Query Optimization
TiDB Migration Services
TiDB Backup & Disaster Recovery
TiDB High Availability Solutions
TiDB Solutions by Industry
TiDB Cloud Services
ScyllaDB ScyllaDB
ScyllaDB Consulting
ScyllaDB Administration & Maintenance
ScyllaDB Security and Privacy Maintenance
ScyllaDB Performance & Query Optimization
ScyllaDB Migration Services
ScyllaDB Backup & Disaster Recovery
ScyllaDB High Availability Solutions
ScyllaDB Solutions by Industry
ScyllaDB Cloud Services
DevOps
DevOps DevOps Services
Version Control Version Control
Kubernetes Kubernetes
Infrastructure Infrastructure Management
Web Servers Web Servers
Networking
Networking Networking Services
Basic Basic
Advanced Advanced
MySQL MySQL
MongoDB MongoDB
PostgreSQL PostgreSQL
MSSQL MSSQL
Aerospike Aerospike
Clickhouse Clickhouse
TiDB TiDB
ScyllaDB ScyllaDB
Version Control Version Control
Kubernetes Kubernetes
Infrastructure Infrastructure Management
Web Servers Web Servers
Basic Basic
Advanced Advanced
MySQL Consulting
MySQL Migration Services
MySQL Optimization & Query Tuning
MySQL Database Administration
MySQL Backup & Recovery
MySQL Security & Maintenance
MySQL Cloud Services (AWS RDS, Aurora, Google Cloud SQL, Azure)
MySQL for Ecommerce
MySQL High Availability & Replication
MongoDB Consulting
MongoDB Migration Services
MongoDB Optimization & Query Tuning
MongoDB Database Administration
MongoDB Backup & Recovery
MongoDB Security & Maintenance
MongoDB Cloud (Atlas)
MongoDB Solutions by Industry
MongoDB High Availability & Replication
PostgreSQL Consulting
PostgreSQL Migration & Upgrades
Performance Tuning & Query Optimization
PostgreSQL Administration & Managed Services
High Availability, Clustering & Replication
PostgreSQL Backup, Recovery & Disaster Planning
PostgreSQL Security, Compliance & Auditing
PostgreSQL for Analytics & Data Warehousing
PostgreSQL on Cloud & Containers
PostgreSQL Extensions & Open-Source Integrations
PostgreSQL for Every Industry
MSSQL Consulting
MSSQL Migration Services
MSSQL Optimization & Query Tuning Services
MSSQL Database Administration Services
MSSQL Backup & Recovery Services
MSSQL High Availability & Replication Services
MSSQL Security & Compliance Services
MSSQL Performance Monitoring & Health Checks
MSSQL Solutions by Industry
Aerospike Consulting
Aerospike Migration Services
Aerospike Performance Optimization & Tuning
Aerospike Backup & Recovery
Aerospike High Availability
Aerospike Cloud & Hybrid Deployments
Aerospike for Real-Time Applications (AdTech, FinTech, Retail, IoT)
ClickHouse Consulting
ClickHouse Migration Services
ClickHouse Optimization & Query Tuning
ClickHouse Database Administration
ClickHouse Backup & Recovery
ClickHouse Security & Maintenance
ClickHouse Cloud Services (ClickHouse Cloud, AWS, GCP, Azure)
ClickHouse Solutions by Industry
ClickHouse High Availability & Replication
TiDB Consulting
TiDB Administration & Maintenance
TiDB Security and Privacy Maintenance
TiDB Performance & Query Optimization
TiDB Migration Services
TiDB Backup & Disaster Recovery
TiDB High Availability Solutions
TiDB Solutions by Industry
TiDB Cloud Services
ScyllaDB Consulting
ScyllaDB Administration & Maintenance
ScyllaDB Security and Privacy Maintenance
ScyllaDB Performance & Query Optimization
ScyllaDB Migration Services
ScyllaDB Backup & Disaster Recovery
ScyllaDB High Availability Solutions
ScyllaDB Solutions by Industry
ScyllaDB Cloud Services
  1. Home
  2. > Blogs
  3. > MySQL
  4. > Real time data streaming from MariaDB to Columnstore

Real time data streaming from MariaDB to Columnstore

How real time data from transactional database(mariadb) to analytic database(columnstore) is achieved using MariaDB's maxscale binlog streaming

sukan September 11, 2026

Subscribe for email updates

Summarize with AI: ChatGPT Google AI Perplexity Claude Grok

In the recent past there has been a massive business shift towards analytics database. Unlike traditional databases, analytics database always stands out in running big data queries. No matter how much we optimize, transaction databases are not suitable for these kind of workload.

 

Business these days are gripped in running statistics on top of everything they have, for them these reports are critical as they can analyse, optimize and apply them to boost their productivity. So it’s no brainer to look for an analytic storage database.

 

Choosing the right analytic storage database is not an easy task, as you have to consider many factors such as type of workload, queries, storage, compression.. Etc. If you are looking for one, please refer to the previous blog in this series. Next thing would be streaming the data to the analytic database, ETL jobs are the foremost thing that comes in everyone’s mind. Although it is a proven one, it takes a lot of time to write queries and unfortunately if you are on a tight pipeline you don’t have much time to write these jobs.

 

Fortunately we have few options with maxscale, proxysql , tungsten which eliminates the need for ETL.

 

In this blog, we can see how we have achieved real time data streaming from transactional to analytics database.

 

Background of the setup

  • MariaDB as Transactional DB
  • MariaDB columnstore as Analytics DB
  • MariaDB Maxscale as the binlog reader

Requirements on MariaDB

  • Need to enable binary logging
  • Need to update binary log format as ROW
  • Need to create a user to stream the data from MariaDB

Requirements on Maxscale

  • Configure replication_router and replication_listener in the maxscale config
  • Replication user needs to be used in the maxscale config to communicate with the MariaDB.

Architecture

MariaDB ColumnStore - Architecture
 

Steps

  • Maxscale will read the events from the binary log and it uses avro / json format to load the data into the columnstore using the CDC adaptor.
  • CDS adaptor is a utility which will act as an intermediate between Maxscale service and analytics DB.
  • Once it is ready, we can call the mxs_adapter utility, it will start the data streaming to columnstore thus eliminating the need for an ETL job.

Conclusion

ETL is time consuming which is tough to maintain, needs to be updated regularly, whereas this service is easy to deploy and maintain. With this recent trend on analytics, data streaming from transactional database becomes much simpler and hassle free.

FAQ

MariaDB data can be streamed to ColumnStore by reading changes from the MariaDB binary log and sending those change events to the analytics database. In the architecture described in this article, MariaDB MaxScale reads the binlog, while the CDC adapter processes the events and loads them into MariaDB ColumnStore.
MaxScale acts as the binlog reader and streaming component. It reads row-based binary-log events from the MariaDB transactional database and makes those changes available to the downstream streaming process. In the described setup, MaxScale uses the replication configuration to connect to MariaDB and stream the changes toward ColumnStore through the CDC adapter.
It can eliminate the need for a traditional scheduled ETL pipeline for this type of data movement. Instead of periodically extracting data from MariaDB, transforming it, and loading it into ColumnStore, changes can be captured from the binary log and streamed continuously. This can reduce data latency and the effort required to maintain custom ETL jobs. However, transformations, data cleansing, enrichment, and complex business logic may still require a dedicated data pipeline depending on the use case.

Author Bio

sukan

Sukan is Database Team Lead at Mafiree with over a decade of experience in database systems, architecture, and performance optimization. He specializes in MySQL, MongoDB, TiDB, and ClickHouse, developing architectural improvements that make data platforms faster, more efficient, and cost-effective. Sukan writes about practical database engineering topics, real-world performance tuning, data replication, and high-scale system design, drawing from extensive hands-on experience solving complex technical challenges.

Leave a Comment

Comments

PA
Praveen Adepu August 02, 2020

Nice article. We are trying to implement the real time data streaming from MariaDB 10.5 to ColumnStore 1.5 using MaxScale 2.5. Unfortunately, the CDC adaptor is deprecated in MaxScale 2.5. We tried HTAP, but it is very slow. Can I please know what is the alternative for the CDC adapter in MaxScale 2.5?

Related Blogs

MySQL Incident Response: When Does Slow Response Signal a Support Model Gap?

Slow MySQL incident resolution is often a symptom of a weak support model—not just a technical problem. Learn how to identify gaps in escalation, monitoring, automation, DBA expertise, and post-incident reviews before they lead to recurring downtime.

  71 views
MySQL Deadlock Analysis: Diagnosing and Resolving Lock Contention in High-Concurrency Workloads

A MySQL deadlock occurs when two or more transactions hold locks that each other needs, creating a circular dependency that prevents them from proceeding. InnoDB detects the deadlock and rolls back one transaction; applications typically receive MySQL error 1213 and should safely retry the transaction. To diagnose a deadlock, start with SHOW ENGINE INNODB STATUS; for recurring incidents, use innodb_print_all_deadlocks and Performance Schema data_locks and data_lock_waits to investigate lock relationships. Common causes include inconsistent transaction ordering, long-running transactions, and inefficient indexing or locking patterns. The most effective fixes are to keep transactions short, access shared resources in a consistent order, optimize the indexes used by locking statements, and implement safe retry handling.

  441 views
MySQL Consulting vs In-House DBA: Which Is Right for Your Team?

Choose an in-house DBA when you need continuous database ownership, deep knowledge of internal systems, and day-to-day operational control. Choose MySQL consulting when you need specialized expertise, performance tuning, migrations, complex troubleshooting, or additional database capacity without hiring a full-time specialist. A hybrid model can be the best fit when your team handles routine database operations internally but needs external expertise for complex projects, peak workloads, or specialized MySQL problems. The right choice depends on workload consistency, technical complexity, required coverage, internal expertise, scalability, and total cost—not simply the hourly or salary cost. Mafiree's managed database services provide ongoing MySQL support alongside specialized database expertise for organizations that need more flexibility than a purely in-house model.

  292 views
MySQL Consultant: When to Call in a Performance Expert

Most MySQL slowdowns are fixable in-house. A smaller set of problems keep coming back, keep growing, or sit outside what routine tuning can solve — and that's the line worth knowing before an incident forces the decision for you.

  364 views
MySQL Performance Issues: 7 Signs You Need Professional Tuning

MySQL performance issues are rarely sudden — they build over time through slow queries, InnoDB buffer pool misses, replication lag, lock contention, thread pile-ups, tablespace bloat, and unstable query plans. This post identifies the seven most reliable signs that your MySQL environment needs professional DBA attention, with diagnostic queries and remediation guidance for each.

  13245 views
Column-Level Security: Enterprise Data Protection Without the Infrastructure Overhead

Column-level security is a native database feature that restricts access to specific table columns by user role. Mafiree implemented this for a client as a cost-effective replacement for a planned CDC replication architecture that existed solely to strip sensitive columns. The result: zero additional infrastructure, single source of truth, full GDPR/HIPAA compliance posture, and validated in production with no performance impact.

  1256 views
MySQL Schema Migration Without Downtime: A Real Fintech Case Study

Schema changes on large MySQL tables can bring production systems to a halt if not handled correctly. This case study walks through how Mafiree helped a fintech client execute a zero-downtime MySQL schema migration on a 500M+ row production database — covering the real challenges faced, the three-phase tool strategy using gh-ost, pt-online-schema-change, and MySQL 8.0 INSTANT DDL, production configuration settings with performance benchmarks, and best practices for safely evolving your MySQL schema without impacting users

  4382 views
MySQL Architecture Explained: Performance Tuning & Troubleshooting Guide

MySQL features a unique tiered architecture that separates query processing from data storage through its pluggable storage engine model. This guide explores the core components—from connection handling and the SQL optimizer to the physical storage of data on disk. By understanding how engines like InnoDB provide ACID compliance and row-level locking, you can significantly improve your database's scalability. We also break down the query execution workflow and provide actionable tips for performance tuning, such as optimizing the buffer pool. Whether you're managing a replica set or a standalone instance, mastering MySQL’s internal structure is essential for building high-performance applications.

  2328 views
MariaDB vs MySQL: What's Different in 2026 and Which One Should You Use

Discover how MariaDB 11.x is redefining open-source databases with cutting-edge features like system-versioned tables, native AI-ready vector support, UUIDv7 for scalable inserts, and enterprise-grade security; all in the Community Edition, without the paywall.

  1759 views
MySQL’s Connection Control Plugin for Brute-Force Protection: A DBA Security Guide

“Fortify Your MySQL Security: Slow Down Attackers with Connection Control Plugin” Learn how the MySQL Connection Control Plugin helps defend against brute-force login attempts by introducing intelligent, progressive delays—without locking out legitimate users.

  13228 views
MySQL 8.4 Migration: Key Changes for IT Leaders

Explore the 8 critical changes in MySQL 8.4 you need to know before migrating your production environment. From authentication updates to InnoDB defaults, this release brings significant operational implications for developers and DBAs.

  348 views
Load Balancing in MySQL: Read and Write Traffic Optimization Using MySQL Router

Optimize MySQL traffic with MySQL Router — smart load balancing made easyRouter Architecture image widget

  2000 views
Check Constraints in MySQL

Master Database Integrity with Check Constraints: Improve Data Quality and System Performance

  13221 views
Guarding Your Account: 5 Essential Tips to Evade Common Password Mistakes.

How to Secure your mysql database with the mysql validate password component

  3584 views
Access Control & Account Management In MySQL 8

Discover how MySQL 8 enhances security through features like password expiry, password validation, account lock/unlock, privilege management, and dual passwords. Safeguard your valuable data and streamline access control in your MySQL environment.

  3910 views
Significance of using Invisible Primary key (GIPK) with MySQL 8.0

How Generated Invisible Primary Keys (GIPK) Can Boost Your Database Efficiency

  3746 views
Optimize Your Database: Exploring MySQL 8.0 JSON Functions

This blog highlights the some of the important JSON function in MySQL 8.0 with examples. Level up your SQL skills with MySQL 8.0's JSON functions.

  5183 views
Best practices on enabling Audit log on MySQL, MariaDB and ProxySQL

Importance of audit log in MySQL, MariaDB and ProxySQL. Best practices to enable audit log

  5228 views
High Availability for MySQL using Orchestrator and ProxySQL

Auto slave promotion without any DBAs intervention using Orchestrator and ProxySQL. If you are looking for a HA solution without going for synchronous replication or Aws RDS then Orchestrator with ProxySQL is a great choice.

  3535 views
Understanding the Spider Storage Engine

MariaDB Spider is a storage engine that supports federation and sharding across multiple MySQL or MariaDB servers. It helps distribute data and workloads, making it useful for scaling large datasets, balancing traffic, and improving data retrieval through parallel processing.

  1783 views
Avoid deadlocks, certification failures on Galera Cluster using ProxySQL

Dealing with deadlocks, certification failures on Galera cluster by running single node writer using ProxySQL

  7141 views
MySQL Character Set: UTF8MB4 vs UTF8 and How to Choose the Right Encoding

Choosing the right MySQL character set is critical for ensuring accurate data storage, multilingual support, and proper text comparison. While the legacy utf8 (utf8mb3) character set supports only up to 3-byte characters, it fails to handle modern Unicode requirements such as emojis and certain international symbols. In contrast, utf8mb4 provides full 4-byte Unicode support, making it the recommended standard for modern applications. This guide explains the differences between UTF8MB4 and UTF8, their impact on storage and performance, and provides step-by-step instructions to configure and migrate your MySQL databases safely.

  5880 views
Fixing the MariaDB Galera Bug: WSREP Lock Waits

A MariaDB Galera cluster can experience severe stalls when concurrent updates to the same primary key trigger an unexpected deadlock issue. This blog explains how we reproduced the issue in MariaDB 10.2, why the affected node became unresponsive, and the recovery and upgrade options used to resolve it.

  4606 views

Subscribe for email updates

Get in touch with us

Highlights

More than 6000 Servers Monitored

Happy Clients

Certified DBAs

24 x 7 x 365 Support

PCI

Database Services

MySQL MongoDB PostgreSQL SQL Server Aerospike Clickhouse TiDB ScyllaDB

Quick Links

Careers Blog Contact Privacy Policy Disclaimer Policy

Contacts

Linkedin Mafiree Facebook Mafiree Twitter Mafiree

Nagercoil Office

Miru IT Park, Vallankumaranvillai,

Nagercoil, Tamilnadu - 629 002.

Bangalore Office

Unit 303, Vanguard Rise,

5th Main, Konena Agrahara,

Old Airport Road, Bangalore - 560 017.

Call: +91 6383016411

Email: sales@mafiree.com


Copyright © - All Rights Reserved - Mafiree