Mafiree logo
  • About
  • Services
  • Blogs
  • Careers
  • Products
    • orbit logo Orbit
    • streamer logo Xstreami
  • Contact
Schedule a Call
Menu
  • About
  • Services
  • Blogs
  • Careers
  • Products
    • orbit logo Orbit
    • streamer logo Xstreami
  • Contact
  • Schedule a Call
Database
Database Managed Database Services
MySQL MySQL
MySQL Consulting
MySQL Migration Services
MySQL Optimization & Query Tuning
MySQL Database Administration
MySQL Backup & Recovery
MySQL Security & Maintenance
MySQL Cloud Services (AWS RDS, Aurora, Google Cloud SQL, Azure)
MySQL for Ecommerce
MySQL High Availability & Replication
MongoDB MongoDB
MongoDB Consulting
MongoDB Migration Services
MongoDB Optimization & Query Tuning
MongoDB Database Administration
MongoDB Backup & Recovery
MongoDB Security & Maintenance
MongoDB Cloud (Atlas)
MongoDB Solutions by Industry
MongoDB High Availability & Replication
PostgreSQL PostgreSQL
PostgreSQL Consulting
PostgreSQL Migration & Upgrades
Performance Tuning & Query Optimization
PostgreSQL Administration & Managed Services
High Availability, Clustering & Replication
PostgreSQL Backup, Recovery & Disaster Planning
PostgreSQL Security, Compliance & Auditing
PostgreSQL for Analytics & Data Warehousing
PostgreSQL on Cloud & Containers
PostgreSQL Extensions & Open-Source Integrations
PostgreSQL for Every Industry
SQL Server MSSQL
MSSQL Consulting
MSSQL Migration Services
MSSQL Optimization & Query Tuning Services
MSSQL Database Administration Services
MSSQL Backup & Recovery Services
MSSQL High Availability & Replication Services
MSSQL Security & Compliance Services
MSSQL Performance Monitoring & Health Checks
MSSQL Solutions by Industry
Aerospike Aerospike
Aerospike Consulting
Aerospike Migration Services
Aerospike Performance Optimization & Tuning
Aerospike Backup & Recovery
Aerospike High Availability
Aerospike Cloud & Hybrid Deployments
Aerospike for Real-Time Applications (AdTech, FinTech, Retail, IoT)
Clickhouse Clickhouse
ClickHouse Consulting
ClickHouse Migration Services
ClickHouse Optimization & Query Tuning
ClickHouse Database Administration
ClickHouse Backup & Recovery
ClickHouse Security & Maintenance
ClickHouse Cloud Services (ClickHouse Cloud, AWS, GCP, Azure)
ClickHouse Solutions by Industry
ClickHouse High Availability & Replication
TiDB TiDB
TiDB Consulting
TiDB Administration & Maintenance
TiDB Security and Privacy Maintenance
TiDB Performance & Query Optimization
TiDB Migration Services
TiDB Backup & Disaster Recovery
TiDB High Availability Solutions
TiDB Solutions by Industry
TiDB Cloud Services
ScyllaDB ScyllaDB
ScyllaDB Consulting
ScyllaDB Administration & Maintenance
ScyllaDB Security and Privacy Maintenance
ScyllaDB Performance & Query Optimization
ScyllaDB Migration Services
ScyllaDB Backup & Disaster Recovery
ScyllaDB High Availability Solutions
ScyllaDB Solutions by Industry
ScyllaDB Cloud Services
DevOps
DevOps DevOps Services
Version Control Version Control
Kubernetes Kubernetes
Infrastructure Infrastructure Management
Web Servers Web Servers
Networking
Networking Networking Services
Basic Basic
Advanced Advanced
MySQL MySQL
MongoDB MongoDB
PostgreSQL PostgreSQL
MSSQL MSSQL
Aerospike Aerospike
Clickhouse Clickhouse
TiDB TiDB
ScyllaDB ScyllaDB
Version Control Version Control
Kubernetes Kubernetes
Infrastructure Infrastructure Management
Web Servers Web Servers
Basic Basic
Advanced Advanced
MySQL Consulting
MySQL Migration Services
MySQL Optimization & Query Tuning
MySQL Database Administration
MySQL Backup & Recovery
MySQL Security & Maintenance
MySQL Cloud Services (AWS RDS, Aurora, Google Cloud SQL, Azure)
MySQL for Ecommerce
MySQL High Availability & Replication
MongoDB Consulting
MongoDB Migration Services
MongoDB Optimization & Query Tuning
MongoDB Database Administration
MongoDB Backup & Recovery
MongoDB Security & Maintenance
MongoDB Cloud (Atlas)
MongoDB Solutions by Industry
MongoDB High Availability & Replication
PostgreSQL Consulting
PostgreSQL Migration & Upgrades
Performance Tuning & Query Optimization
PostgreSQL Administration & Managed Services
High Availability, Clustering & Replication
PostgreSQL Backup, Recovery & Disaster Planning
PostgreSQL Security, Compliance & Auditing
PostgreSQL for Analytics & Data Warehousing
PostgreSQL on Cloud & Containers
PostgreSQL Extensions & Open-Source Integrations
PostgreSQL for Every Industry
MSSQL Consulting
MSSQL Migration Services
MSSQL Optimization & Query Tuning Services
MSSQL Database Administration Services
MSSQL Backup & Recovery Services
MSSQL High Availability & Replication Services
MSSQL Security & Compliance Services
MSSQL Performance Monitoring & Health Checks
MSSQL Solutions by Industry
Aerospike Consulting
Aerospike Migration Services
Aerospike Performance Optimization & Tuning
Aerospike Backup & Recovery
Aerospike High Availability
Aerospike Cloud & Hybrid Deployments
Aerospike for Real-Time Applications (AdTech, FinTech, Retail, IoT)
ClickHouse Consulting
ClickHouse Migration Services
ClickHouse Optimization & Query Tuning
ClickHouse Database Administration
ClickHouse Backup & Recovery
ClickHouse Security & Maintenance
ClickHouse Cloud Services (ClickHouse Cloud, AWS, GCP, Azure)
ClickHouse Solutions by Industry
ClickHouse High Availability & Replication
TiDB Consulting
TiDB Administration & Maintenance
TiDB Security and Privacy Maintenance
TiDB Performance & Query Optimization
TiDB Migration Services
TiDB Backup & Disaster Recovery
TiDB High Availability Solutions
TiDB Solutions by Industry
TiDB Cloud Services
ScyllaDB Consulting
ScyllaDB Administration & Maintenance
ScyllaDB Security and Privacy Maintenance
ScyllaDB Performance & Query Optimization
ScyllaDB Migration Services
ScyllaDB Backup & Disaster Recovery
ScyllaDB High Availability Solutions
ScyllaDB Solutions by Industry
ScyllaDB Cloud Services
  1. Home
  2. > Blogs
  3. > MySQL
  4. > Fixing the MariaDB Galera Bug: WSREP Lock Waits

Fixing the MariaDB Galera Bug: WSREP Lock Waits

A MariaDB Galera cluster can experience severe stalls when concurrent updates to the same primary key trigger an unexpected deadlock issue. This blog explains how we reproduced the issue in MariaDB 10.2, why the affected node became unresponsive, and the recovery and upgrade options used to resolve it.

sukan September 10, 2026

Subscribe for email updates

Summarize with AI: ChatGPT Google AI Perplexity Claude Grok

The MariaDB Galera Cluster Issue

Our setup has a three-node MariaDB Galera cluster running with 10.2.5 version, our cluster is serving reads and writes on all the three nodes. All our nodes have same hardware specification and application hits are split equally on these servers using ProxySQL. While everything is working fine, suddenly all the queries hang on random nodes and eventually the cluster stalls. Even though we have all deadlock detection mechanisms and other timeout settings, still cluster hangs and it brings the entire application down.

We have observed this behavior when we upgraded from MariaDB 10.1 to 10.2 and this bug is highly unpredictable. Even in low peak time with very minimal hits, we have faced this issue. The only option we had at the moment is to trigger an alert when we hit the error and schedule a shell script to force restart the affected node

After detailed study of the environment, queries, hits etc we were able to reproduce this bug.

 

How Is the Bug Triggered?

Consider the below table used to store youtubeviews status for different services.

 

Table Structure

  1. CREATE TABLE `youtube_views` (
  2.   `id` int NOT NULL AUTO_INCREMENT,
  3.   `status` int NOT NULL,
  4.   `token_id` varchar(50) DEFAULT NULL,
  5.   `session_id` varchar(50) DEFAULT NULL,
  6.   `response` int DEFAULT NULL,
  7.   PRIMARY KEY (`id`)
  8. ) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Table Data

Table Data

 
From the application perspective, status column is being changed every time. Thus having the possibility to fire multiple updates for the same ID column [Primary key]

In this situation, whenever there is an update for the same primary key on a transaction, it should throw a deadlock exception. But GC accepted the writes and triggered a situation which caused this bug.

The node would no longer accept writes even for another ID and it continues to be in hang state, here is a snippet of running processes inside the cluster.

 

  1. +---------+-------------+-----------+------+---------+-------+---------------------------------------+------------------------------------------------+----------+
  2.  
  3. | Id      | User        | Host | db   | Command | Time | State                      | Info  | Progress |
  4.  
  5. +---------+-------------+-----------+------+---------+-------+---------------------------------------+------------------------------------------------+----------+
  6.  
  7. |       1 | system user |           | NULL | Sleep | 1495 | Update_rows_log_event::find_row(2630) | update youtubeviews set status=100 where id=5  |   0.000 |
  8.  
  9. |       2 | system user |           | NULL | Sleep | 73330 | wsrep aborter idle                    | NULL        | 0.000 |
  10.  
  11. | 1052481 | root        | localhost | prod | Query   | 1496 | query end                             | update youtubeviews set status=100 where id=5  |   0.000 |
  12.  
  13. | 1052973 | root        | localhost | prod | Killed  | 924 | query end                             | update youtubeviews set status=101 where id=11 |    0.000 |
  14.  
  15. | 1052974 | root        | localhost | NULL | Sleep   | 190 |                                       | NULL        | 0.000 |
  16.  
  17. | 1052980 | root        | localhost | prod | Query   | 0 | init                                  | show processlist        | 0.000 |
  18.  
  19. +---------+-------------+-----------+------+---------+-------+---------------------------------------+------------------------------------------------+----------+

You could see below error on the MariaDB error logs.

 

How to bring the cluster up?

The first response from the DBA is to manually kill the query which has stalled the cluster. Unfortunately this bug is so worse that killing the query doesn’t bring up the cluster; in fact we will not able to restart of node as well.

 

One has to manually kill the affected MariaDB Cluster node and force restarting would help bringing back the cluster online.

 

Fix

Running Galera Cluster in a single node

 

We have put the Galera cluster writes in to a single node and keep other nodes for reads. Having said that it would avoid hitting this error but again it is more or less similar to normal Master / Slave MariaDB.

 

Upgrade the cluster to 10.2.13+ version

 

As rightly pointed out by MariaDB in their release notes, upgrading cluster to 10.2.13 version is highly recommended as they have provided the fix for this bug. [MDEV-12837]

 

The same test was applied to this newer version, understandingly when we fire updates for the same ID in different nodes below error is thrown and transaction gets rollbacked.

 

  1. Cannot run query update youtubeviews set status=100 where id=5; 
  2. ERROR : Deadlock: wsrep aborted transaction

NOTE: Since applications are connected to Galera cluster via ProxySQL, a rolling restart is all need for upgrading the cluster.

 

Downgrading the cluster to 10.1.x version

As such cases, where upgradation not possible, downgrading the cluster to 10.1 can avoid from this bug. As in this 10.1 version, it immediately thrown an deadlock error once it hit this situation.

 

Conclusion

This issue showed how an unpredictable Galera bug can bring down an entire application even when timeout and deadlock protections are already in place. After reproducing the problem, we found that upgrading to MariaDB 10.2.13+ resolved the issue, while routing writes through a single node provided an alternative workaround.

 

At Mafiree, we help businesses troubleshoot and optimize high-availability database environments, including MariaDB Galera clusters. From identifying replication issues to resolving unexpected cluster failures, our focus is on keeping production databases stable, reliable, and available.

 

FAQ

The WSREP BF lock wait issue is a Galera-related problem in which conflicting transactions can leave a MariaDB Galera node waiting for an extended period instead of being aborted normally. In the incident described in this article, concurrent updates to the same primary key from different Galera nodes caused the cluster to enter a stalled state.
WSREP stands for Write Set Replication. It is the replication interface used by Galera-based MariaDB clusters to replicate transactions between nodes. WSREP coordinates transaction certification, replication, conflict handling, and cluster state between Galera nodes.
A common trigger is concurrent modification of the same data from multiple Galera nodes. For example, if multiple application connections send updates for the same primary-key row to different nodes at approximately the same time, Galera must resolve the resulting write conflict. The article demonstrates this behavior using repeated updates to the same id in the youtube_views table.
Typical symptoms can include: Queries remaining in Query end or other wait states for unusually long periods. Updates to unrelated rows also becoming blocked. Increasing numbers of stalled application connections. wsrep-related processes appearing stuck. Galera nodes becoming unresponsive to writes. Application traffic eventually failing because database connections cannot complete. In the documented incident, even updates against another ID could no longer proceed once the affected node entered the problematic state.
Yes. In the specific historical MariaDB version described in the article, the affected node could enter a state where normal queries stopped completing. The article reports that killing the apparently problematic query did not recover the node and that the affected node had to be forcibly restarted.

Author Bio

sukan

Sukan is Database Team Lead at Mafiree with over a decade of experience in database systems, architecture, and performance optimization. He specializes in MySQL, MongoDB, TiDB, and ClickHouse, developing architectural improvements that make data platforms faster, more efficient, and cost-effective. Sukan writes about practical database engineering topics, real-world performance tuning, data replication, and high-scale system design, drawing from extensive hands-on experience solving complex technical challenges.

Leave a Comment

Comments

P
Ponarasu July 25, 2020

good article, nice information, thanks for your knowlwdge sharing.

Related Blogs

MySQL Deadlock Analysis: Diagnosing and Resolving Lock Contention in High-Concurrency Workloads

A MySQL deadlock occurs when two or more transactions hold locks that each other needs, creating a circular dependency that prevents them from proceeding. InnoDB detects the deadlock and rolls back one transaction; applications typically receive MySQL error 1213 and should safely retry the transaction. To diagnose a deadlock, start with SHOW ENGINE INNODB STATUS; for recurring incidents, use innodb_print_all_deadlocks and Performance Schema data_locks and data_lock_waits to investigate lock relationships. Common causes include inconsistent transaction ordering, long-running transactions, and inefficient indexing or locking patterns. The most effective fixes are to keep transactions short, access shared resources in a consistent order, optimize the indexes used by locking statements, and implement safe retry handling.

  334 views
MySQL Consulting vs In-House DBA: Which Is Right for Your Team?

Choose an in-house DBA when you need continuous database ownership, deep knowledge of internal systems, and day-to-day operational control. Choose MySQL consulting when you need specialized expertise, performance tuning, migrations, complex troubleshooting, or additional database capacity without hiring a full-time specialist. A hybrid model can be the best fit when your team handles routine database operations internally but needs external expertise for complex projects, peak workloads, or specialized MySQL problems. The right choice depends on workload consistency, technical complexity, required coverage, internal expertise, scalability, and total cost—not simply the hourly or salary cost. Mafiree's managed database services provide ongoing MySQL support alongside specialized database expertise for organizations that need more flexibility than a purely in-house model.

  246 views
MySQL Consultant: When to Call in a Performance Expert

Most MySQL slowdowns are fixable in-house. A smaller set of problems keep coming back, keep growing, or sit outside what routine tuning can solve — and that's the line worth knowing before an incident forces the decision for you.

  329 views
MySQL Performance Issues: 7 Signs You Need Professional Tuning

MySQL performance issues are rarely sudden — they build over time through slow queries, InnoDB buffer pool misses, replication lag, lock contention, thread pile-ups, tablespace bloat, and unstable query plans. This post identifies the seven most reliable signs that your MySQL environment needs professional DBA attention, with diagnostic queries and remediation guidance for each.

  13186 views
Column-Level Security: Enterprise Data Protection Without the Infrastructure Overhead

Column-level security is a native database feature that restricts access to specific table columns by user role. Mafiree implemented this for a client as a cost-effective replacement for a planned CDC replication architecture that existed solely to strip sensitive columns. The result: zero additional infrastructure, single source of truth, full GDPR/HIPAA compliance posture, and validated in production with no performance impact.

  1196 views
MySQL Schema Migration Without Downtime: A Real Fintech Case Study

Schema changes on large MySQL tables can bring production systems to a halt if not handled correctly. This case study walks through how Mafiree helped a fintech client execute a zero-downtime MySQL schema migration on a 500M+ row production database — covering the real challenges faced, the three-phase tool strategy using gh-ost, pt-online-schema-change, and MySQL 8.0 INSTANT DDL, production configuration settings with performance benchmarks, and best practices for safely evolving your MySQL schema without impacting users

  4304 views
MySQL Architecture Explained: Performance Tuning & Troubleshooting Guide

MySQL features a unique tiered architecture that separates query processing from data storage through its pluggable storage engine model. This guide explores the core components—from connection handling and the SQL optimizer to the physical storage of data on disk. By understanding how engines like InnoDB provide ACID compliance and row-level locking, you can significantly improve your database's scalability. We also break down the query execution workflow and provide actionable tips for performance tuning, such as optimizing the buffer pool. Whether you're managing a replica set or a standalone instance, mastering MySQL’s internal structure is essential for building high-performance applications.

  2273 views
MariaDB vs MySQL: What's Different in 2026 and Which One Should You Use

Discover how MariaDB 11.x is redefining open-source databases with cutting-edge features like system-versioned tables, native AI-ready vector support, UUIDv7 for scalable inserts, and enterprise-grade security; all in the Community Edition, without the paywall.

  1701 views
MySQL’s Connection Control Plugin for Brute-Force Protection: A DBA Security Guide

“Fortify Your MySQL Security: Slow Down Attackers with Connection Control Plugin” Learn how the MySQL Connection Control Plugin helps defend against brute-force login attempts by introducing intelligent, progressive delays—without locking out legitimate users.

  13164 views
MySQL 8.4 Migration: Key Changes for IT Leaders

Explore the 8 critical changes in MySQL 8.4 you need to know before migrating your production environment. From authentication updates to InnoDB defaults, this release brings significant operational implications for developers and DBAs.

  210 views
Load Balancing in MySQL: Read and Write Traffic Optimization Using MySQL Router

Optimize MySQL traffic with MySQL Router — smart load balancing made easyRouter Architecture image widget

  1921 views
Check Constraints in MySQL

Master Database Integrity with Check Constraints: Improve Data Quality and System Performance

  13172 views
Guarding Your Account: 5 Essential Tips to Evade Common Password Mistakes.

How to Secure your mysql database with the mysql validate password component

  3542 views
Access Control & Account Management In MySQL 8

Discover how MySQL 8 enhances security through features like password expiry, password validation, account lock/unlock, privilege management, and dual passwords. Safeguard your valuable data and streamline access control in your MySQL environment.

  3863 views
Significance of using Invisible Primary key (GIPK) with MySQL 8.0

How Generated Invisible Primary Keys (GIPK) Can Boost Your Database Efficiency

  3700 views
Optimize Your Database: Exploring MySQL 8.0 JSON Functions

This blog highlights the some of the important JSON function in MySQL 8.0 with examples. Level up your SQL skills with MySQL 8.0's JSON functions.

  5139 views
Best practices on enabling Audit log on MySQL, MariaDB and ProxySQL

Importance of audit log in MySQL, MariaDB and ProxySQL. Best practices to enable audit log

  5169 views
High Availability for MySQL using Orchestrator and ProxySQL

Auto slave promotion without any DBAs intervention using Orchestrator and ProxySQL. If you are looking for a HA solution without going for synchronous replication or Aws RDS then Orchestrator with ProxySQL is a great choice.

  3422 views
Understanding the Spider Storage Engine

MariaDB Spider is a storage engine that supports federation and sharding across multiple MySQL or MariaDB servers. It helps distribute data and workloads, making it useful for scaling large datasets, balancing traffic, and improving data retrieval through parallel processing.

  1734 views
Avoid deadlocks, certification failures on Galera Cluster using ProxySQL

Dealing with deadlocks, certification failures on Galera cluster by running single node writer using ProxySQL

  7065 views
MySQL Character Set: UTF8MB4 vs UTF8 and How to Choose the Right Encoding

Choosing the right MySQL character set is critical for ensuring accurate data storage, multilingual support, and proper text comparison. While the legacy utf8 (utf8mb3) character set supports only up to 3-byte characters, it fails to handle modern Unicode requirements such as emojis and certain international symbols. In contrast, utf8mb4 provides full 4-byte Unicode support, making it the recommended standard for modern applications. This guide explains the differences between UTF8MB4 and UTF8, their impact on storage and performance, and provides step-by-step instructions to configure and migrate your MySQL databases safely.

  5819 views
Real time data streaming from MariaDB to Columnstore

How real time data from transactional database(mariadb) to analytic database(columnstore) is achieved using MariaDB's maxscale binlog streaming

  1472 views

Subscribe for email updates

Get in touch with us

Highlights

More than 6000 Servers Monitored

Happy Clients

Certified DBAs

24 x 7 x 365 Support

PCI

Database Services

MySQL MongoDB PostgreSQL SQL Server Aerospike Clickhouse TiDB ScyllaDB

Quick Links

Careers Blog Contact Privacy Policy Disclaimer Policy

Contacts

Linkedin Mafiree Facebook Mafiree Twitter Mafiree

Nagercoil Office

Miru IT Park, Vallankumaranvillai,

Nagercoil, Tamilnadu - 629 002.

Bangalore Office

Unit 303, Vanguard Rise,

5th Main, Konena Agrahara,

Old Airport Road, Bangalore - 560 017.

Call: +91 6383016411

Email: sales@mafiree.com


Copyright © - All Rights Reserved - Mafiree