Xstreami: The ETL Solution

What is ETL?

ETL (Extract, Transform, Load) is the process of capturing events from a source based on Change Data Capture (CDC) and storing them in a designated destination, with or without applying data transformation logic. For example, CRUD (Create, Read, Update, Delete) events from a MySQL database can be processed, transformed, and stored in another database such as MySQL, MongoDB, TiDB, ClickHouse, or PostgreSQL.

ETL is an essential process for data analytics, business reporting, forecasting, and even training AI models using transformed data derived from CDC events.

What is Xstreami?

Xstreami is a product by Mafiree designed to handle ETL processes. Mafiree brings decades of expertise in database management across a wide range of database systems and provides 24/7 support for database monitoring.

Xstreami is engineered to support databases like MySQL, TiDB, ClickHouse, PostgreSQL, and MongoDB. It can be configured for various ETL flows, including single-source to single-destination, single-source to multiple destinations, multiple sources to a single destination, and multiple sources to multiple destinations. The key highlights of Xstreami include:

1. High performance
2. Easy deployment
3. Diverse database and log support
4. Flexible customization
5. Minimal server setup requirements
6. Cost-effective compared to other ETL or queuing tools

Architecture of Xstreami

Xstreami’s ETL process is divided into three phases: Listener, Applier, and Loader.

• The Listener phase monitors CDC events, filters data based on configurations, and sends the filtered data to the Applier.
• The Applier phase applies business logic or conditions to transform the data before forwarding it to the Loader.
• The Loader phase stores the transformed data in a database or logs it to files.

Each phase operates in parallel to maximize performance. Below is a flow diagram illustrating the process.

Xstreami Capabilities

     1. Multi-threading
        Xstreami reads and writes data in parallel. Written in Go, Xstreami leverages Go’s strong support for multi-threading, which makes high performance a key feature of its ETL processes.

     2. Data Filtering
        Xstreami allows configuration of data filtering to exclude unnecessary data during the ETL process, ensuring only relevant data is included.

     3. Event Logging
        Xstreami features event logging to keep a record of all incoming events, aiding in monitoring and diagnostics.

     4. Intelligent Business Rules
        Xstreami allows users to define business rules and conditions for data transformation, applying logic to data throughout the ETL process.

     5. Scalability
        Xstreami is easily scalable, allowing for new logic, events, and parallel ETL processes to be added as needed.

Where Can Xstreami Be Used?

Take logistics as an example—a complex industry with intricate connections between shipments, addresses, tracking, contact information, product details, payment, and insurance. Business insights and reporting become costly due to numerous database interconnections, and if the data source is non-relational, the challenge grows. Shipment tracking alone is a major operation when handling large sets of data for both domestic and international needs.

By flattening and organizing data for tracking and insights, Xstreami simplifies these processes. It transforms data in real-time, capturing CDC events, applying the required transformations, and storing the data in a structured format for efficient tracking and business insights.

Looking for a Solution?

Do you want to enhance your business insights or organize your data more effectively? Let’s discuss how Xstreami can meet your ETL needs!