1. Introduction
In hospitality which is a fast-paced environment, a unified customer data platform with comprehensive and accurate customer information is mandatory in order to deliver best customer experiences. A top tier Casino Chain recently encountered a big challenge when tried to build such a customer data platform. This article is to introduce how Tapdata helped to create an unified customer profile data model and synchorized data from various application system into a centralized data store, to be used by the new customer 360 application.
2. Problem Statement
Customer currently face a significant challenge as their customer information is dispersed across many databases systems like Sql server, Oracle and PostgreSql. This fragmentation complicates data management and accessibility, hindering efficient operations.
To address this challenge, the customer aims to consolidate all customer information into a single unified view, creating a comprehensive customer 360-degree perspective. This unified view is anticipated to streamline data management processes, providing a centralized repository for customer information. The primary objective is to enhance data accessibility and facilitate informed decision-making by enabling easier extraction of valuable insights from customer data.
The current architecture comprises siloed databases, resulting in fragmented customer data spread across numerous tables. This structure presents obstacles in terms of data accessibility, management, and analysis, impeding the organization's ability to gain holistic insights into customer behavior.
The implementation of a single customer view is intended to optimize data management practices, alleviating the complexities associated with dispersed customer information. By consolidating data into a unified view, the organization, particularly Top Tier Casino, can gain a cohesive understanding of each customer's interactions, preferences, and history. Ultimately, this initiative aims to enhance customer service delivery, facilitate data-driven decision-making, and improve overall operational efficiency.
3. Solution Provided by Tapdata
3.1 Proposed Solution:
To address the challenge of scattered customer information, Top Tier Casino is implementing a solution involving real-time synchronization between SQL Server and MongoDB. In this setup, SQL Server acts as the source database, and MongoDB serves as the target database. The key objective is to consolidate data into a unified structure within MongoDB, with all tables seamlessly synchronized in real-time.
3.2 Data Transformation and Consolidation:
As part of this solution, Customer plans to merge all existing tables from SQL Server into a single, comprehensive table within MongoDB. This consolidation aims to create a wide-ranging dataset that encompasses various aspects of customer information. To achieve a unified customer 360 view, complex data transformations will be applied during the synchronization process. These transformations are designed to harmonize disparate data formats and structures, providing end users with a cohesive and enriched understanding of customer interactions.
3.3 Tapdata Deployment Architecture Diagram
|
Host |
Service |
|---|---|
|
tapdata-01 tapdata-02 |
Tapdata Management Tapdata Flow Engine Tapdata API Server |
|
mongodb-01 mongodb-02 mongodb-03 |
MongoDB ReplicaSet |
3.3.1 Explanation of Tapdata On-Prem Single-Point Architecture：
Tapdata Management：User management, Task Management, UI etc.
Tapdata API Server：Responsible for API data publishing.
Tapdata Flow Engine：Responsible for data synchronization & processing.
MongoDB：Metadata, CDC Event Cache, Customer Data (when used as Data Service Platform)
4. What is Data Replication and Transformation
Real Time data replication involves the continuous copying and synchronization of data from a source to a target, ensuring consistency between multiple databases or systems in real-time or at scheduled intervals. This process is vital for maintaining redundancy, supporting backup strategies, and enabling efficient data access across distributed environments. Concurrently, data transformation is the conversion of data from one format to another, encompassing cleaning, enrichment, or restructuring to meet specific requirements. In the realm of database management, these transformations are crucial when migrating data between systems with different structures. By harmonizing data types, formats, and structures, data transformation ensures compatibility, facilitates integration, and plays a key role in deriving meaningful insights from consolidated information. Together, data replication and transformation form a powerful duo, supporting data integrity, accessibility, and informed decision-making within organizational frameworks.
Tapdata Features
-
Support 100+ built-in data connectors, offering extensive connectivity options.
-
Provides end-to-end real-time data integration capabilities, enhancing operational efficiency.
-
Flexible and scalable with a pay-per-instance pricing model, making it cost-effective for casinos of all sizes.
-
Lightweight and user-friendly with a low learning curve, featuring a visual interface for ease of use.
-
Offers real-time data processing and UDF (User Defined Function) customization for efficient data processing.
-
Supports sub-second data synchronization latency, enabling near real-time analytics and decision-making.
-
Provides features such as data API publishing and integrated storage for seamless data access and management.
-
Synchronization of multiple tables into one single collection in Mongo by merging homogeneous and heterogeneous database tables.
-
MongoDB Relational Migrator integration for relational to document model mapping and continuous data replication.
-
Automatic adaptation of DDL and data type compatibility among common databases (including but not limited to MySQL, Oracle, SQL Server, PostgreSQL, etc.).
6. Steps performed by Tapdata Platform to build customer 360 Table
Objective:
The overall goal is build centerlized data platform using the Tapdata platform. This gives a unified and improved view of customer information, helping better decisions and customer service. In short, the objective is to optimize access to data, streamline replication processes, and provide a holistic Customer 360 view through efficient data management with Tapdata.
Establish Connection:
-
Establish secure and reliable SQL Server and MongoDB connections with Tapdata.
Real-time Data Replication:
-
Configure Data Pipeline to enable real-time data replication of 25 tables from SQL Server to MongoDB.
-
Monitor replication processes to ensure data consistency and integrity.
Data Structure Simplification:
-
Utilize Tapdata's processing nodes, specifically the Master-Slave-Merge Node, to consolidate all tables into a unified structure within MongoDB.
Real-time Data Transformation:
-
Implement a real-time data transformation pipeline to construct a Customer 360 table within MongoDB, ensuring continuous updates and integration of relevant customer information
Top tier Casino Sql Server Database ER Diagram
Target Data Model
Establishing Database Connections
-
Establish connections to both SQL Server and MongoDB through the Tapdata platform, ensuring robust and reliable links between the source and target databases.
-
Connect to Sql Server With tapdata
-
Connect to MongoDB with tapdata
Data Replication Pipeline (Replicate data from SQL server to FDM(Foundation data Model) Layer powered by mongodb)
Merging Tables
-
Create a real-time data pipeline using Tapdata's built-in processing nodes, such as the Master-Slave merge node, to merge all 25 tables within MongoDB into a comprehensive wide table. This consolidation enhances data accessibility and simplifies data structure.
Complex Data Transformations for customer 360 Table:
-
Create a real-time data transformation pipeline to perform intricate data transformations on the wide table within MongoDB. The objective is to create a Customer 360 table, providing a comprehensive and unified view of customer data. This involves harmonizing disparate data formats and structures to deliver valuable insights for improved customer service and decision-making.
Customer 360 Table after data Transformation
6. Conclusion
Tapdata's transformative solution not only addressed the dispersed data challenges faced by top tier casino chain but also paved the way for a new era of operational efficiencies. Customer information gathered from multiple tables and continuous data replication and transformation highlighted Tapdata's expertise in providing better data management.
By seamlessly connecting SQL Server and MongoDB, establishing a dynamic data pipeline for real-time replication, and utilizing advanced features for merging and transforming tables into wide unified table. The intricate data transformations performed on the wide table yielded a comprehensive Customer 360 view, providing end-users with valuable insights.
