Transforming Shipbuilding with Real-Time Data: A Case Study on Integrating TapData and Real-Time Data Warehousing

Faced with the challenges of intelligent manufacturing and the complexities of flexible production, traditional manufacturers are actively seeking new digital models to stay ahead.

Beyond merely adopting advanced technologies like the Internet of Things (IoT) and Artificial Intelligence (AI) for efficient and intelligent production processes, manufacturers are also recognizing the crucial role of flexible production in adapting to ever-changing market demands and evolving environmental regulations.

In this context, data becomes increasingly valuable. Companies are connecting various production lines to ensure real-time monitoring and data collection throughout the production process. This enables them to gain accurate insights into production conditions, optimize their supply chains, and lay the groundwork for personalized production. Comprehensive data analysis provides deep insights into market trends and customer demands, empowering manufacturers to make intelligent decisions. These digital tools are vital for navigating the increasingly competitive global market and offer internal benefits such as optimizing personnel management, reducing energy consumption, and increasing production capacity.

This evolution is guiding the traditional manufacturing sector towards a smarter, more adaptable, and innovative future, promoting sustainable development within the industry. It also encourages enterprises to find a balance in their digital transformation efforts—seeking the optimal mix of technological advancement and cost management to ensure operational efficiency and maximize economic benefits during the digital transition.

As a prominent state-owned shipbuilding enterprise that had led the way in digital transformation, the company used for this case study faced challenges, a few years ago, in striking the balance between innovation and cost management.

Raising Anchor: The Critical Need for Data Extraction in the Digital Transformation Era

This prominent shipbuilding company stood out for its scalability, modernization, specialization, and influence within the industry. In recent years, it actively embraced cutting-edge technologies like Augmented Reality (AR), 5G, Artificial Intelligence (AI), the Internet of Things (IoT), and big data analytics. This rapid adoption extended to intelligent control systems, smart warehouses, 3D digital modeling platforms, and remote inspection studios. These advancements effectively integrated digital technologies into its production and operational management.

However, data – the lifeblood of any digital application – became increasingly critical. The company faced a growing demand for enterprise-wide data extraction and value mining. They recognized the power of data to fuel further digital transformation and business innovation.

As shipbuilding integrated new technologies with industry-specific practices, new data challenges inevitably emerged:

Challenges in Material Management

Extensive Number of Components: Shipbuilding requires millions of parts, leading to large intermediate products. Efficient management and effective digital tracking systems are crucial for a stable supply chain.
Material Coordination and Scheduling: Digital transformation must address the coordination and scheduling of widely distributed materials to enhance logistical efficiency and manage assembly complexity.

Process Optimization and Complexity Management

Complex Outfitting Processes: Outfitting involves installing various equipment and structures, a process made complex by diverse ship designs. Real-time monitoring, feedback, and virtual simulations are necessary to optimize efficiency and accuracy.
Engineering Plan Control: Given the lengthy and complex nature of shipbuilding projects, comprehensive management and real-time data collection are essential for timely adjustments and problem prevention.

Special Environmental and Climatic Conditions

Outdoor Operation Challenges: Shipbuilding often occurs in challenging environments, requiring real-time monitoring of wind speed, sea conditions, and temperature to ensure safety and anticipate potential dangers.

Human Resources and Skills Challenges

Personnel Scheduling and Management: Shipbuilding is labor-intensive, necessitating real-time monitoring for effective scheduling and resource allocation, optimizing efficiency, and reducing costs.
Managing Large Teams: For large teams, precise performance evaluations and refined welfare management depend on data-driven approaches.
Training and Skill Enhancement: Digital training and skill enhancement are crucial. New digital tools must be user-friendly to ensure smooth adoption, especially in a workforce not dominated by IT professionals.

To meet the real-time visualization needs of management and enhance international competitiveness, the company needed a solution that reliably gathered data from various sources in real time and provided it flexibly to different departments, ensuring the data fully realized its value.

Navigating: Establishing a Real-Time Data Platform by Implementing Real-Time Data Pipelines

Key Attributes of the Company's Data:

① High Demand for Real-Time Data Analysis: Applications like "smart brain" required real-time data analysis for warehouse logistics, equipment status, and personnel information to enable informed decision-making.

② Diverse Operational Systems: The company operated various systems including equipment management, IoT platforms, vehicle networks, manufacturing, dispatching, supply chain, employee attendance, and business intelligence (BI) systems. Managing diverse data sources and destinations posed significant challenges.

③ Variety of Data Types: Data spanned multiple databases such as Oracle, MySQL, TiDB, MongoDB, and TDengine. Synchronizing data between these different databases was a major task.

The increasing amount and variety of data required a two-fold approach to data management. First, a multi-source real-time data collection system was essential. Second, a unified data storage and management platform was needed to create a single source of truth, simplifying data access and analysis.

Initially, Kettle, an open-source tool, was used for data synchronization. However, its limitations in cost control and operational burden became apparent as business needs grew, highlighting the need for a more robust and scalable data management solution.

Challenges with Kettle:

Lack of Task Management System: Kettle lacked robust task scheduling and monitoring, relying on external tools, which made issue detection and management difficult.
Batch Processing Orientation: Primarily designed for offline data processing, Kettle failed to meet the increasing demand for real-time data processing.
Lack of Support for Distributed Deployment: Kettle did not support distributed deployment, making it difficult to handle the growing number of data transfer tasks.

The company's requirements for a new data solution were clear:

Comprehensive support for diverse data sources.
Real-time synchronization of heterogeneous database.
User-friendly visual task management interface.
Low learning curve and technical requirements, featuring a simple interface, lightweight design, and ease of operation.

Its introduction provided organizations with an enhanced experience in real-time data management.

TapData, a modern data platform with low-latency data replication, perfectly met these needs and data warehousing is one of its typical use cases. Its introduction provided the company with an enhanced experience in real-time data management.

Setting Sail: Integrating TapData, Real-Time Data Warehousing, and BI Systems to Make Data Useful, Usable, and Visible

After careful consideration, the shipbuilding company chose TapData for its real-time data synchronization capabilities. They streamlined data collection from various business systems to create a real-time data integration platform. This platform provided data to manufacturing, dispatch, supply chain, BI systems, and more, addressing needs such as report analysis, smart dashboards, attendance, health check-ins, project management, and intelligent equipment management.

TapData was selected for its ability to easily connect data sources and visualize data processes without requiring specialized programming skills. It allowed DBAs, architects, data engineers, and business users to efficiently manage necessary data.

A. Overall Plan for Real-Time Data Platform in Shipbuilding Manufacturing

By integrating real-time data capture, stream processing, and tools like TapData with a core real-time data warehouse, the company established a comprehensive real-time data platform. This platform met the needs of accessing and using data during digital transformation.

The architecture adopted by the shipbuilding company consists of the following layers:

Data Collection layer:
1. Tapdata: Serving as an integration tool, it connects to multiple data sources, including TiDB, MongoDB, Oracle, TDengine, etc. Additionally, TapData supports data connection, synchronization, transformation, cleansing, and distribution, thereby supporting the establishment of robust data workflows.
Data storage layer FDM:
1. 1. TapData: Serving as a real-time data capture and stream processing tool, TapData connects source database to TiDB, capturing real-time changes and facilitating stream data processing. With its user-friendly interface and robust functionality, TapData is well-suited for constructing real-time data processing workflows.
  2. TiDB Cluster: A 1:1 replication of data from the data source is transformed into structured data and stored within the unified data caching layer of the real-time platform, TiDB Cluster. This process entails establishing foundational data models encompassing multiple base tables, such as equipment tables, maintenance records, employee information, equipment inspection records, and storing production data, equipment statuses, and other pertinent information.
Data Processing Layer MDM:
1. TapData: Serving as a real-time data processing tool, TapData handles data synchronization, field processing, and field assignment from the FDM (Foundational Data Model) layer to the MDM (Master Data Model) layer.
2. TiDB Cluster: Functioning as the platform processing layer, TiDB merges a wide table from a master table and multiple slave tables in the MDM layer. Real-time tasks are employed to complete data cleaning, enhancement, enrichment, and standardization. Business data processing is performed here to ensure real-time updates. The MDM layer manages the enterprise's critical data, providing a unified source of reference. At this stage, data deduplication, standardization, and normalization are conducted to prevent erroneous data from entering the system. The resulting master data set serves as a single, trusted data pool for the enterprise's ongoing data needs.
Visualization and Reporting:
1. Fanruan BI: Connect to TiDB for building visual reports and dashboards.

Overall, TapData's solution made it easier to build real-time data processing processes, and its visual operation interface simplified the configuration process of stream processing and data integration. This kept data accurate and up to date while enhancing team productivity.

Owing to the exceptional benefits of the real-time data platform solution, the shipbuilding company achieved real cost reduction and efficiency improvement in the innovative practice of digital transformation data level. In addition to the optimization of labor and capital expenses, the improvement of data timeliness was also one of the key highlights of this plan.

In the shipbuilding industry, real-time data was crucial for ensuring the safety and efficiency of the production process.

One example of this encountered was to minimize safety impacts from forklift speeding. Shipbuilding sites frequently see large volumes of moving machinery, with forklifts employed to handle and transport critical materials. In this context, overspeeding forklifts pose significant risks of serious accidents and damage. By tracking forklift movement speeds in real-time, the system can promptly detect instances of excessive speeding and trigger alarms. TapData's real-time capabilities facilitate immediate notification to relevant personnel, allowing them to take swift corrective actions such as stopping the forklift, adjusting its speed, or warning the operator. This proactive approach significantly reduced the risk of potential accidents.

Furthermore, real-time data is essential for enhancing and refining the production process's efficiency. The traditional model of delivering periodic information reports to decision-makers was increasingly insufficient for the agile management operations required internally. And there was a pressing need for real-time access to current trends. By acquiring critical data promptly, the management team could monitor every aspect of ship construction in real-time and take swift management actions. This enabled quick decision-making, optimization of the production process, and overall efficiency improvements. Ensuring timely project delivery, minimizing production downtime, and adhering to strict engineering schedules all depended on this real-time capability.

For example, consider the new management system recently deployed internally. The ERP system, commonly used in the shipbuilding industry, includes a fundamental data set: the attendance module. The company collected employee attendance information using access control, facial recognition, and other technologies, which was then stored in and subscribed to via an Oracle database. As the company expanded, limitations in the analysis and aggregation queries initially based solely on Oracle became apparent. Data output, including working hours statistics, was slow, with mobile terminal data interfaces taking up to two minutes to refresh.

TapData’s real-time data capabilities addressed these issues by providing faster, more efficient data synchronization and processing. This enhancement ensured that the ERP system operated seamlessly, providing instant updates and reducing latency in data availability, thereby supporting better management decisions and operational efficiency.

This issue was effectively resolved with TapData's assistance. By using TapData to load data into TiDB in real-time and leveraging pre-built data models, the performance of data processing and analysis was significantly enhanced. TapData ensured data timeliness by instantly transmitting continuously generated real-time data to TiDB through its real-time data capture technology. Concurrently, the data was pre-organized and optimized utilizing TiDB's powerful processing capabilities, improving the efficiency of data storage and retrieval.

The combination of real-time data flow and optimization models successfully reduced data processing delays, enabling businesses to obtain real-time insights more quickly and make swift decisions. This enhancement not only provided a more reliable foundation for data-driven decision-making but also granted staff and production departments immediate access to real-time working hours statistics on mobile applications. This significantly improved business response times and user experience.

Compared to self-developed solutions, this modern data platform improved data timeliness by a factor of 60, reducing delays from minutes to seconds. Additionally, its scalable and lightweight features significantly reduced labor expenses, saving businesses substantial costs. By leveraging TapData's full and incremental synchronization capabilities, dispersed information was effectively integrated, making related data truly "integrated" and readily accessible. This accelerates the implementation of new business needs.

Ultimately, this solution facilitates the seamless incorporation of digital innovation into every aspect of production applications and business operations. It provides enterprises with the time and data resources needed to gain critical advantages in highly competitive industries. This enables them to consistently set global standards for the development of maritime and shipbuilding equipment, driving digital innovation across production, operations, and management.

Exceeding Expectations: Innovative Real-Time Data Warehouse Practices with DBT Tools for Enhanced Metric Calculation

"Don't stick to one tool and don't set limits on solutions" is the innovative approach adopted by the company's team in building a real-time data warehouse. Leveraging the exceptional real-time support capabilities of the TapData platform, the enterprise has effectively met numerous internal real-time business needs.

However, when it comes to calculating batch indicators, such as daily, weekly, and monthly material consumption and project hours, the flow calculation method can become complex and cumbersome. To address this issue, the enterprise team introduced DBT, an open-source data transformation tool that supports modular data modeling and SQL-driven processes, effectively meeting batch task requirements.

DBT enables the quick processing of numerous indicators, converting batch computation workloads into more adaptable and manageable data models, and calculating large sets of indicators more efficiently. By combining the strengths of TapData and DBT, the enterprise enhances the scalability and flexibility of the real-time data warehousing solution, improving cost efficiency and data processing performance.

This overview highlights just a portion of the capabilities demonstrated by the real-time data platform. There are even more potentials of real-time and reusable data resources waiting to be unlocked with TapData.

Advantages of Adopting the TapData Real-Time Data Platform Solution:

Extensive Data Source and Target Support: TapData supports over 100+ built-in data connectors, ensuring stable real-time collection and transmission capabilities.
Low Learning Cost, Lightweight, and Easy to Use: The platform is ready to use out of the box with low-code, visual operations. It supports data model previews and allows users to complete complex data integration and development without requiring professional programming skills.
Enhanced Real-Time Efficiency: TapData offers real-time data computing capabilities with second-level response times, alongside stable and user-friendly real-time data service features.
Support for Data and Task Classification: Customizable labels for different projects facilitate quick filtering and search, aiding in cross-department collaborative management and subsequent maintenance.
Platform-Level Data Verification: Ensures data consistency effectively.
Visual Task Running Monitoring and Alarming: The platform provides over 20 observability indicators for real-time monitoring of task statuses, ensuring immediate awareness and response to any issues.