A Digital Twin Framework for Real-Time Monitoring and Full-Field Analysis of Bridges

Abstract

Digital twin technology has been recognized as a transformative approach for civil structural health monitoring (SHM), but real-time structural analysis and result visualization remain challenging because a large-scale civil structure contains thousands of components, making real-time finite element analysis very time-consuming. This study proposes an integrated digital twin framework to address the limitations of traditional finite element methods for real-time analysis through a synergistic combination of SHM and advanced machine learning techniques. A footbridge with an SHM system is used as the testbed. A novel transformer-based surrogate model, named SAE-Transformer, is developed for real-time calculation of the three-dimensional full-field temperature distribution and thermal stress of the entire bridge. One year of bridge data was adopted for training the SAE-Transformer. Equipped with a sequence-aware embedding module and a SwiGLU activation layer, the SAE-Transformer model reduces the temperature prediction error by 39% and the stress estimation error by 45% compared to the state-of-the-art deep learning models. Moreover, the modal superposition technique is adopted to recover the full-field vibration of the bridge using monitoring data from sparse sensors, enabling real-time visualization at 50 frames per second. The implementation demonstrates that the proposed machine learning-empowered technique realizes real-time full-field analysis of large-scale civil structures for the first time.

Introduction

The digital twin technology^1–13,14 plays a significant role in enhancing the structural health monitoring (SHM) of bridges.^15–24 By creating a real-time virtual representation of the physical bridge structure, digital twins facilitate continuous monitoring and assessment of structural integrity, leading to cost-effective maintenance decisions. The integration of real-time monitoring data from sensors and predictive analytics not only improves the accuracy of health assessments but also supports scenario-based planning and operational optimization throughout the lifespan of the bridge. Moreover, the visual representation provided by digital twins enables engineers to intuitively understand the structural conditions through three-dimensional (3D) models, facilitating potential defect identification and timely alarms to avoid further structural damage and failure.

Due to these merits, the digital twin technology has been rapidly developed in civil SHM during the past years. For example, Jiang et al.²⁵ proposed a digital twin-based aerodynamic admittance function identification method using field measurement data collected by an SHM system. Ye et al.²⁶ developed a digital twin-based framework for identifying the crystal plasticity material parameters of weld joints in orthotropic steel decks through the fusion of a multiscale crystal plasticity finite element (FE) model with macroscale stress–strain material tests. Lin et al.⁷ proposed a digital twin-based collapse fragility assessment method for long-span cable-stayed bridges under strong earthquakes, demonstrating its feasibility and accuracy for seismic collapse assessment. Teng et al.²⁷ combined digital twins of bridges with transfer learning and achieved 97.3% accuracy in damage detection for a real bridge structure. Lu et al.²⁸ developed a digital twin-based anomaly detection process flow.

However, existing digital twin systems fall short in delivering real-time analysis and visualization across the entire structure. This is because a large-scale civil structure contains thousands or even millions of components, and thus the real-time FE analysis is very time-consuming. For example, Shan et al.²⁹ analyzed the Hong Kong–Zhuhai–Macao Bridge and took 300 sec to obtain the temperature distribution of a bridge segment with approximately 75,000 elements, preventing the real-time analysis and visualization, not mentioning the whole structure and dynamic analysis. Consequently, this inefficiency of the FE analysis leads to most SHM system using offline analyses only or adopting threshold-based criterion according to the measurement data, thereby limiting the full potential of digital twin applications for SHM.

To address these challenges, this study proposes an integrated approach combining deep learning (DL)-based surrogate models and high-performance visualization techniques to enhance the real-time digital twins for bridge SHM. The footbridge in The Hong Kong Polytechnic University campus is used as the testbed. The main contributions of this paper are as follows: A surrogate DL model named SAE-Transformer is developed to enable a rapid calculation of temperature fields and thermal stresses of the entire bridge. Compared to the state-of-the-art transformer model, the proposed method reduces the temperature error by 39% and the stress error by 45%. A web-based computing and rendering platform is implemented using TensorFlow.js³⁰ and WebGL to enable seamless visualization of dynamic structural responses. Various backends are evaluated under different conditions to balance throughput and latency.

Footbridge and the Health Monitoring System

Footbridge descriptions

The footbridge connecting Blocks Z and X at The Hong Kong Polytechnic University’s main campus was opened on October 15, 2019. With a total length of 84.24 meters, the structure comprises a 64.26-meter main span crossing the underlying highway and a 19.98-meter side span³¹. As illustrated in Figs. 1 and 2, the bridge features a distinctive structural system in which the deck is supported by two butterfly-shaped steel tube arches. These arches connect to deck beams that are supported by a tower at Block X on one end and anchored to the podium floor of Block Z on the opposite end. The primary structural components, including arches, beams, and columns, are made of steel, while the bridge deck utilizes reinforced concrete construction.

Figure 1. Bird’s-eye view of the footbridge

Figure 2. Elevation of the footbridge

Health monitoring system of the bridge

The SHM system was implemented concurrently with the construction of the bridge, commencing data acquisition on September 28, 2019. Following comprehensive testing and commissioning, the system became fully operational on October 13, 2019. The integrated sensor network comprises 88 permanent sensors of 13 distinct types,³¹ systematically organized to monitor three types of parameters: environmental conditions (air temperature, barometric pressure, humidity, precipitation, and solar radiation), loads (wind forces, pedestrian traffic, and structural temperature variations), and structural responses (stress/strain, displacement, and acceleration).

As depicted in Fig. 3, the sensor array features a strategically distributed configuration. Notably, a pyranometer is mounted on the rooftop of Block X to capture real-time solar radiation data, and a weather station on the roof of Block X provides the ambient temperature. Eight accelerometers (four uniaxial and four triaxial) are placed on the deck beams to measure the vertical and horizontal accelerations. Thirty-six vibrating wire strain gauges are deployed on steel girders, arches, and the concrete deck to provide strain measurements. Seven autonomous data acquisition units (DAU-1 to DAU-7) are distributed along the bridge to collect monitoring data from nearby sensors and then transmit it to the Hub for processing at the Data Center.

Figure 3. Overall framework. Note: GNSS‒Global Navigation Satellite System; MAC‒Media Access Control; BOTDA‒Brillouin Optical Time-Domain Analysis; FBG‒Fiber Bragg Grating; DAU‒Data Acquisition Unit; CCTV‒Closed-Circuit Television

Overview of the proposed framework

This study will realize real-time digital twinning of the footbridge through two tasks: The first is to develop a DL model to calculate the full-field temperature distribution and thermal stress of the entire bridge. The second is to recover the vibration of the structure using real-time acceleration measured at eight accelerometers. These tasks will be detailed in Sections 3 and 4.

As shown in Figs. 3 and 4, the Data Center on the structure aggregates sensor data from DAUs and routes it through the switch to the frontend workstation, which manages the touchscreen interface. The backend server simultaneously handles time-series data storage and executes DL neural network inferences.

Figure 4. Data flow of the proposed framework

DL-Based Temperature and Stress Real-Time Calculation

The task aims to compute the full-field temperature distribution and temperature-induced stress of the bridge in real time under varying environmental conditions, utilizing measured air temperature and solar radiation data from the SHM system. As described in the Introduction, the FE analysis of the full-field temperature distribution and associated stress is very time-consuming. A DL model will be developed to replace the FE model, enabling real-time calculation.

As the SHM system has only a limited number of sensors, the FE model is utilized to generate datasets for the training and evaluation process of the DL model.

FE model

A 3D FE model is developed using the commercial FE analysis software ANSYS 16.0.³² The arches, main beams, deck, and columns are modeled using solid elements for the global 3D heat-transfer analysis. The full 3D model consists of 8,785 elements and 22,585 nodes, as shown in Fig. 5.

Figure 5. 3D view of the FE model

The 3D FE model has been validated in our prior studies.^{31, 32} The initial FE model was updated so that its vibration properties (frequencies and mode shapes) closely match the field measurement data in an optimal manner.³¹ Moreover, a heat-transfer analysis was conducted to calculate the temperature distribution of the bridge.³² The temperature data were compared with the field monitoring data across all four seasons. The obtained temperature data were then input into the FE model to calculate the temperature-induced stress and displacement of the bridge via structural analysis. The numerical data were also compared with the field measurements. The numerical results showed good agreement with the field monitoring data, verifying the effectiveness of the 3D model and the associated numerical heat-transfer and structural analyses.

The validated FE model will be used first for heat-transfer analysis and then for structural analysis. Ambient air temperature and solar radiation are applied as thermal boundary conditions in the analysis. The hemisphere technique is employed to determine the view factors between various bridge surfaces, facilitating the calculation of solar radiation exposure on all external surfaces at various times and dates. A 3D heat-transfer analysis is performed to determine the thermal distribution across the entire bridge structure. Details of the heat-transfer analysis can be found in a previous study.³²

The resulting temperature data for each node of the FE model are then seamlessly input into the same FE model as thermal loads to compute temperature-induced structural responses (stress in this study) through the structural analysis. This strategy avoids manual data input, as in the traditional approach. It is noteworthy that both analyses utilize the same FE model while employing different element types for their specific purposes.

Inputs and outputs of DL-based methods

In the present DL model, ambient temperature and solar radiation are used as model inputs, and the temperature distribution/stresses at all nodes of the FE model are the outputs.

The solar elevation angle and solar azimuth angle vary at different times and dates. They affect the radiation intensity received by the external bridge surface and thus are also included as DL inputs. Material thermal properties, such as the heat transfer coefficient and absorption coefficient, affect heat transfer between the surfaces and the environment of the bridge. As these coefficients are constant, they can be learned by the neural network implicitly and thus do not need to be included as inputs.

Daily maximum and minimum temperature and daily total solar radiation data obtained from the Hong Kong Observatory in 2023 are used to generate the hourly dataset for DL. The hourly input data are generated as follows: The hourly air temperature data are calculated from the maximum and minimum values using the following equation: in which where $t^{\ast }$ denotes any hour of the day. The solar radiation data before sunrise and after sunset are set to zero, and the data for other hours are calculated from the following equation according to the solar elevation angle. where $I_0$, $h$, and $p$ are the solar constant, the elevation angle, and the atmospheric transparency factor, respectively. The solar elevation angle and solar azimuth angle at different hours and dates are calculated using the following equations. where $\varphi$ and $\lambda$ denote the latitude and longitude, respectively; $\delta$ and $\omega$ are the solar declination angle and solar hour angle, respectively; $D$ is the day number in the year counted from January 1; $S_d$ is the local solar time; and $S$ and $F$ are the local hour and minute values, respectively.

(1) $$T_a=T_{ave}+T_{dif}\cdot sin\left[\frac{\pi }{12}(t^{\ast }-9)\right]$$

(2) $$T_{ave}=\frac{1}{2}[max{T}_a(t)+min{T}_a(t)]$$

(3) $$T_{dif}=\frac{1}{2}[max{T}_a(t)-min{T}_a(t)]$$

(4) $$I_m=I_0\frac{\sinh }{sinh+\frac{1-p}{p}}$$

(5) $$h=cos\varphi cos\delta cos\omega +sin\varphi sin\delta$$

(6) $$\sin y_z=\frac{cos\delta sin\omega }{cosh}$$

(7) $$\delta =23.45sin(360\mathbf{\times }\frac{284+D}{365})$$

(8) $$\omega =(s_d-12)\mathbf{\times }{15}^{\circ }$$

(9) $$s_d=S+\frac{F-(120-\lambda )\mathbf{\times }4}{60}$$

It is noted that the thermal effect of ambient temperature and solar radiation on the structure has a considerable time lag, which is closely related to the heat conduction and heat capacity characteristics of the structure. In this regard, the ambient temperature and radiation at the current hour and the previous 5 hours are taken as model inputs to accurately reflect the thermal time lag of the structure.

Using the above-mentioned FE model and heat-transfer analysis, the hourly outputs (temperature distribution and thermal stresses) at all nodes of the entire bridge in 2023 are calculated. The overall formula is presented as follows: where $T_t^b$ and ${\sigma }_t^b$ denote the full-field ambient temperature and temperature-induced stress at time $t$, respectively; $T^a$, $R$, $H$, and $A$ are the ambient temperature, direct solar radiation intensity, solar altitude angle, and solar azimuth angle at time $t$, respectively.

(10) $$T_t^b,{\sigma }_t^b=f\left(\left[\begin{array}{cccc}{T}_{t-5}^a& R_{t-5}& H_{t-5}& A_{t-5}\\ ⋮& ⋮& ⋮& ⋮\\ T_t^a& R_t& H_t& A_t\end{array}\right]\right)$$

The dataset consists of 8,760 (365 $\times$ 24) samples, which are partitioned into a training set (90%) and a validation set (10%) for model training and evaluation.

Data preprocessing

The scales of the model inputs and outputs differ significantly. For example, the temperatures range from 0 to 50°C, the solar angles span from 0° to 90°, the radiation values range from 0 to 1000 W/m², and the stress values vary greatly from 0 to 100 MPa. These significantly different feature scales will cause elliptical “contour lines” of the loss function, which in turn make the gradient descent paths oscillate inefficiently. Therefore, the inputs and outputs are normalized using the min–max method so that the scales of all feature dimensions align, making the loss function’s contours “circular.” This results in a more efficient optimization, a higher learning rate, and rapid convergence.

Structure of the proposed DL model

This DL model aims to predict the responses of structure from the environmental data and related geometric data at different times. Sequence modeling is appropriate for addressing these types of time-series problems.

The transformer³³ model is built upon the attention mechanism and was originally designed to improve neural language processing tasks. Its long-range dependency modeling capability makes it excellent for a wide range of sequential tasks. Therefore, a customized transformer model is developed in this study to predict real-time temperature and thermal stress. Its architecture is demonstrated in Fig. 6.

Figure 6. The architecture of the proposed SAE-Transformer method

The input tensor undergoes parallel processing through two distinct embedding pathways: the input embedding and sequence embedding branches. The resulting feature maps from both branches are then layer-normalized³⁴ (LN in Fig. 6) and concatenated along the feature dimension, forming a comprehensive representation that serves as input to four cascaded SwiGLU Transformer encoder layers. Finally, the processed features are flattened and transformed into the output space through a linear projection layer.

The proposed architecture is detailed in the following subsections.

Embedding mechanisms

In the input embedding branch, the inputs are projected into a high-dimensional latent space, where learnable position encoding is incorporated to preserve spatial–temporal relationships. The process involves linear embedding, where a linear layer is utilized to transform the feature dimensions from 4 to 24 at each time step. Subsequently, learnable position encoding is applied to enhance the expressive performance of the model, particularly since the input includes heterogeneous data types such as temperature, solar radiation, and angles.

The sequence embedding branch is dedicated to sequence embedding, which is designed to augment the sequence processing capabilities of the embedding. Specifically, this module comprises two stacked long short-term memory (LSTM)³⁵ layers, enabling it to effectively capture the sequential information within the input sequence and integrate it into the feature vector.

Both components are independently normalized and then combined to produce the final input embedding.

Encoder and linear decoder

Given that our goal is to compute temperature or stress values at each node, the output is of fixed length and no sequences need to be generated. It is sufficient to capture contextual information and generate high-quality representations of the features using only the transformer encoder, thus eliminating the need for a transformer decoder. The linear layer is thus used as a decoder to map these representations to the output.

To improve the nonlinear performance of the model, a SwiGLU feed-forward layer is used to replace the traditional feed-forward network. This function combines the GLU³⁶ gating mechanism and the Swish activation function.³⁷ The gating mechanism allows the network to transfer information selectively, thus improving the nonlinear performance.

The general formula for SwiGLU is shown as

(11) $$FF{N}_{SwiGLU}(x,W,V)=(Swish(xW)\mathbf{\otimes }xV)$$

It is revised as follows in practice: where $W,V,\text{and}{W}_2$ are weights of the linear layers, and $\mathbf{\otimes }$ is the element-wise product.

(12) $$FF{N}_{SwiGLU}(x,W,V,W_2)=(Swish(xW)\mathbf{\otimes }xV)W_2$$

Implementation

The DL network is developed in PyTorch³⁸ and trained using the AdamW optimizer³⁹ with a batch size of 128 for 300 epochs. The learning rate is configured with a maximum value of 10⁻³ and a minimum value of 10⁻⁷. Specifically, the reduce-on-plateau scheduling strategy is employed, where the learning rate is halved if there is no reduction in loss for 10 consecutive epochs. The L1 loss is chosen as the supervised loss function, which can effectively reduce the influence of outliers.

Model training (7,884 samples) is run on NVIDIA Tesla A100 GPUs, taking approximately 20 minutes for temperature field convergence and 50 minutes for thermal stress training for the entire year of 2023. The trained DL model is then validated on the 876 test samples. The quantitative and qualitative evaluation results are shown in the following sections.

DL model quantitative evaluation

The DL model’s performance is evaluated under three temperature conditions from the test dataset: (1) the hottest day, (2) the coldest day, and (3) a normal temperature day. The histograms of the temperature errors under these three conditions are shown in Figs. 7–9, where the error is the difference between the model-predicted temperature and the FE analysis result.

Figure 7. Histogram of temperature errors on the hottest day

Figure 8. Histogram of temperature errors on the coldest day

Figure 9. Histogram of temperature errors on a normal temperature day

The histogram figures reveal larger errors and a more dispersed error distribution on the coldest day, which can be attributed to the limited number of samples representing low temperatures in Hong Kong. This observation suggests that the elevated errors under extreme cold conditions may result from sample imbalance. In contrast, the error distributions on both the hottest day and the normal temperature day exhibit notable concentration. In particular, the smallest error occurs under the normal temperature condition, specifically less than 1°C, demonstrating the robustness of the model in predicting full-field temperature distributions with high accuracy.

The performance of the proposed DL model is also compared with three other DL models: a multilayer perceptron (MLP), a hybrid MLP model incorporating LSTM,³⁵ and the Vanilla Transformer.³³ Their errors, computational costs, and inference speeds are presented in Table 1. Here, the absolute error is averaged over all nodes and all samples.

Table 1. Accuracy and computational cost of the proposed method

	Temperature				Stress
	Error(°C)	Params(M)	MACs(M)	Time(ms)	Error(MPa)	Params(M)	MACs(M)	Time(ms)
MLP	0.27	23.85	23.83	0.4	0.59	70.15	70.08	0.4
LSTM + MLP	0.27	23.92	23.94	0.6	0.22	70.22	70.20	0.6
Vanilla Transformer	0.23	6.96	8.92	1.4	0.09	20.02	21.97	1.4
SAE-Transformer (proposed)	0.14	6.99	8.92	1.6	0.05	20.05	21.97	1.6

The test results show that the proposed method has a much smaller error than the other three models. In particular, the proposed model achieves a 49%, 49%, and 39% reduction in temperature prediction error and a 92%, 77%, and 44% reduction in stress prediction error, compared with the other methods. Moreover, the proposed SAE-Transformer uses around 30% of the parameters (Params) and total multiply-accumulate operations (MACs) required by the MLP and LSTM + MLP methods.

In terms of inference speed, the proposed SAE-Transformer and the Vanilla Transformer exhibit lower efficiency. This is primarily attributed to the multiple branching structures within the transformer architecture, which result in lower parallelism and higher memory access overhead. Nevertheless, considering that the digital twin platform usually adopts a 60 Hz display device, the rendering time of a single frame is about 16.7 ms (even in high refresh rate scenarios, such as the current state-of-the-art 300 Hz display device, the time for a single frame is 3.3 ms). The inference times of the above methods are significantly shorter than the frame rendering time; that is, all models can fully meet the real-time requirement.

DL model qualitative evaluation

The temperature distributions of the entire bridge under the three temperature conditions (hottest day, coldest day, and normal temperature day) are plotted in Figs. 10–12. The ground truth and the error are also shown.

Figure 10. Temperature distribution of the bridge on the hottest day

Figure 11. Temperature distribution of the bridge on the coldest day

Figure 12. Temperature distribution of the bridge on a normal temperature day

The errors on the extreme temperature days (both highest and lowest) are more significant than those on the normal temperature day because the former have much fewer samples than the latter.

The results also reveal that the temperature prediction errors are predominantly concentrated in the arch regions. This phenomenon may be attributed to the intricate geometric features of these areas, given that the architecture of the model does not incorporate geometric structures as input parameters. It is plausible that integrating occlusion models or surface normals would improve accuracy within these regions. Moreover, daytime shows more significant errors than the evening.

Figs. 13–15 show the temperature-induced stress of the bridge under different temperature conditions. Only the stress in the X-direction (or longitudinal direction) of the bridge is illustrated due to length limitations. Again, the main arches and legs exhibit higher errors than other components, and the errors on the extreme weather days are larger than those on the normal day. These findings are consistent with the temperature results.

Figure 13. Distribution of thermal stress (X-direction) on the hottest day

Figure 14. Distribution of thermal stress (X-direction) on the coldest day

Figure 15. Distribution of thermal stress (X-direction) on a normal temperature day

Data-Driven Vibration Calculation in Real Time

The footbridge vibrates continuously due to dynamic loads, such as pedestrians and wind. The vibrations can be measured quantitatively via accelerometers. Therefore, the monitoring data from the accelerometers can be combined with advanced computing frameworks for real-time visualization of the complex vibrations.

As the vibration consists of many different modes, the modal superposition method⁴⁰ is used to decompose complex vibrations into different natural modes. Natural frequencies and corresponding mode shapes derived from the FE analysis are utilized.

Theory

In this task, data from accelerometers are used to reconstruct the full-field displacement. The principle is demonstrated as follows.

Assume that the displacements at all degrees of freedom of a structure (1, 2, …, N) at one instance are composed of $M$ modes, and $\left\{r\right\}$ is the modal participation coefficient. where ${\mathrm{\Phi }}_i$ is the ith mode shape of the structure at $N$ degrees of freedom. Then, the displacements at $n$ sensor locations at that instant are also composed of $M$ modes, where ${\phi }_i$ is the ith mode shape at $n$ the sensor locations ($n\ll N$). To avoid ill-conditioning in the calculation, $M$ can be set to the same value as $n$. As displacements are not available in the SHM system, acceleration data are used instead. Double differentiation of Eq. (14) will lead to the following equation, which indicates that the acceleration data are also a superposition of the lowest modes

(13) $${\left\{\begin{array}{c}{\text{X}}_1\\ {\text{X}}_2\\ ⋮\\ {\text{X}}_N\end{array}\right\}}_{N\times 1}={\left[\begin{array}{llll}{\mathrm{\Phi }}_1& {\mathrm{\Phi }}_2& \cdots & {\mathrm{\Phi }}_M\end{array}\right]}_{N\times M}\ast {\left\{r\right\}}_{M\times 1}$$

(14) $${\left\{\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right\}}_{n\times 1}={\left[\begin{array}{llll}{\phi }_1& {\phi }_2& \cdots & {\phi }_M\end{array}\right]}_{n\times M}\times {\left\{r\right\}}_{M\times 1}$$

(15) $${\left\{\begin{array}{c}{\ddot{x}}_1\\ {\ddot{x}}_2\\ ⋮\\ {\ddot{x}}_M\end{array}\right\}}_{M\times 1}={\left[\begin{array}{llll}{\phi }_1& {\phi }_2& \cdots & {\phi }_M\end{array}\right]}_{M\times M}\times {\left\{\ddot{r}\right\}}_{M\times 1}$$

The following relationship exists between the acceleration mode participation factor and the displacement mode participation factor:

(16) $$\begin{array}{rl}{\left\{\ddot{r}\right\}}_{M\times 1}& =-{\left[{\omega }^2\right]}_{M\times M}\times {\left\{r\right\}}_{M\times 1}={{\left[\begin{array}{llll}{\phi }_1& {\phi }_2& \cdots & {\phi }_M\end{array}\right]}^{-1}}_{M\times M}\\ & \times {\left\{\begin{array}{c}{\ddot{x}}_1\\ {\ddot{x}}_2\\ ⋮\\ {\ddot{x}}_M\end{array}\right\}}_{M\times 1}\end{array}$$

Therefore, the displacement modal participation coefficient can be calculated from the measured acceleration as

(17) $${\left\{r\right\}}_{M\times 1}=\frac{}{}{\left[{\omega }^2\right]}^{-1}{{\left[\begin{array}{llll}{\phi }_1& {\phi }_2& \cdots & {\phi }_M\end{array}\right]}^{-1}}_{M\times M}\times {\left\{\begin{array}{l}{\ddot{x}}_1\\ {\ddot{x}}_2\\ ⋮\\ {\ddot{x}}_M\end{array}\right\}}_{M\times 1}$$

Substituting Eq. (17) into Eq. (13) results in the full-field displacement as

or simplified as

$$\begin{array}{rl}& {\left\{\begin{array}{c}{\text{X}}_1\\ {\text{X}}_2\\ ⋮\\ {\text{X}}_N\end{array}\right\}}_{N\times 1}={\left[\begin{array}{llll}{\varphi }_1& {\varphi }_2& \cdots & {\varphi }_M\end{array}\right]}_{N\times M}\\ & \times {\left[\begin{array}{lll}-{\omega }_1^2& & \\ & \ddots & \\ & & -{\omega }_M^2\end{array}\right]}^{-1}\times {{\left[\begin{array}{llll}{\phi }_1& {\phi }_2& \cdots & {\phi }_M\end{array}\right]}^{-1}}_{M\times M}\\ & \times {\left\{\begin{array}{c}{\ddot{x}}_1\\ {\ddot{x}}_2\\ ⋮\\ {\ddot{x}}_M\end{array}\right\}}_{M\times 1}\end{array}$$

(18) $${\left\{\begin{array}{c}{\text{X}}_1\\ {\text{X}}_2\\ ⋮\\ {\text{X}}_N\end{array}\right\}}_{N\times 1}={\left[T\right]}_{N\times M}\times {\left\{\begin{array}{c}{\ddot{x}}_1\\ {\ddot{x}}_2\\ ⋮\\ {\ddot{x}}_M\end{array}\right\}}_{M\times 1}$$

Matrix [T] can be calculated in advance as

(19) $$\begin{array}{rl}{\left[T\right]}_{N\times M}& ={\left[\begin{array}{llll}{\varphi }_1& {\varphi }_2& \cdots & {\varphi }_M\end{array}\right]}_{N\times M}\\ & \times {\left[\begin{array}{lll}-{\omega }_1^2& & \\ & \ddots & \\ & & -{\omega }_M^2\end{array}\right]}^{-1}\\ & \times {{\left[\begin{array}{llll}{\phi }_1& {\phi }_2& \cdots & {\phi }_M\end{array}\right]}^{-1}}_{M\times M}\end{array}$$

Modal selection

Due to the negligible axial vibration observed in the bridge structure, axial acceleration measurements are omitted, and the analysis focuses solely on vertical and horizontal displacements. Consequently, only 12 sensors data are used to reconstruct the displacement at all nodes in three directions, that is M = 12 and N = 67,755 in the study. The frequencies and full-field mode shapes are calculated from the FE model. As the calculated mode shapes may contain local modes, which should be removed from the above reconstruction. Therefore, appropriate 12 modes are selected from the FE analysis and listed in Fig. 16.

Figure 16. Selected 12 mode shapes

Performance optimization

Due to the real-time requirements of visualization, the computation should be fast with low latency, thus requiring a series of optimizations.

The displacement reconstruction process must meet a 50-Hz processing requirement to achieve real-time visualization. However, the system’s B/S (Browser/Server) architecture restricts the front-end implementation to JavaScript. Given the single-threaded nature of native JavaScript, matrix computations are limited to sequential element-wise operations, creating a critical bottleneck in computational performance. This single-threaded computational model introduces additional challenges. For instance, the 50-Hz refresh rate requires processing network requests every 20 ms, yet network conditions cannot consistently guarantee such low latency. While buffering could mitigate this issue, insufficient computational throughput leads to sample accumulation in the buffer, eventually causing buffer overflow. These computations also block the UI thread, potentially resulting in degraded responsiveness of the user interface.

To build a high-performance visualization platform, we use TensorFlow.js³⁰ for matrix computation and WASM⁴¹ as its backend. For the footbridge, one-step displacement reconstruction takes 0.08 ms on a laptop powered by an AMD Ryzen 7940HX CPU and an NVIDIA RTX 4060 Laptop GPU. Therefore, this approach can achieve excellent performance even though the FE model contains 67,755 degrees of freedom. It has the potential to achieve the real-time visualization for even large-scale structures with millions of nodes.

Conclusions

A DL-based model has been developed to obtain the full-field temperature and temperature-induced stress of a footbridge from the SHM system, enabling real-time monitoring and analysis. The developed SAE-Transformer model reduces the temperature prediction error by 39% and the stress estimation error by 45% compared to the state-of-the-art DL models. The vibration of the entire structure is also reconstructed from the sparse sensors on the bridge, enabling real-time visualization. This study is the first of its kind to demonstrate a real-time digital twin framework on a real civil structure.

The temperature distribution and the associated temperature-induced stresses are calculated in this paper, while the proposed framework can also be applied to calculate other structural responses.

The neural network architecture developed in this study can be extended to larger or more complex structures by adjusting the network depth (number of encoders) and width (feature dimension of the encoders) as required. The computational overhead is maintained at low levels due to the input sparsity characteristics. Real-time inference capability is enabled by the powerful parallel computing capacities of modern GPUs. In future work, we will apply the developed framework to a large-scale structure.

Under out-of-distribution conditions, increased prediction errors in temperature fields were observed during cold weather, which was consistent with theoretical expectations. To address this limitation, one can employ data augmentation strategies, such as generating simulation data to broaden the coverage of the data distribution.