A reinforcement-learning search reached the highest-reward design in a simulated mine-pump system after 26 episodes, or 130 executable simulations, according to a preprint. It was reported to converge within 32 episodes. The result suggests that reusing experience from earlier simulations may reduce the effort needed to explore the system’s design space, but the comparison with other methods was not made under the same repeated-run protocol.
The study asks whether prior simulation traces and temporal-property outcomes — the results of checks applied as system behaviour unfolds over time — can be reused through policy learning to reach a high-quality design with fewer simulations than genetic-algorithm or Bayesian-optimisation search under the same executable model and reward. Its stated hypothesis was an association between that reuse and lower simulation effort.
A million encoded choices
To test that question, the workflow assembled candidate models from subsystem variants and ran them with stochastic environmental and fault behaviour. The RL agent selected designs during exploration, rather than controlling the CPS at runtime. The case study was a methane-sensitive mine-pump CPS, and its executable model was provided as part of the work.
The model had three configurable subsystems, with 10 discrete levels for each dependability and recoverability dimension. Together, those choices produced 1,000,000 encoded candidate designs. Those figures defined the design space used in the comparison.
What counted as a good design
DDPG was the main RL method, using an actor–critic structure in which one component proposes an action and another estimates its value. In this study, the action was a bounded design-space choice, with integer levels from 1 to 10. The agent was not a runtime pump controller.
The reward, or score used to rank designs, combined five Boolean — yes-or-no — temporal-property indicators, sp1 through sp5. Their contributions were accumulated over time and weighted by severity, alongside terms for budget, recoverability, sustained compliance and operational use at the end of the simulation.
RL-guided, GA-based and BO-based searches were evaluated under the same executable model, design encoding and reward. Simulation effort meant counting executable Simulink evaluations before a high-reward design was identified; reward quality meant the best design reward recorded during the search.
A split result on speed and reward
For the baselines, BO and GA each had at least 20 runs. Runs stopped when the relative half-width criterion reached δn ≤ 0.10, with a maximum of 50 runs. RL used its available training campaign instead of the same repeated-run setup.
Within the common executable setup, RL found the highest-reward design after 26 episodes, corresponding to 130 executable simulations, and converged within 32 episodes.
BO looked stronger on final reward quality. Its mean best reward was 176,765 and its median was 196,504, compared with 130,993 and 118,931 for GA. Both methods reached approximately 209,000 at their best. The reported effect size was d = 0.98, alongside rank-sum and two-sample t-test p-values of 4.94 × 10−4 and 1.55 × 10−4.
The pattern held at the study’s chosen thresholds. BO exceeded 190,000 reward units in 68% of runs and 200,000 in 36%, while GA did so in 24% and 10% of runs. BO stopped after 25 runs; GA reached the 50-run cap. Those run histories are part of why the results do not amount to a fully like-for-like statistical comparison with RL.
Why the comparison remains provisional
The ablation results were less uniform. Actor-only RL was reported as slower and less stable and required more episodes to reach high-reward designs. A Q-informed GA variant had better reported search performance than standard reward-driven GA.
Taken together, the reported evidence is strongest for a narrower point: the RL campaign reached a high-reward design with the stated evaluation count, and BO produced higher final-reward summaries and threshold rates than GA in the reported baseline runs. Because the methods did not all share the same repeated-run statistical protocol, the numerical comparison across RL, GA and BO remains limited.
The manuscript header identifies arXiv:2608.23440v2, dated 25 Aug 2026, and describes it as an accepted manuscript for IECON 2026. The work was supported by the Luxembourg National Research Funds through the VARIANCE project grant C23/IS/18177547/VARIANCE.
Paper data and sources
Original title: Temporal Property-driven Design Space Exploration with Reinforcement Learning for Cyber-Physical Systems
Authors: Tagir Fabarisov, Maxime Cordy
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-24
DOI: Not available
Original paper · Full text