Deep peak-shaving in coal-fired boilers causes non-linear dynamics, rendering traditional PID control insufficient for NOx emission management. This paper proposes a Modelica-based digital twin framework for a 600 MW boiler. The CFD-PINN model is coupled with a physicsbased steam-side dynamic model and an adaptive correction layer driven by DCS (Distributed Control System) data to ensure high-precision synchronization under operating condition drift. An intelligent reinforcement learning agent using the Recurrent Proximal Policy Optimization (Recurrent PPO) algorithm is developed to inject residual bias signals into the main loop, optimizing NOx emissions without altering the original architecture. Validated in seven scenarios spanning 300–600 MW, this strategy reduces the 95thpercentile NOx by 19%–23% and emission fluctuations by 32.7%–43.2% across four dynamic scenarios relative to both manually tuned and gain-scheduled PID baselines, while maintaining combustion efficiency. This approach provides a deployable, low-risk pathway for achieving stricter NOx compliance under deep peak shaving operation.