Abstract: The interaction between detonation waves and boundary layers in supersonic flows plays a critical role in achieving efficient and stable combustion in oblique detonation engines (ODEs), directly affecting their thrust performance and operational reliability. Previous studies have shown that interactions between oblique detonation waves (ODWs) and the upper wall of the combustor can trigger local flow blockage, resulting in flow field instability. In practical ODE configurations, the combustor is typically confined by sidewalls, introducing combined effects from both upper and lateral boundary layers. In this study, three-dimensional numerical simulations are conducted by solving the Reynolds-averaged Navier-Stokes equations with chemical reactions to investigate the structure and instability mechanisms of ODWs under sidewall constraints. The results reveal that the interaction between shock/detonation waves and boundary layers induces local boundary layer separation and generates complex flow structures such as separation shocks and corner shocks. As the corner shocks converge toward the center in the spanwise direction and intersect near the channel centerline, an "X-shaped" three-dimensional detonation front is ultimately formed. The spanwise morphology of the X-shaped front depends on the relative position between the shock intersection point and the throat: when the intersection occurs downstream of the throat, an "M-shaped" wavefront forms; conversely, when the intersection is upstream, the structure transitions to a central "Ω-shaped" pattern. Further analysis of unstable ODWs reveals that the spanwise expansion of the separation region is the key mechanism leading to global instability of the detonation wave system. This study, for the first time, elucidates the formation and destabilization mechanisms of complex three-dimensional ODWs under sidewall confinement, providing insights for the optimized design of ODE combustor flow paths.