Abstract: The smoothness indicator determines whether WENO (weighted essentially-non-oscillatory) schemes can achieve the optimal convergence accuracy and maintain the essentially non-oscillatory characteristics near discontinuities; consequently, it is the key to constructing WENO schemes. This paper proposes a new type of smoothness indicator for the widely used fifth-order WENO scheme by the operator function method. Compared to existing smoothness indicators, this new smoothness indicator has two advantages: firstly, it quantifies the smoothness of stencils and identifies discontinuities more accurately in discontinuous regions; secondly, the nonlinear weights are closer to the linear weights in smooth regions. The resulting WENO scheme maintains the optimal fifth-order accuracy at the first-order critical point and has lower dissipation, better shock-capturing capability, and higher resolution for short waves than those using existing smoothness indicators. It, therefore, captures strong shock waves and flow structures such as shear layers and sound waves more accurately, suggesting the great potential of this smoothness indicator in simulating complex flows.