Out-of-step oscillation separation devices based on apparent impedance angle trajectory are widely used in the power grid in China, and their reliability is of great importance. In this paper, the influence mechanism of series compensation on apparent impedance angle trajectory is analyzed. It reveals that series compensation capacitors reduce the equivalent impedance angle and consequently the apparent impedance angle cannot pass through all four zones, resulting in the risk of failure for the separation devices. A revised method of apparent impedance angle based on compensation principle is discussed, and then an improved out-of-step oscillation detection criterion is proposed. In view of the fact that the apparent impedance angle at the moment of maximum current is equal to the equivalent impedance angle during an oscillation process, a practical algorithm based on the compensated apparent impedance angle is proposed. RTDS tests are conducted to verify the effectiveness of the new criterion, considering transmission lines with a high series compensation level.