Detailed model of STATCOM controller have many difficulties in stability validation. This paper adopts the transient stability (TS) model instead of the detailed model and uses improved Prony algorithm to extract the parameters of the power system. It avoids low accuracy of amplitude and phase calculated and overcomes the shortcoming in weak anti-interference ability of traditional algorithm. Standard control and supplementary control analysis of the detailed model and TS model is made in PSCAD/EMTDC. Simulation result show that TS model and the improved Prony method can solve the complex problem of controller stability analysis effectively.