Stable and safe operation of power grids is an important guarantee for economy development. Support Vector Machine (SVM) based stability analysis method is a significant method started in the last century. However, the SVM method has several drawbacks, e.g. low accuracy around the hyperplane and heavy computational burden when dealing with large amount of data. To tackle the above problems of the SVM model, the algorithm proposed in this paper is optimized from three aspects. Firstly, the gray area of the SVM model is judged by the probability output and the corresponding samples are processed. Therefore the clustering of the samples in the gray area is improved. The problem of low accuracy in the training of the SVM model in the gray area is improved, while the size of the sample is reduced and the efficiency is improved. Finally, by adjusting the model of the penalty factor in the SVM model after the clustering of the samples, the number of samples with unstable states being misjudged as stable is reduced. Test results on the IEEE 118-bus test system verify the proposed method.