In order to meet the requirements of real-time analysis of power system transient stability, the Generalized Backward Differentiation Formulae (GBDF), which is a kind of boundary value methods, is applied to the transient stability numerical calculation, and a new algorithm is proposed. The proposed algorithm uses GBDF to carry on the continuous time discretization to the differential equations, and then uses the Newton method to solve the whole nonlinear system of the discretized nonlinear equations. Based on the band structure characteristic of the global Jacobian matrix, a special matrix equation split-combination technique is used to avoid the triangular factorization of the global Jacobian matrix or multiple block sub-matrices, thus to improve the efficiency of numerical calculation of transient stability. The test results of two example systems show that the proposed algorithm has obvious advantages over the classical implicit trapezoidal rule in terms of computational efficiency. This work is supported by National Natural Science Foundation of China (No. 51377098).