Primal-dual interior point method has higher efficiency and accuracy while solving OPF problem of AC/DC system with VSC-HVDC, but it cannot solve the OPF problem with discrete variables (such as reactive power optimization), the intelligent algorithm is easy to fall into local optimal solution when solving the problem and cost too much time. Therefore, a hybrid interior point method with discrete penalty function is proposed. The main idea of this algorithm takes interior point method as framework, and optimizes the continuous variable. When the duality gap is less than a certain value, it introduces the penalty function into discrete variables calculation, and at the same time adjusts the penalty factor as the iteration difference change. The numerical example shows that the algorithm has good stability, strong ability of searching and can solve the optimization problem of the AC/DC system with VSC-HVDC containing discrete variables.