Transient stability of a meshed AC/DC system when there is decreased effective inertia with high penetration of renewables has become a major challenge. This paper proposes an emergency power modulation scheme of optimal based control of an MTDC grid for improving the rotor angle stability of a power system after considering the reduction of effective inertia and effective damping. First, an AC/DC meshed system model with power modulation control of a DC grid is established by the linearization of power flow equations of the DC grid. Then, the optimal based control rule of an MTDC grid is deduced by the online minimization of the rotor speed deviations of generators and the control energy, when the limits of DC voltage, the DC line current, and the power of converters are satisfied. Finally, the nonlinear numerical simulations of one simple three-generator nine-bus AC/DC meshed system demonstrate the effectiveness of the proposed scheme in three-phase fault and sudden load changes in AC system. The result shows that the proposed scheme stands out for being better able to suppress the first swing damping oscillation performance of a multi-machine system and improving the transient stability of the AC and DC system. This work is supported by Science and Technology Project of State Grid Corporation of China (No. 5216A018000H).