Recently the study on reactive power compensation in AC-DC grids has received more and more attention. Based on improved particle swarm optimization (PSO), this paper presents a novel method to optimize reactive power of AC-DC grids. Firstly, the locations of reactive power compensation are optimized. According to the principles of the singular value decomposition, the methods of multiple flow calculation and successive determined-point are adopted to avoid over-compensation. Then, an improved PSO method based on grouping optimizing is applied to optimize the compensation capacity, which can overcome the problem of premature convergence and improve optimization speed. Finally, the proposed optimal algorithm is tested through the modified IEEE-57 bus system. Compared with traditional methods, this method can effectively improve power quality of AC-DC grids, reduce reactive power compensation capacities, and has better optimization effect.