In this paper, an optimal nonlinear robust sliding mode control (ONRSMC) based on mixed H2/H∞ linear matrix inequalities (LMIs) is designed for the excitation system in a “one machine-infinite bus system” (OMIBS) to enhance system stability. Initially, the direct feedback linearization method is used to establish a mathematical model of the OMIBS incorporating uncertainties. ONRSMC is then designed for this model, employing the mixed H2/H∞ LMIs. The chaos mapping-based adaptive salp swarm algorithm (CASSA) is introduced to fully optimize the parameters of the sliding mode control, ensuring optimal performance under a specified condition. CASSA demonstrates rapid convergence and reduced likelihood of falling into local optima during optimization. Finally, ONRSMC is obtained through inverse transformation, exhibiting the advantages of simple structure, high reliability, and independence from the accuracy of system models. Four simulation scenarios are employed to validate the effectiveness and robustness of ONRSMC, including mechanical power variation, generator three-phase short circuit, transmission line short circuit, and generator parameter uncertainty. The results indicate that ONRSMC achieves optimal dynamic performance in various operating conditions, facilitating the stable operation of power systems following faults.