How to choose an appropriate fault diagnosis method with optimal comprehensive performance is a very meaningful question, which, if answered, could significantly raise the accuracy of fault diagnosis and therefore enhance the security of a power system. To help answer the question, an index system and evaluation method based on diagnostic performance of different fault diagnosis methods are proposed. First, the basic processes of four main fault diagnosis methods are introduced. Then, a quantitative index system of diagnostic performance is put forward. Overall dominance degrees and scores of fault diagnosis methods based on the quantitative index system can be calculated with index dominance degrees and Choquet integration. Without the assumptions that the indices are independent, the method considers the relationship between indices based on the index values of fault diagnosis models and dominant relationships between two schemes, and thus successfully achieves a more credible unified comparison among various methods. The examples show the effectiveness and correctness of the index system and evaluation method. This work is supported by the Science and Technology Project of State Grid Corporation of China “Research on Key Technologies of Power Grid Fault Diagnosis Based on Artificial Intelligence” (SGNC0000DKJS1900168).