There are problems of insufficient types of dissolved gas fault features in transformer oil and a low accuracy of diagnosis. Thus a hybrid feature selection method is proposed. The improved optical microscope algorithm (IOMA) is used to optimize convolutional neural networks (CNN) to realize transformer fault diagnosis. First, a 30 dimensional transformer fault candidate feature set is constructed based on the correlation ratio method, and the hybrid feature selection method is used to determine the feature dimension of the input set through two feature selections. Secondly, a Tent chaotic mapping, adaptive t-distribution mutation and dynamic selection strategy are introduced to improve the optical microscope algorithm (OMA) and enhance its optimization performance. Then, the learning rate, the size and number of convolution kernels of the CNN model are optimized using the IOMA algorithm. Finally, the IOMA-CNN transformer fault diagnosis model is constructed and its performance is evaluated by numerical example analysis. Experiments show that the fault diagnosis accuracy of the proposed method is 98.5%. Compared with the conventional feature selection method, the fault diagnosis accuracy can be effectively improved by using the input features selected by the hybrid feature selection method. Compared with other optimized diagnosis models, IOMA-CNN has higher accuracy and better stability.