Fault location algorithm of ultra high voltage Direct current (LCC-HVDC) transmission system plays an important role in the safe and stable operation of smart grid. Aiming at the shortcomings of low accuracy and poor rapidity of fault location methods for long distance UHVDC transmission systems, a double terminal traveling wave fault location method based on variational mode decomposition (VMD) and Teager energy operator (VMD) is proposed. Firstly, the propagation characteristics of fault voltage traveling wave in LCC-HVDC lines are studied. Based on the characteristic that zero-mode voltage attenuates obviously with line propagation, the time-frequency characteristics of zero-mode voltage traveling wave at sampling point are extracted by VMD algorithm. In order to solve the mode aliasing problem caused by the improper selection of VMD parameters, K-L divergence was used to optimize the extracted modal indexes. Then, TEO was used to extract the instantaneous energy spectrum of the decomposed signal, and the arrival time of the wave head was calibrated accurately. Finally, the fault distance was solved iteratively by the two-terminal iterative fault location method. The ±800kV LCC-HVDC simulation model was established in PSCAD/EMTDC for verification. The results show that the proposed method has strong robustness under different fault locations, transition resistance and fault types.