The traditional Multivariate Empirical Mode Decomposition (MEMD) to estimate the dominant oscillation modes from measurements may have estimation errors caused by inter-channel imbalances and correlations, and mode mixing is still not effectively resolved. This paper proposes a new method based on Adaptive-Projection Intrinsically Transformed Multivariate Empirical Mode Decomposition (APIT-MEMD) to estimate the dominant oscillation modes from the measurements in a power system. The Intrinsic Mode Functions (IMF) signals with different oscillation mode scales in each measurement channel are accurately separated by APIT-MEMD; the critical IMF associated with the dominant oscillation mode are determined by the Teager energy operator; the Hilbert-Huang Transform (HHT) is applied to identify the oscillation frequency and damping ratio of the dominant oscillation mode contained in each IMF. The method is analyzed and validated by the IEEE-68 bus test system and the PMU data collected from the Liaoning Power Grid. The results validate the feasibility and effectiveness of the proposed method. This work is supported by Science and Technology Project of State Grid Corporation of China (No. SGTYHT17- JS-199) and Science and Technology Project of State Grid Liaoning Electric Power Co., Ltd. (No. SGTYHT17JS201).