This paper designs a Passive Fractional-Order Sliding-Mode Controller (PFSMC) for PV inverter to achieve the Maximum Power Point Tracking (MPPT) under various operation conditions. At first, a storage function is constructed based on the tracking errors, in which the beneficial terms are retained to increase the tracking rate. Meanwhile, other system nonlinearities are fully compensated to realize a globally consistent control performance. Then, the Fractional-order Sliding-Mode Control (FSMC) is employed as the additional input to reshape the energy of the storage function, such that an improved dynamical response of the PV inverter could be realized. Meanwhile, the closed-loop system robustness can be significantly enhanced by the use of FSMC framework. Two case studies are undertaken, e.g. solar irradiation variation and power grid voltage drop, to verify the effectiveness of the proposed approach. Simulation results show that PFSMC can achieve the fastest dynamical responses and the lowest overshoot in comparison to that of conventional PID control, Passivity-Based Control (PBC), and FSMC under different operation conditions. Lastly, a dSpace based Hardware-In-the-Loop (HIL) test is carried out to validate the implementation feasibility of PFSMC. This work is supported by National Natural Science Foundation of China (No. 51777078) and Science and Technology Project of Guangdong Power Grid Co., Ltd. (No. 031900KK52170132).