This paper presents a coordination strategy of Load Frequency Control (LFC) and digital frequency protection for an islanded microgrid (MG) considering high penetration of Renewable Energy Sources (RESs). In such MGs, the reduction in system inertia due to integration of large amount of RESs causes undesirable influence on MG frequency stability, leading to weakening of the MG. Furthermore, sudden load events, and short circuits caused large frequency fluctuations, which threaten the system security and could lead to complete blackouts as well as damages to the system equipment. Therefore, maintaining the dynamic security in MGs is one of the important challenges, which considered in this paper using a specific design and various data conversion stages of a digital over/under frequency relay (OUFR). The proposed relay will cover both under and over frequency conditions in coordination with LFC operation to protect the MG against high frequency variations. To prove the response of the proposed coordination strategy, a small MG was investigated for the simulation. The proposed coordination method has been tested considering load change, high integration of RESs. Moreover, the sensitivity analysis of the presented technique was examined by varying the penetration level of RESs and reducing the system inertia. The results reveal the effectiveness of the proposed coordination to maintain the power system frequency stability and security. In addition, the superiority of the OUFR has been approved in terms of accuracy and speed response during high disturbances.