Recent developments in agent-based systems provide an effective solution to many operational problems of power systems. This paper proposes a protection scheme that uses agent-based relays in a cooperative multiagent structure. The relay primarily starts as an overcurrent relay to detect the fault. Then, it starts data exchange with one peer relay in case of line faults or few neighboring relays in case of busbar faults to confirm the fault location. At this later step, it acts as a differential relay that compares the current phasors at both line ends in the case of line faults and computes the net outgoing current in the case of busbar faults. The scheme design is presented, and the agent cooperation protocol is described. To enforce the scheme against false data injected by hackers via intruding communication facilities, an anomaly detection device is prepared and integrated into each agent. The proposed tool is based on a one-class support vector machine and can firmly discriminate real fault data from injected false data. The tool also enables the relay to recognize the challenging high impedance fault. The proposed method is tested by dynamic simulation on the IEEE 9- and 39-bus systems under various conditions. The performance is evaluated by comparing it with that of other recent techniques.