With the prevalence of renewable distributed energy resources (DERs) such as photovoltaics (PVs), modern active distribution networks (ADNs) suffer from voltage deviation and power quality issues. However, traditional voltage control methods often face a trade-off between efficiency and effectiveness, and rarely ensure robust voltage safety under typical state perturbations in practical distribution grids. In this paper, a robust model-free voltage regulation approach is proposed which simultaneously takes security and robustness into account. In this context, the voltage control problem is formulated as a constrained Markov decision process (CMDP). A safety-augmented multi-agent deep deterministic policy gradient (MADDPG) algorithm is the trained to enable real-time collaborative optimization of ADNs, aiming to maintain nodal voltages within safe operational limits while minimizing total line losses. Moreover, a robust regulation loss is introduced to ensure reliable performance under various state perturbations in practical voltage controls. The proposed regulation algorithm effectively balance efficiency, safety, and robustness, and also demonstrates potential for generalizing these characteristics to other applications. Numerical studies validate the robustness of the proposed method under varying state perturbations on the IEEE test cases and the optimal integrated control performance when compared to other benchmarks.