Thanks to high-speed Internet access and feature-rich mobile devices, the demand for ubiquitous and secure surveillance systems has increased. In this paper, we propose a privacy-protected video surveillance system that makes use of scalable video coding (SVC). SVC can be used to fulfill the requirement of omnipresence. Further, to address privacy concerns, we detect face regions and subsequently scramble these regions-of-interest (ROIs) in the compressed domain. To demonstrate the feasibility of the proposed video surveillance system, simulation results are provided. The results show that our system is able to provide a good level of security, while offering access to surveillance video content in heterogeneous usage environments.