Surface-enhanced Raman scattering (SERS)-based sensors utilize the electromagnetic-field enhancement of plasmonic substrates with the chemical specificity of vibrational Raman spectroscopy to identify trace amounts of a wide variety of different target analytes while being minimally affected by photobleaching. However, despite many advantageous features of this method, SERS sensors, particularly for detecting hazardous chemicals, suffer from several limitations such as requirement of gigantic signal enhancement that is often poorly controllable, subtle change and degradation of the SERS substrate, consecutive fluctuation of the signal, the lack of reliable receptors for capturing targets of interest and the absence of general principles for detecting various chemicals in different phases and matrices. To overcome these limitations and for SERS sensors to find practical use, one must (1) acknowledge the characteristics of the matrices of target systems, (2) finely engineer and tune the receptors of the SERS sensor to properly extract the target analyte from the phase, and (3) implement additional mechanistic modifications to enhance the plasmonic signal. This minireview underlines the difficulties associated with different phases and a wide range of target analytes, and introduces the practical measures undertaken to overcome the respective difficulties in SERS-based detection of hazardous chemicals.