Hardhats are an important safety measure used to protect construction workers from accidents. However, accidents caused in ignorance of wearing hardhats still occur. In order to strengthen the supervision of construction workers to avoid accidents, automatic non-hardhat-use (NHU) detection technology can play an important role. Existing automatic methods of detecting hardhat avoidance are commonly limited to the detection of objects in near-field surveillance videos. This paper proposes the use of a high precision, high speed and widely applicable Faster R-CNN method to detect construction workers' NHU. To evaluate the performance of Faster R-CNN, more than 100,000 construction worker image frames were randomly selected from the far-field surveillance videos of 25 different construction sites over a period of more than a year. The research analyzed various visual conditions of the construction sites and classified image frames according to their visual conditions. The image frames were input into Faster R-CNN according to different visual categories. The experimental results demonstrate that the high precision, high recall and fast speed of the method can effectively detect construction workers' NHU in different construction site conditions, and can facilitate improved safety inspection and supervision.