An intelligent railways safety system is very essential to avoid the accidents. The motivation behind the problem is the large number of collisions between trains and various obstacles, resulting in reduced safety and high costs. Continuous research is being carried out by distinct researchers to ensure railway safety and to reduce accident rates. In this paper, a novel method is proposed for identifying objects (obstacles) on the railway tracks in front of a moving train using a thermal camera. This approach presents a novel way of detecting the railway tracks as well as a deep network based method to recognize obstacles on the track. A pre-trained network is used that provides the model understanding of real world objects and enables deep learning classifiers for obstacle identification. The validation data is acquired by thermal imaging using night vision IR camera. In this work, the Faster R-CNN is used that efficiently recognize obstacles on the railway tracks. This process can be a great help for railways to reduce accidents and financial burdens. The result shows that the proposed method has good accuracy of approximately 83% which helps to enhance the railway safety.