We aim for detection and recognition of planar objects in natural outdoor scenes under varying illumination conditions. To achieve this, we propose Relative Color Polygons (RCPs) using component colors of objects for color matching. They can be defined on many color spaces, and it is found that a 2D color space (XY space) is the optimal color space for our relative color method compared with other color spaces. To evaluate the invariance to illumination changes for object recognition, experiments have been carried out using 500 outdoor scene images. By using the proposed model, the color matching rate of the input images with the standard one was 95%. This framework is potentially applicable to image retrieval, image segmentation, image recognition, and so on.