We introduce the “Hands-Free Video Phone System” that uses a visual sensor network. The system makes it easy for a user in his house to take a video phone call without disturbing his current activity. This system uses fisheyes lens cameras as visual sensors because they offer both sensing and image generation and are cheaper than other types of sensors for tracking people such as pressure sensors or infrared sensors. Its key advance is an algorithm that detects user location based on disparity maps of multiple stereo cameras, and selects the best shot camera based on criteria that consider optical properties. We describe a prototype system and a field trial that confirms the feasibility of the system in an experimental house.