He was born Peter Williams, but he was adopted soon after he was born. His adoptive parents named him James Whittaker. He chose his stage surname as a composite, made up of the B from Brown, his mother's maiden name, followed by his wife's maiden name, Owen.