Carey Patton Jr. was the youngest of seven children born to Carey Patton Sr. and his wife, Gertrude. He came into a family deeply rooted in Macon County, North Carolina, and in military service to the United States. His ancestors migrated to Western North Carolina in the 19th century, settling on 15 acres off Mashburn White Road during the late 1800s. About 1935 or 1936, the family moved to Bidwell Street in the town of Franklin. Carey and his wife, Nancy currently live in the family home place on Mashburn White.
His father served in World War I, and four of his brothers served in the United States Navy. All his brothers were in naval electronics, involved in anti-submarine warfare. His oldest brother, Claybourne, (26 years older than Carey) was killed in an airplane crash off Long Island, N.Y., during World War II. The airplane was taking off to investigate a reported submarine sighting off the east coast of the United States. The pilot objected to the mission because of a terrible electrical storm in the area. Shortly after take-off, the airplane was struck by lightning and crashed into the harbor.