Musselwhite was born in the rural hill country of Mississippi. He has said that he is of Choctaw descent, and he was born in a region originally inhabited by the Choctaw. However, in a 2005 interview, he said his mother had told him he was actually Cherokee. His family considered it normal to play music, with his father playing guitar and harmonica, his mother playing piano, and a relative who was a one-man band. At the age of three, Musselwhite moved to Memphis, Tennessee.