Mary Lou Retton Says She Was 'Considered the Fat One' at 4 Feet 9 and 94 lbs at Height of Gymnastics Fame (Exclusive)

Mary Lou Retton knows plenty about the pressure Olympic athletes are under while training for the Games.

 When she was just 12 years old, she left her home in Fairmont, West Virginia, and moved to Houston to live with a host family and train with infamous gymnastics coach Bela Karolyi, who had told her he could make her a star.

 Four years later, at the 1984 Olympics in Los Angeles, Retton's hard work paid off: She was the first American woman to take home the gold in the women’s all-around, and she became an overnight national sensation.

The PEOPLE Puzzler crossword is here! How quickly can you solve it? Play now!

“My coach Bela Karolyi looked at me and said, ‘Mary Lou, you need to give a 10,'" Retton, 56, tells PEOPLE of the moments before her big win. "He’d never said that before."

She said, “I was like, ‘You’re putting pressure on me? I’ll show you!’” She then delivered the most technically perfect vault she could muster.

“You can see on the video that I was smiling before my feet touched the floor,” she says, remembering how she stuck the landing. “The Pauley Pavilion was shaking with all the cheering,” she says. “They were all shouting, ‘Ten! Ten! Ten!’”

Retton became a star, but she says her fame and glory came at a price.

"I was starved," she admits, recalling how she was told she needed to look like her extremely thin Eastern European gymnastics counterparts. "I was considered the fat one."

In the decades since then, several top gymnasts have accused Karolyi and his wife Marta, both now retired, of being verbally and psychologically abusive.

"Verbally, we were not abusive,” Marta told Dateline NBC in 2018. “Emotionally, it depends on the person. You have to be a strong person to be able to handle the pressure.”

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from celebrity news to compelling human interest stories.

But Retton says it wasn't just her former coaches who considered her "a big girl." The media did, as well—and she says it hurt her deeply.

"I was considered a big girl. I was 4'9" and 94 pounds. A big girl! But that's what I was considered," she says. "I was considered 'the fat one.' And I wasn't fat in any way shape or form."

She adds, "It just hurts. It hurts to be called something that you're not."

"They'd say, 'You are heavy. This is not what gymnastics is supposed to look like.' They used to call me—who's the guy with the big thighs? Earl Campbell. They'd say, 'Mary Lou with the Earl Campbell thighs,'" she recalls of being likened to the former football running back. "It wasn't a compliment."

She says the scrutiny, fame, and the fact that she was just 16-years-old when all of this was happening, left her feeling lonely.

“It was so silly to think, ‘I’m going to win the Olympics and my life will go back to being normal,’” she says.

“That didn’t happen. My level of fame was crazy.” She recalls unveiling a new Wheaties box featuring basketball star Michael Jordan. “He told me, ‘Mary Lou, fame doesn’t change you. It changes everyone around you.’ I still felt like the same little hillbilly from West Virginia, but my God, they all changed. I felt very alone.” 

Still now, when she sees gymnasts who take pride in looking strong and muscular, she feels a sense of accomplishment.

"I feel like I started that," she says with a smile.

To read more about Mary Lou Retton, pick up this week's issue of PEOPLE on stands Friday.