"We as a group come down here to race," the Hall of Fame driver told me Saturday as we stood outside the No. 43 hauler he made famous. "This should be on the social page, not the dadgum sports page.
"It takes away from the racing deal, the stars that really bring the people and the show that brings the people and the people that put on the show."
That might be about to change.
And, ironically, it's Patrick who can change it.
Based on her best lap from Saturday's practice, which was eight-tenths of a mile per hour faster than anybody else, she has a legitimate chance Sunday to win the pole for the Daytona 500.
If she does, the headlines won't read: "Girlfriend of Stenhouse wins Daytona 500 pole." They'll read: "Patrick becomes first woman to win Daytona 500 pole."
That would be huge for the sport in that -- at least for the moment -- it would return the conversation to racing and provide a boost for the diversity movement.
It would be huge for Patrick because it would force those who have spent the past few days making jokes about her social life to take her more seriously as a competitor.
"Well, I have always felt in my career that when things go well on the track, the media responds to it," said Patrick, fastest in the second practice and third in the first.
That kind of attention Petty would like to see.
"It would get national news," the seven-time Daytona 500 winner said. "If I won the pole, you'd have to go to the sport page. If she wins, it will be on CNN and everywhere."
Patrick, 30, has been everywhere lately because of her love life. Her relationship with Stenhouse, 25, overshadowed everything that happened during Thursday's media day. The relationship has overshadowed just about everything that has happened in NASCAR since Patrick went public with it last month.
More people have asked about their first date than how the new "Gen 6" handles. The sport needed a break from this. It needed the attention to return to the biggest event of the season, to the introduction of the new car that has manufacturers and fans excited.
It needed the jokes and this TMZ-type drama to end, no matter the number of Web hits, no matter the jump in television ratings.
"'Peyton Place' and the racetrack," Petty said. "I keep them separated."
That has been tough of late. But in the time it took Patrick to circle this 2.5-mile track four times -- twice in each practice -- the attention returned to what most of us came here for: the competition.
"Back in our day, if that were to happen, nobody but the racers would ever know," Petty said of personal relationships going public. "Now when the press puts it out there, the people are eager to pick it up.
"Back then, it was just gossip, and the general public didn't know about that stuff."
Not that Petty believes "that stuff" is bad for the sport. He had a public relations person when he competed who told him he "could take nothing and make something big out of it."
"He said, 'I don't care what you do, just don't kill nobody. I can get a good story out of any other damn thing you do,'" Petty said with a laugh.
There are aspects of the Patrick-Stenhouse relationship that fall into that category.
But the bigger story would be if Patrick wins the pole for the Daytona 500.
"She's magical," NASCAR team owner Rick Hendrick said. "You look at the Super Bowl [commercials], and she's all over the Super Bowl. She's a good little driver and she does a lot for our sport. It would really be special if she could be on the front row. It would be special. It never happened before, so that would be cool."
Then all extra attention Patrick gets for being a woman in a male-dominated sport would have some legitimacy.
She'd return to being a driver instead of a girlfriend.
And the sport would benefit in a way it needs to benefit.
"It would put us in a different arena because people will pick up on it that don't usually pick up on the race," Petty said. "From a PR standpoint, it will be great."