SI detailed accusations made against Richardson that include sexual harassment of multiple women and the use of a racial slur toward a scout who has since left the team.
The NFL has taken over a workplace misconduct investigation into Richardson at the request of the Panthers, who announced Friday that they were looking into accusations made against the owner.
Richardson, 81, allegedly made verbal comments about women's appearances, inappropriately touched female employees and made advances to women that included asking whether he could shave their legs and for them to give him foot rubs.
Along with the allegation of using a racial slur that led to a settlement with the scout, SI notes comments made by Richardson about black players' appearances and his threat to discipline players who addressed social issues.
According to SI, Richardson and the Panthers reached confidential settlements with complainants that included nondisclosure and non-disparagement clauses. The SI report describes the settlements' value as "significant."
Neither the NFL nor the Panthers have revealed the nature of the investigation. The team declined to address specifics when contacted by SI, but Panthers spokesman Steven Drummond said Sunday that the team asked the NFL to handle the investigation for "transparency reasons."
The league will retain outside counsel to conduct its inquiry.
Members of the organization have been mostly quiet on the situation, though coach Ron Rivera said in a statement that he has "enormous respect for the man but will wait for the results of the investigation before making any judgment."
Panthers players were asked for their reaction to the report following Carolina's 31-24 victory over the Green Bay Packers. Tight end Greg Olsen said the players were given a briefing Saturday night that something might be coming soon regarding the allegations against Richardson.
Quarterback Cam Newton said the Panthers used the report as their "chip" in Sunday's game.
"That was pretty much our chip, being that information did get pointed out to us," he said. "People have been sitting on this story for a long time. And the fact that we're in the position right now, as well as not getting the notoriety of our team success up until this point, it's been about another individual."
Newton said Richardson "has been an unbelievable source in my life."
"I know, personally, for me, he's given me an opportunity to make a big impact in my family," he said.
A Panthers player told ESPN's Josina Anderson on Saturday that he had not encountered racial issues with Richardson, but he supports others for speaking up.
"I respect them for speaking up, no matter who it is," the player said. "... Equality is what we are going for."
Richardson was at Bank of America Stadium for the Panthers' home game against the Packers, but the windows to his end zone suite were not open, like they typically are by game time.
Richardson became the first former NFL player since George Halas to own an NFL team when he was awarded an expansion franchise on Oct. 26, 1993. The Panthers began play two years later in 1995.
Richardson has worked primarily behind the scenes during his tenure with the Panthers and rarely does interviews. He has been active through the years with league matters.
ESPN's David Newton and The Associated Press contributed to this report.