Baylor died of natural causes Monday and was surrounded by his wife, Elaine, and his daughter, Krystal, the Los Angeles Lakers said in a statement.
"Elgin was the love of my life and my best friend," Elaine said in the statement. "And like everyone else, I was in awe of his immense courage, dignity and the time he gave to all fans. At this time we ask that I and our family be allowed to mourn his passing in privacy."
Considered one of basketball's greatest players, Baylor was an 11-time All-Star and 10-time All-NBA selection during his 14 seasons with the Lakers from 1958 to 1971. He was the 1958-59 Rookie of the Year as well as the All-Star Game MVP that year. He averaged a double-double for his career, posting 27.4 points and 13.5 rebounds per game.
With the Lakers, who moved from Minneapolis to Los Angeles in 1960, Baylor made eight NBA Finals appearances but never won a title, losing seven-game series to the Boston Celtics on three occasions. He holds the single-game Finals scoring record with 61 points against the Celtics in 1962.
"Elgin was THE superstar of his era -- his many accolades speak to that," Lakers owner Jeanie Buss said in a statement. "He was one of the few Lakers players whose career spanned from Minneapolis to Los Angeles. But more importantly he was a man of great integrity, even serving his country as a U.S. Army reservist, often playing for the Lakers only during his weekend pass. He is one of the all-time Lakers greats with his No. 22 jersey retired in the rafters and his statue standing guard in front of STAPLES Center. He will always be part of the Lakers legacy. On behalf of the entire Lakers family, I'd like to send my thoughts, prayers and condolences to Elaine and the Baylor family."
Jerry West, who was a teammate of Baylor's on the Lakers and is now a Clippers consultant, said Baylor was like family to him.
"I will forever cherish my days spent with him as a teammate, he was one of the most gifted and special players that this game will ever see and he has never gotten his just due for what he accomplished on the court," West said in a statement. "My first few years in the league he cared for me like a father would a son, he nurtured me and encouraged me like no one else had during that period of my life. We shared the joy of winning and the heartbreaking losses during the championship finals. He was a prince both on and off of the court. There are no words to describe how I feel at this time. My deepest condolences to his dear wife Elaine and his loving family, and his many fans and friends. I loved him like a brother."
Baylor's scoring highlights also included becoming the first player to score 70 points in a game and going for 71 against the New York Knicks in November 1960. That stood as the Lakers' single-game record until Kobe Bryant scored 81 against the Toronto Raptors in January 2006.
Baylor retired early in the 1971-72 season because of knee issues. He just missed winning an NBA title, as the Lakers would go on to defeat the Knicks in five games in the Finals that season.
"Elgin Baylor set the course for the modern NBA as one of the league's first superstar players," NBA commissioner Adam Silver said in a statement.
"I loved him like a brother." Jerry West
After his playing career, Baylor coached the New Orleans Jazz for three seasons (1976-79) without a playoff appearance before stepping down.
He then found a long-term home in the Clippers' front office, serving as general manager from 1986 to 2009, though the team made only two playoff appearances during his tenure. He was named NBA Executive of the Year in 2005-06, when the Clippers won 47 games and made the Western Conference semifinals.
Baylor said he was forced out of his job because of age and racial discrimination, though he dropped racial discrimination claims against the Clippers as part of a lawsuit that was ultimately dismissed entirely in 2011.
Baylor served as GM under owner Donald Sterling, who was banned from the NBA for life in 2014 after private recordings of him making racist comments were made public. Baylor said that he worked for Sterling because of limited job opportunities for Black former players and the need to provide for his family.
In a statement, the Clippers called Baylor "a transcendent player, a beloved teammate, and a pioneering executive." The team held a moment of silence for him prior to Monday night's game at Staples Center.
Baylor was born in Washington, D.C., on Sept. 16, 1934. Named after his father's favorite watch, an "Elgin'' timepiece, Baylor became infatuated with his sport even before he could afford a basketball, instead learning to shoot with a tennis ball.
Baylor played at Seattle University from 1956 to 1958, averaging 31.3 points a game and leading the team to its only Final Four appearance in 1958, when the team lost to coach Adolph Rupp's Kentucky Wildcats in the title game. Baylor was named Most Outstanding Player.
After the season, the Minneapolis Lakers, who were near bankruptcy after a steep fall from their first championship era, picked Baylor first in the 1958 NBA draft. Baylor immediately saved the Lakers with his scoring and style.
It was actually the second time the Lakers had chosen Baylor in a draft. They picked him in the 14th round in 1956, but Baylor chose to continue playing in college at the time.
Baylor, who was inducted into the Naismith Memorial Basketball Hall of Fame in 1977, became the sixth person to be honored by the Lakers with a statue, which was unveiled in April 2018. In 1983, the team retired his No. 22 jersey.
The Associated Press contributed to this report.