With Scott out, there is mutual interest between Brown and the Cavaliers for a possible reunion, according to multiple league sources.
Much of it may depend on Brown, who has told those close to him that he is not sure he's ready to return to coaching after being let go by the Los Angeles Lakers earlier this season. Brown, who has two years and more than $8 million left on his deal with the Lakers, has said he may take time off to be with his family.
The Cavs are expected to contact several possible candidates in the next few days.
Despite Brown being fired by owner Dan Gilbert after back-to-back 60-win seasons (2008-2010), sources told ESPN.com there were no grudges held by either side that would prevent a dialogue about a fresh start.
Cavs general manager Chris Grant declined to address a direct question about Brown's candidacy during a news conference Thursday. Grant worked alongside Brown as assistant general manager from 2005 to 2010, and the two maintained a good relationship.
"We have not started that process," Grant said. "We're looking for someone with proven success, looking for someone who has a strong defensive system, someone who is a teacher, grinder and a worker."
Brown, who reached the second round of the playoffs in each of his six full seasons as coach in Cleveland and Los Angeles, is known as one of the best defensive coaches in the league.
It is an open secret that Brown and his family are planning a move from the L.A. area back to Cleveland this summer. Brown's son, Elijah, signed a national letter of intent this week to play for Brad Stevens at Butler, and Brown wanted to be closer to see him play.
Historically, Gilbert has chased big names when looking for coaches. In 2005, when he hired Brown, he first interviewed Phil Jackson. In 2010, Gilbert offered the job to Michigan State coach Tom Izzo before hiring Scott.
Despite three consecutive losing seasons, the Cavs' coaching job is more attractive now than it was three years ago, after Brown was fired and LeBron James left in free agency. The roster is now stocked with young players, upcoming draft picks and tradable assets.
The Cavs could have between $15 million to $20 million in salary-cap space this summer, in addition to two draft picks in the top 20. They have a 15.6 percent chance of winning the lottery next month.
Scott was dismissed Thursday morning after the Cavaliers lost 16 of their final 18 games to finish at 24-58, the third-worst record in the NBA. The team blew several large leads in some games, including a 27-point home lead against the Miami Heat in the final weeks of the season.
"I wish Byron Scott and his entire family the best going forward," Gilbert said. "Byron is a class guy, both on and off the court, and I thank him for his three years of coaching the Cavaliers. I fully support the difficult move that was made today. Although we saw progress with young individual player development, we did not see the kind of progress we expected on the team level this past season.
"We understand it was challenging with the injuries, but when you are at our stage in the building process, you don't only measure team progress in wins and losses."
Scott's three-season tenure was marred by poor team defense and injuries to key players as the Cavs started a long-term rebuilding project in the wake of James' 2010 departure. The Cavs ranked 26th in defensive efficiency in each of the past two seasons, and was 29th in Scott's first season, in 2010-11.
During training camp, the Cavs picked up the option of Scott's contract for next season, which was valued at more than $4 million. He had a record of 64-166; his overall record as a coach is 416-521 with the Nets, New Orleans Hornets and Cavs.
"I have tremendous respect for Byron professionally and a great deal of admiration for him personally," Grant said. "At the same time, it is critical for where we are as a team to ensure that we capitalize on every opportunity for development and success, and we have fallen short of that on the court. I believe we needed to make this change in order to get to a better position to achieve our goals."
Information from The Associated Press was used in this report.