Davis will be introduced as the Rebels' 22nd head coach on Monday. Ole Miss was 12-20 this season.
Davis, 58, has spent the past 16 years at Middle Tennessee and took the program to three NCAA tournament appearances.
"After assessing the entire landscape of college basketball and speaking to basketball experts across the country to determine the greatest fit for Ole Miss Basketball, the answer was clear -- Kermit Davis is the best coach and best leader to take our program to the next level in the SEC and NCAA," Ross Bjork, the school's vice chancellor for intercollegiate athletics, said in a statement.
"Coach Davis also possesses a great sense of purpose for his program and does it the right way off the court. During the process, Coach Davis' energy, drive, and vision for our program was unmatched, and we can't wait to welcome him and his family back to the great state of Mississippi and into the Ole Miss family."
The Blue Raiders won the Conference USA regular-season championship, but were left out of the NCAA tournament field after losing to Southern Miss in the C-USA tourney.
Davis' Middle Tennessee team pounded Ole Miss this season and was 4-1 against the Rebels since 2011-12.
Davis, a Mississippi native, will replace Andy Kennedy.
In 1990, Davis coached at Texas A&M, which is now part of the SEC. That year, the team was given two years' probation and a one-year postseason ban by the NCAA for recruiting violations under Davis. He resigned after one season and then took a job as an assistant coach at Chipola Junior College. He worked as an assistant at Utah State afterward, until landing his next head-coaching job, at Idaho.
He joined the Louisiana State University staff as an assistant coach after a year at Idaho and took over at Middle Tennessee in 2002.
ESPN's Jeff Goodman contributed to this report.