The deal is for five years and will pay Mendenhall $3.25 million a year, the school said.
"Bronco Mendenhall's teams have consistently won at a high level, and he's demonstrated the ability to create a strategic vision to build a program and then implement his plan to be successful," Virginia athletic director Craig Littlepage said in a statement. "His emphasis on the overall development of student athletes and a commitment to academic achievement is in line with our goals of uncompromised excellence. We're excited to begin a new era of Virginia football and support Bronco and his staff."
Mendenhall said in a statement that he was drawn to the "high standards both academically and athletically at Virginia."
Mendenhall went 99-42 in 11 seasons at BYU, good for 12th among all FBS coaches during that period. Over the past 11 years, the Cougars are one of only 11 programs to advance to a bowl game each season, including this season, winning six of those games. BYU won two Mountain West titles under Mendenhall before becoming independent in 2011.
"BYU has played the pivotal role in my professional and personal life, and I will be forever indebted to the outstanding young men and exceptional people I have had the opportunity to work with at BYU," Mendenhall said.
Mendenhall will coach BYU in the Las Vegas Bowl on Dec. 19. BYU said in a statement that it would immediately begin the search for a new head coach.
"I am extremely grateful for the opportunity I've had to work with Bronco as our head football coach," athletic director Tom Holmoe said. "He has worked tirelessly on behalf of our student-athletes to give them the best possible collegiate experience, on and off the field. Bronco has been loyal through and through to BYU. I'm grateful for everything he's done for the football program."
The Virginia job opened Sunday, when Mike London resigned as head coach after going 27-46 in six seasons with the team. Virginia went 4-8 this season.
"We expect our football program to compete for the Coastal Division title on an annual basis, which puts us in a position to win the ACC championship and be competitive nationally," Littlepage said at the time of London's resignation.
The Atlantic Division has won the past four ACC championship games. In the past week, including Virginia's hire of Mendenhall, three teams from the ACC's Coastal Division have hired new head coaches who all won at least nine games this season. The others are Virginia Tech, which hired Justin Fuente from Memphis, and Miami, which hired Mark Richt from Georgia.
Information from ESPN's Stats & Information was used in this report.