Sitake, 40, played four years at BYU as a fullback and graduated in 2000. He spent the past year as Oregon State's defensive coordinator after serving as an assistant at Utah from 2005 to 2014.
"I'm honored, excited and humbled by the opportunity to lead the BYU football program," Sitake said in a statement. "I'm grateful for everything BYU gave me as a player. It's a dream come true for me to return home. I love the university and what it stands for, and I'm looking forward to helping the young men in the program reach their goals athletically, academically and spiritually.
"I want to thank Tom Holmoe, President (Kevin) Worthen and the administration for their faith and trust. I'm excited to get to work and continue the incredible legacy of BYU football."
Asked what he liked about Sitake, Holmoe cited his ability to recruit and "meld together defenses and our style," adding in a statement that Sitake is "an outstanding leader and coach ... and knows BYU through and through."
Oregon State struggled defensively this season under Sitake, ranking 116th out of 128 FBS teams in total defense.
Utah coach Kyle Whittingham had high praise for his former assistant, who he said was ready for a head-coaching gig.
"I think that's awesome for Kalani. ... I'm very excited for Kalani," he said. "He deserves it. He'll do a great job. I think he's a great hire. I'm proud of him. He was with me for 10 years, with us for 10 years, did a great job for us.
"He's going to do a fine job there. He's ready for this, in my opinion."
Sitake replaces Bronco Mendenhall, who was hired earlier this month by Virginia after spending the past 11 years at BYU. Mendenhall returned to coach the Cougars in the Royal Purple Las Vegas Bowl, which BYU lost 35-28 to No. 22 Utah on Saturday.
The Cougars finished the season 9-4, losing to their bitter rival in the first matchup between the two teams played at a neutral site.
Holmoe said he informed the team of Sitake's hire after the game.
"I had an opportunity to go around the room, and I talked to a lot of players, particularly those who are coming back, and individually they're very, very excited," Holmoe said.
Information from ESPN's Joe Schad, Adam Rittenberg and Chantel Jennings and The Associated Press was used in this report.