Nolan replaces Paul Pasqualoni, who was fired after the Dolphins' 7-9 season. They allowed 390 points, the third-highest total in the AFC.
Bill Parcells, the Dolphins' football czar, had the same job with the Jets when Nolan worked there.
Nolan became available after the Dolphins' courtships of Al Groh and Keith Butler failed to produce an agreement. Groh became defensive coordinator for the Georgia Tech Yellow Jackets, and Butler decided to remain with the Pittsburgh Steelers as linebackers coach.
Miami coach Tony Sparano favors a 3-4 defense, and Nolan converted the Broncos last year to that scheme. Denver improved to seventh in the NFL in defense from 29th in 2008, but the team faded badly and missed the postseason for a fourth consecutive year
In their final four games, the Broncos gave up 122 points. Still, Nolan's departure was a surprise, and the reason for the split with coach Josh McDaniels was unknown.
Miami allowed a franchise-record 140 points in the fourth quarter, most in the NFL. And in the final three games, desperate to stay in the playoff race, the Dolphins fell behind 24-6, 27-0 and 27-10.
Now that the Dolphins have a new defensive coordinator, they can intensify their shopping for linebackers and a free safety. Linebackers Jason Taylor and Joey Porter and nose tackle Jason Ferguson, all in their 30s, are among those who might not be back in 2010.
Nolan was the 49ers' head coach in 2005-08. He spent one year with Denver and has been a defensive coordinator with four other NFL teams: Baltimore (2002-04), the New York Jets (2000), Washington (1997-99) and the New York Giants (1993-96).
The Dolphins made another move Tuesday, hiring former Giants assistant coach Bill Sheridan as the inside linebackers coach.
Sheridan was fired two weeks ago as the Giants' defensive coordinator. In Miami, he replaces George Edwards, who left after five seasons to become the Florida Gators' defensive coordinator.
Sheridan coached the Giants' linebackers in 2005-08 and was their defensive coordinator in 2009. The Giants gave up 427 points this season, third-worst in the NFL.
Information from The Associated Press was used in this report.