"After spending considerable time with Jim over the last week, we think he will be an excellent fit with our team, coaching the quarterbacks and helping with our offense," Ravens coach John Harbaugh said in a statement.
"We believe he enhances our staff. Jim has a tremendous history coaching at the college and pro level, especially working with quarterbacks and providing help with offenses. The timing is right to add a quarterbacks coach after (offensive coordinator) Cam (Cameron) and Joe (Flacco) worked so closely and well together this year. It's the right step for us now."
Caldwell was the Colts' quarterbacks coach from 2002 to 2008, helping to guide Peyton Manning to three of his four MVP awards. Manning won his fourth MVP award while Caldwell was the team's head coach.
"I am really excited to work with Coach Harbaugh, Cam and the rest of the coaching staff," Caldwell said in the statement. "It's a great fit for me, and I'm happy they saw it that way. I can't wait to get started with the Ravens, an organization that from top to bottom is one of the NFL's best. I'm looking forward to coming to Baltimore."
Caldwell was fired by the Colts after a 2-14 season in Indianapolis during which the team was without Manning, who missed the entire year while recovering from neck surgery.
The Colts fired Caldwell after three seasons. The team went to the Super Bowl during Caldwell's first year, but this year locked up the No. 1 overall draft pick with a horrid performance that also cost team vice chairman Bill Polian and his son, general manager Chris, their jobs.
Information from The Associated Press was used in this report.