But that has never really been the case.
Brady has performed consistently well against the Colts, with the exception of a dreadful four-interception game in 2006. His 97.5 passer rating in seven wins is not significantly higher than his 85.2 rating in four losses.
Compare that to Manning: Since 2000, he has a 106.6 passer rating in six wins against New England, and 71.6 in eight losses.
The mental battle between Patriots coach Bill Belichick and Manning truly defines this rivalry.
And it's a battle the Colts quarterback has dominated of late.
After Belichick took the helm in 2000, the Patriots won seven of eight games against the Colts. For Manning, it was a stretch that included 10 interceptions over the course of three games in Foxborough, a pair of playoff defeats, and complaints about both the Patriots' physical defense and his team's own play calling.
It culminated in the 2004 playoffs, a game that seemingly set up perfectly for Manning.
He was playing at an unprecedented level, coming off an NFL-record 49 touchdown passes in the regular season. Perhaps more significantly, the Patriots' defense was decimated, playing without Richard Seymour and both starting cornerbacks (Ty Law and Tyrone Poole). Among others, the secondary would include an undrafted rookie (Randall Gay) and a 33-year-old wide receiver (Troy Brown).
Not only did the Colts lose in the snow, but the makeshift Patriots defense held Manning without a touchdown. An offense that average 32.6 points per game managed just a field goal in a 20-3 loss. Manning brought only three drives inside the New England 40-yard line.
"It was just the best game plan that we've had since I've been here," Patriots safety Rodney Harrison said after the game.
Manning was visibly dejected, later saying he felt "responsible and accountable and so disappointed I couldn't do my job better to help my team win."
Two opinions seemed unanimous after the game. Belichick was a defensive genius, and the Patriots were in Manning's head. It was reasonable to ask if Manning would ever overcome New England.
Less than a year later, he threw for three touchdowns in a 40-21 drubbing of the Patriots at Gillette Stadium.
Since starting 1-7 against Belichick's Patriots, Manning and the Colts have won five of the past six meetings. That includes two straight wins in Foxborough, where he was previously 0-7.
Manning finally got the postseason monkey off his back in 2006, embarrassing the Patriots along the way to a title. The Colts became just the fifth team in postseason history to overcome a halftime deficit of 15 points or more.
And if anyone doubted Manning's momentum in the rivalry, last season's meeting made it clear. The Patriots blew a 17-point fourth-quarter lead. Before the loss, New England was 203-0 in franchise history with a 10-point lead going into the fourth quarter. Even worse, Belichick's much-debated fourth-and-2 call revealed the defensive genius' lack of faith in his own defense.
For years, the Patriots wouldn't even let Manning have a lead. Now, no lead appears safe.
Oh, how the tides have turned.
So what happened? How did Manning get Belichick out of his head and gain the upper hand? Let's take a look at what changed.
A key part of the Patriots' strategy against Manning was to take away the deep ball in the middle of the field. From 2000-2004 against New England, Manning had a 76.4 passer rating on balls thrown in the middle of the field, according to STATS LLC. Against the rest of the NFL, his rating was 109.0 on those throws.
The solution? Manning stopped forcing it. From 2000-2004, Manning averaged 9.1 attempts down the middle against the Patriots. He has since more than cut that in half to just 4.3 per game.
At the same time, he found success down the right sideline. In his career, Manning has thrown 95 touchdown passes in that direction, more than any other part of the field. Since Marvin Harrison roamed that side of the field, it should come as little surprise.
However, that's also where you could find Ty Law during Belichick's first five seasons in New England. From 2000-2004, Manning had a 53.8 passer rating down the right sideline with just two touchdowns and five interceptions, according to STATS LLC.
It's been a different story since Law left town; seven of Manning's 13 touchdown passes against the Patriots have been to the right sideline. He has a 120.7 passer rating on throws in that direction.
New England's defensive scheme also forced Manning to focus on the short passing game, another area where he's excelled.
From 2000-2004 against New England, Manning had a 78.6 passer rating with seven touchdowns and six interceptions on passes thrown 10 yards or fewer. Since then, he's thrown for nine touchdowns, just one interception and a 107.5 rating.
In his past three games, he is just 2-of-15 with no touchdowns and two interceptions on passes thrown over 20 yards. Compare that to the first six games of the season, when he threw five touchdowns and no interceptions for a 125.0 passer rating.
Meanwhile, Manning has made backup tight end Jacob Tamme the NFL's most targeted receiver thus far in November.
Much like Belichick did in that 20-3 playoff victory, Sunday's game is Manning's chance to assert his dominance regardless of the personnel at hand.
Having lost the mental advantage against the Colts, it's Belichick's opportunity to adjust and reaffirm his reputation as a defensive genius.
Jeremy Lundblad is a researcher with ESPN Stats & Information. He provides statistical analysis for ESPNBoston.com.