Harrison, a former teammate of Brady's and current NBC Sports football analyst, told WEEI 93.7 FM in Boston that it "hurt" him to see the Patriots struggle as badly as they did in Monday night's 41-14 loss to the Kansas City Chiefs.
The Patriots' offensive line played musical chairs in the loss as left tackle Nate Solder, right tackle Sebastian Vollmer and others were swapped out and moved around at various points during the game. Brady was sacked twice, threw two interceptions and was unable to establish himself in the pocket.
To Harrison, the problem with the Patriots' offense isn't as much Brady as it is those who are protecting him and the players he is throwing to.
"Tom Brady can still play," Harrison said. "But when you surround him -- there's a reason why Brandon LaFell was let go [by Carolina]. He's not a great player. He's a young guy, and he has to make his way in this league.
"And Danny Amendola, you look at him, no one ever said he was a great player. He's always been hurt. The history is behind it. Rob Gronkowski obviously coming off that ACL injury, he's been hurt. So it's not like when you look at the Patriots on paper they just have all these weapons and teams are afraid of them."
Patriots coach Bill Belichick pulled Brady after his second interception was returned for a touchdown by Chiefs safety Husain Abdullah with 10:34 left in the fourth quarter. Rookie quarterback Jimmy Garoppolo took over in his first NFL regular-season action and drove the Patriots 81 yards for a touchdown.
"Whether Bill pulled [Brady] because he's afraid he was going to get injured, or Bill was trying to send him a non-subtle message, as a player your mind kind of starts playing tricks on you and you start thinking like, 'What's going on?'" Harrison said. "But at the end of the day, Tom needs to play better. The offensive line needs to protect him, but Tom -- we've said it week in and week out -- he's missing opportunities that are there; he's just floating the ball in the air."
Brady looks frustrated to Harrison, who sees trouble ahead for the Patriots (2-2) in their next matchup against the Cincinnati Bengals on Sunday night.
"I think it's one of those situations where Brady, he's really, really frustrated," Harrison said. "He doesn't have any confidence in his offensive line. And unfortunately, they're meeting one of the most complete teams in the National Football League."