"I don't mind saying that," Kelly told ESPN's Chris Berman on Monday Night Countdown.
"The consistency, every single year he does it. And he doesn't have, all time, the greatest players around him. A lot of times they are in and out -- [Wes] Welker is gone, a couple other guys are gone. He does have a great one in [Rob] Gronkowski, but the thing is he makes it work and it doesn't matter what it is."
That has led Kelly, who played for the Buffalo Bills from 1986-1996, to lightheartedly discuss with Brady how much longer he wants to play.
"I continue to tell him year after year, it's time for you to retire. C'mon Tom, retire bud," Kelly joked. "I say it all the time." Brady has 26 wins against Kelly's beloved Bills, which ties the record for most QB wins against one team (Brett Favre had 26 against the Detroit Lions).
Kelly has relayed to the 39-year-old Brady, now in his 17th NFL season, how much he admires what he's accomplishing at this point in his career.
"I talk to Tom, we communicate all the time. I spoke to him before the game [Sunday] and of course after the game and I said, 'How do you do it all the time?' He said, 'The continuity. This coaching staff has been here for years.' He said [it is] the same system he's been under for almost his entire career.
"I was with the Bills for 11 and we had the same system for maybe nine out of 11. It makes such a big difference that you're not going from coordinator to coordinator. You have one system and everybody buys in. Even though they have guys that come in and out [in New England], receiver-wise, they still get it done.
"And Tom doesn't take that many hits, either. He's in that pocket, and if you're going to beat a Tom Brady, you have to pressure him up the middle, making sure he doesn't step up in the pocket. If you don't get any middle pressure, you're in for a long, long day."