Moss, who retired last month, was playing golf at Bungay Brook Golf Club in Bellingham, Mass., on Thursday and was asked if there were a chance he'd be on the field in Foxborough.
"No, I'm done," he said, according to a report on sports radio WEEI's website.
Moss would not elaborate.
"I'm just here golfing, minding my business," he said, according to the report.
Brady said a little more than a week ago that he was open to a reunion.
"Would it surprise me? You never know in the NFL," Brady told WEEI then. "I talked to Randy just before the start of training camp. I love Randy. He's one of the greatest wide receivers in the history of the game, one of the greater players I've ever had the chance to play with.
"Unfortunately, I don't make personnel decisions. If I did, I'd still have all my friends here -- Troy (Brown), David Patten, Deion (Branch) never would have left. I've seen so many guys move on, then guys come back. Who knows? You never know what situation we'll be in here next week, for that matter."
Moss played 13 seasons with Minnesota, Oakland, New England and briefly Tennessee. The six-time Pro Bowler had arguably his best season as a pro with the Patriots in 2007, catching 98 passes for 1,493 yards and a league-leading 23 touchdowns.
The Patriots did add a veteran receiver during training camp, signing former Bengal Chad Ochocinco. But he had only one catch for 14 yards in the Pats' opener and drew the ire of former New England linebacker and current ESPN analyst Tedy Bruschi for tweeting that he had never seen anything like the Patriots' offense.
Bruschi responded on WEEI by saying that Ochocinco should "drop the awe factor" and "stop tweeting and get in your playbook."
Information from ESPNBoston.com's Chris Forsberg was used in this report.