Hoyer, who is the starter heading into training camp, is headed into the final year of his deal, which is scheduled to pay him $1 million this season.
His agent Joe Linta confirmed the sourced report by ESPN NFL Insider Adam Schefter that negotiations have started.
"It will be a very difficult deal to do," Linta told NFL.com. "We're always open to talking with the Browns, but we're content to wait and see what happens down the road."
Coach Mike Pettine told reporters on June 13 that Hoyer was "securely ahead" of first-round draft pick Johnny Manziel on the team's depth chart but that he didn't think the gap was insurmountable.
Hoyer, who has been a backup for most of his five NFL seasons, was 3-0 as a starter for the Browns last season before tearing his ACL.
Browns training camp begins July 26 in Berea, Ohio.