"Cleveland is the only job I would consider," said Arians, who was in Cleveland with CBS as part of the broadcast team for Sunday's Kansas City Chiefs-Browns game.
Arians, 66, also touted his former boss with the Indianapolis Colts as a possibility for Browns general manager John Dorsey to consider.
"My guy would be Chuck Pagano," Arians told the newspaper.
Reached Monday by ESPN, Arians said he is enjoying retirement from coaching and enticing him to return would be a tough sell. He lives on a lake in Georgia, plays golf regularly and works as an analyst for CBS, all of which he enjoys.
"I don't know if anything will come of this, but the door is slightly cracked," he said.
Arians was Pagano's offensive coordinator with the Colts in 2012 and then took over as interim coach that season when Pagano left to undergo treatment for leukemia. He went 9-3 and became the first interim coach to win Coach of the Year. Pagano, 58, led the Colts to the playoffs in both 2013 and '14 but was fired after finishing 4-12 in 2017.
Arians joined the Cardinals in 2013 and won his second Coach of the Year award in 2014 after leading Arizona to an 11-5 record. The Cardinals were 13-3 in 2015 and reached the NFC Championship Game. Including the playoffs, he finished 50-32-1 as the Cardinals' coach.
Arians has previously been a coach with the Browns, serving as offensive coordinator from the 2001 to '04 seasons. He also interviewed for the Browns' head-coaching vacancy in 2013, but the team hired Rob Chudzinski instead.
Arians retired from coaching in January, citing his family as the reason for his decision.
Defensive coordinator Gregg Williams is the Browns' interim coach after the team fired Hue Jackson last week. The Browns are in last place in the AFC North at 2-6-1 but are likely to draw interest from several candidates due to the presence of rookie quarterback Baker Mayfield, taken No. 1 overall in this year's draft, and a strong defense that includes the 2017 No. 1 overall pick, defensive end Myles Garrett, and rookie cornerback Denzel Ward, taken No. 4 overall in 2018.
ESPN's Pat McManamon contributed to this report.