The 30-year-old American, who was seeded 23rd at Flushing Meadows, missed about 2½ months this season because of an accelerated heartbeat and had a medical procedure in May.
"We are not 100 percent sure what the issue is and if it is related to his previous issues," Fish's agent, John Tobias, wrote in an email to The Associated Press. "Mardy is fine and will return home to L.A. tomorrow. This was strictly precautionary and I anticipate that Mardy will play in Asia this fall."
Sources told ESPN.com's Greg Garber that Fish was on site at the Billie Jean King National Tennis Center being examined by doctors.
Fish's departure means 2003 champion Andy Roddick is the only American man left in the field. Roddick, who announced last week that he'll retire after the U.S. Open, plays 2009 champion Juan Martin del Potro in the fourth round Tuesday night.
Fish's third-round victory over Gilles Simon at the U.S. Open went five sets, lasting more than 3 hours and ending after 1 a.m. Sunday. Afterward, Fish did not attend a news conference; the tournament said he was getting medical treatment, but didn't elaborate.
In a statement released by the U.S. Tennis Association on Monday, Fish said he "was reluctant to" withdraw from the year's last Grand Slam tournament but was "following medical advisement." He added that he looks forward "to resuming my tournament schedule in the fall."
After losing a match at Key Biscayne, Fla., on March 29, Fish went to be checked by doctors because his heart started racing uncontrollably that night. He pulled out of the U.S. Davis Cup team's quarterfinal against France the following week.
In May, doctors induced extreme palpitations to try to pinpoint the problem in Fish's heart. He returned to the tour at Wimbledon in June.
The walkover allowed the top-seeded Federer to reach his 34th consecutive Grand Slam quarterfinal, extending his own record.
"I am really sorry for Mardy. I just want to wish him a speedy recovery," Federer said in a statement issued by the tournament. "We all want to see him back on tour soon."
After the walkover announcement, Fish's coach, Mark Knowles, wore a somber expression.
Asked if Fish was going to be all right, Knowles frowned.
"I really can't say," he said. "Sorry."
Federer, who has won five of his record 17 Grand Slam titles at the U.S. Open, can reach the semifinals by getting past No. 6 Tomas Berdych of the Czech Republic, who eliminated No. 11 Nicolas Almagro of Spain 7-6 (4), 6-4, 6-1.
Berdych stunned Federer in the 2010 Wimbledon quarterfinals on the way to reaching the final there.
"It will be a tough match against Tomas. We have played many times in the past and he has always been a tough opponent," said Federer, who has won 11 of 15 career head-to-head matchups. "I will have to continue to serve well and dictate the points."
"I know when he walked in the locker room (Sunday), he didn't look right," said Mike Bryan. "No one really knows much. It's probably the same thing. I heard it was acting up on him in the last match. And then the rumor was it was bothering him after the (third-round) match in the stadium. He stayed really late in the morning to get treatment.
"Anything that goes wrong with your heart is scary. It speeds up, slows down, stops, starts. It's not a good thing. He's really worried about it. I'm sure all his friends and family are, too."
The looks on their faces showed concern.
"He's just got to get past this," Mike said. "Hopefully it's just another hiccup. Hopefully he's past the tough part of this. U.S. Open playing best of five sets, it's probably not great for his condition."
Information from ESPN.com's Greg Garber and The Associated Press was used in this report.