The 37-year-old Spaniard announced earlier that this would be his last tournament because he didn't think he was fit enough to keep competing at a high level on tour.
Ferrer has 733 wins in 1,111 matches, fourth among active players.
Visibly overcome by his emotions, Ferrer had a difficult time serving while facing match point, backing off a few times and asking for a towel to wipe his face.
After losing he received a standing ovation from the crowd at the packed "Magic Box" center court, then went back to leave his headband on the ground near the net.
Ferrer spoke for several minutes in a ceremony to honor him and thanked everyone who helped his career.
"I'll never forget this day," he said with his wife and son by his side. "I've been very lucky. I've always wanted to end my career like this. I couldn't keep playing at the level that I wanted, but I'm very happy and very proud of my career."
Messages from other players were shown on the screens during the ceremony, with most praising Ferrer's toughness and "never-give-up attitude" on the court.
"All the players are very sad he is retiring," said Zverev, who appeared to be moved. "You always left everything on the court, in every single match. It's a privilege for me to be here playing against you in your last match."
Ranked 144th entering the Madrid tournament, Ferrer got off to a good start against Zverev, opening a 4-1 lead in the first set. But the defending Madrid Open champion rallied to win the next five games to close out the set and eventually the match.
Ferrer lost to Nadal in the 2013 French Open final, the only time he made it to a Grand Slam title match. Ferrer was No. 3 in the world that year, his highest ranking. He was among the top 10 for seven seasons. He won three Davis Cup titles with Spain.