The 44-year-old Muster returned to competitive tennis in 2010 after 11 years away but managed just two wins in 26 matches -- both on the lower-tier Challenger circuit. He said before the Vienna tournament that it would be his final event.
"I don't regret anything," said Muster, who fought back tears after the match. "My comeback was not about winning or losing, it was about the feeling of being able to compete at top level again."
Muster said he had "personal reasons" to cancel his initial plan of continuing to play in 2012.
"My family needs me more than tennis does," Muster said, but added that he might take part in a few Champions Tour events for retired players next year.
The 18-year-old Thiem, who reached the juniors' final at the French Open in June, had more power and pace in earning his first win on the ATP circuit. Muster broke to take a 3-1 lead in the second set but lost the next five games.
"The right player has won, Dominic handled the pressure well," Muster said. "And he can use these 20 ATP points more than I could."
Muster, who won the French Open in 1995 and held the No. 1 position for six weeks the following year, won 44 ATP titles.
He first played in the Vienna event in 1984 -- nine years before his opponent Thiem was born -- and reached the final three times but never won the title.
Tommy Haas rallied past Juan Ignacio Chela of Argentina 6-7 (3), 6-3, 7-6 (5). Haas, who won the event in 2001, failed to serve out the match at 5-3 in the third but converted his fourth match point in the tiebreaker.
Also, 2008 champion Philipp Petzschner of Germany routed Pablo Andujar 6-2, 6-2, Xavier Malisse of Belgium downed Albert Ramos of Spain 6-1, 6-4, and Daniel Brands of Germany defeated Martin Fischer of Austria 6-7 (3), 6-2, 7-6 (6).
Fischer, who is yet to win an ATP match this season, served for the match at 5-3 in the third set and lost a 5-4 lead in the tiebreaker.