The Swiss outsider punished Clijsters for her 11 double faults in the two-hour match and handed the Belgian her first loss since she won the Grand Slam tournament last month.
The second-seeded Clijsters struggled throughout with her service, landing only 58 percent of first serves.
Schnyder lost to Clijsters in the latter's first tournament in more than two years, at Cincinnati in August. But the tour veteran kept a cool head in the decisive tiebreaker to beat Clijsters for the first time since 2005.
The loss was Clijsters' second in 26 main draw matches at Luxembourg, where she was seeking a sixth title.
Schnyder, the former top-10 player ranked 47th, will play Sabine Lisicki of Germany in the quarterfinals.