Williams, who will take over the top spot in the rankings from Dinara Safina on Monday, looked sluggish in the first set before recovering in the second. Neither player was able to earn a break in the third set.
"I don't feel like I ever got into high gear," Williams said. "I ran into a girl today who has never played so good in her life."
Petrova, who finished with 14 aces, said her strong serve was the key.
Petrova will face Peng Shuai of China, who eliminated Maria Sharapova in straight sets on Wednesday.
Williams' departure marks the exit of all three of the tournament's highest-seeded players, including sister Venus and Safina.
Marion Bartoli of France also won, defeating Zhang Shuai of China 6-1, 6-4 to reach the quarterfinals. Also, Agnieszka Radwanska of Poland beat Maria Jose Martinez Sanchez of Spain 4-6, 6-4, 6-2, and Vera Zvonareva of Russia defeated Flavia Pennetta of Italy 6-3, 5-7, 7-5.
In the men's draw, Rafael Nadal beat James Blake of the United States 7-5, 6-7 (4), 6-3 to reach the quarterfinals. The second-ranked Spaniard had a match point in the second set, but he double-faulted and allowed Blake to break twice and force the tiebreaker.
In the third set, Nadal broke Blake in the eighth game and won when the American's return on match point hit the net.
"I played a very good match. I had the match under control," said Nadal, who saved five of six break points. "I managed to save very important points and when I had the chance I took it."