Williams, 33, had fought back in each of her first two matches to advance. She seemed to follow the same game plan in this one, rallying from a set down to win the second set and lead 2-1 in the third.
That's when the 20-year-old Canadian took over, breaking Williams' serve twice down the stretch, including the final game.
"She played a lot more consistently than I did," Williams said. "I think my errors really hurt me a lot today. Just a lot of up and down, a lot of errors."
Williams' defeat was part of a difficult day for past champions of the clay-court event. Along with 2004 winner Williams, two other previous Family Circle champs -- Sabine Lisicki (2009) and Samantha Stosur (2010) -- lost third-round matches.
The only previous winner of the five entered (two-time defending champion Serena Williams was ousted Tuesday night) -- still around in the quarterfinals is No. 2 seed Jelena Jankovic, the 2007 winner here, who defeated Croatia's Ajla Tomljanovic 7-5, 6-1.
Bouchard finished off Venus Williams by reaching her opponent's drop shot and pushing it into the open court for her first victory over the seven-time Grand Slam champion.
"I think I've shown to myself how mentally strong I can be," said Bouchard, seeded sixth at the Family Circle. "Today was really a mental battle. It was a bit ugly at times and tough on the court, but I fought through it."
Especially in two key stretches in which it looked as if Williams was ready to take control.
Bouchard was ahead 5-3 in the opening set when Williams rallied and held two set points in the tiebreaker. But Bouchard won the next four points, including a crisp backhand for a winner to secure the set.
When Williams moved on top in the third set, Bouchard did not panic but dug in to move on to her third quarterfinal round this year.
"I've just kind of raised my game a little bit and was extra solid on those important points," Bouchard said.
Bouchard will next face Jankovic, who at No. 8 in the world is the highest-ranked player remaining in the tournament.
Jankovic has reached the Family Circle quarterfinals six times since her championship run seven years ago. She was beaten by a Williams sister each of the past two years, falling to Venus in 2012 and to Serena in last year's championship finals.
Jankovic struggled early as Tomljanovic, 20, fought off a set point to tie the opening set at 5-all. Jankovic then won eight of the next nine games to move forward.
"I missed that overhead on my set point, which was kind of frustrating, and we leveled it at 5-5," Jankovic said. "But I was able to stay calm and regroup and finish that set. So that was very important."
Venus Williams' loss ends a run of recent success for her family on theses courts. Sister Serena was the two-time defending tournament champion before losing Tuesday night. Venus played with the Washington Kastles on the green clay to win the 2012 World Team Tennis title.
Venus Williams had been bothered by an illness that she said limited her endurance this week. She had to come from behind in her first two matches here to face Bouchard and did not have enough stamina to wait out her opponent's solid, flat ground strokes.
"That kind of made it more challenging. I mean I think normally I would probably be able to challenge the points a little bit longer and just make her play more shots and just be more willing to play longer points," said Williams, who reached the semifinals here last year before falling to sister Serena, the eventual champion.
Others advancing included third-seed Sara Errani, who outlasted Peng Shuai 7-6 (6), 7-6 (5). Errani will play Swiss qualifier Belinda Bencic, who topped Elina Svitolina of the Ukraine 6-7 (4), 6-4, 6-1.
Petkovic will face Safarova of the Czech Republic for a spot in the semifinals. For Safarova, the victory moved her to 9-2 all-time against 2011 U.S. Open champion Stosur.
No. 12 seed Daniela Hantuchova and Jana Cepelova will square off in the quarterfinals. Hantuchova defeated Teliana Pereira 6-2, 6-3, while Cepelova continued her surprising run with a 7-6 (4), 3-6, 6-3 victory over 13th-seeded Elena Vesnina.
Cepelova knocked off Serena Williams in the tournament's biggest stunner.