Rybarikova, ranked 68th in the world, broke the serve of the world's second-ranked player six times. The final break put her ahead 4-2 in the third set and from there she served out the second-round win.
"When I was mixing my game with the slice and I was serving good, I think she struggled with this kind of game," Rybarikova said.
The 25-year-old needed just 28 minutes to take the first set, winning the final three games in convincing fashion.
Halep fought back to take a 4-0 lead in the second set and hung on to force the third.
The Romanian had hoped to get in a few hard-court matches in New Haven before next week's U.S. Open. But she was pleased with her forehand and not entirely disappointed after the loss.
"I played three sets and almost two hours, so I think it was a good match," Halep said. "I have enough positive things to take from this match."
Wimbledon champion Petra Kvitova had a much easier time in the night's final match, dispatching Russian Ekaterina Makarova 6-2, 6-1 in less than 49 minutes. The tournament's second seed dropped her first two games, before winning 12 of the next 13, including the final five.
She said that after an emotionally draining summer, she's trying to get motivated again for a run at another Grand Slam title.
"The tournaments after Wimbledon, it's difficult to play," she said "I've felt a little empty without the emotions to fight too much."
The lone American left in the tournament earned five service breaks, four while cruising through the opening set.
"I knew that I had to attack her second serve," Riske said. "She places her first serve very well, but I felt like I was going to get looks on it."
Pennetta, ranked No. 12 in the world, suffered from dizziness and saw a trainer and a doctor between sets. She considered withdrawing, but was able to bounce back from a 3-1 deficit to force the second-set tiebreaker.
"I was on the court and trying until the last point," she said. "Because when it's like this, you just wait and say, `Maybe at this moment it's going to pass and I'm going to feel better."
The 24-year-old Riske is currently ranked 44th in the world, but has been struggling on the U.S. hard courts. She came into the tournament on a 3-match losing streak before beating Australian Casey Dellacqua in the opening round.
"It just took me some time to feel kind of feel comfortable again, to get my groove," she said.
Barbora Zahlavova Strycova of the Czech Republic also moved into the quarterfinals with a 7-5, 6-2 win over Caroline Garcia of France, saving 12 of 13 break points along the way.
She broke the 20-year-old in the 11th game of the first set to go up 6-5 and cruised in the second, taking five of the first six games. The 28-year-old said she took advantage of being the more experienced player.
"I felt like when I got the break, she was doing a little bit of easy mistakes," Zahlavova Strycova said.