Pegula, 29, had no answer to Mertens' versatile game. Mertens, a former world No. 1 in doubles, used her complete arsenal of shots, including sliced forehands, drop shots, lobs and deep groundstrokes, to move her opponent around and pin her back.
Mertens, ranked 28th, was near flawless at the start, breaking the American twice in a row and leading Pegula to toss her racket to the ground. Mertens raced to a 5-0 lead and claimed the first set in just 26 minutes.
"I feel like I was still playing good points," Pegula said. "Elise was just being really tough, not making a lot of errors and making me play every single ball. And with the windy conditions, I felt like it definitely played into her game."
Pegula, who had advanced to the third round after Camila Giorgi retired after the first set, broke Mertens early in the second, but the Belgian player responded in the next game.
Pegula earned another rare break opportunity at 3-2 but then sank three consecutive backhands into the net.
Mertens, 27, took advantage of Pegula's low first-serve percentage to break her again for a 5-3 lead, leaving the frustrated American to shake her head and look at her coaching box for any help.
Her ordeal was over after 82 minutes with her 28th unforced error.
Mertens will attempt to make it beyond the fourth round in Paris for the first time when she plays Sunday against 2021 French Open runner-up Anastasia Pavlyuchenkova, who got off to a slow start before finishing strong to down No. 24 Anastasia Potapova 4-6, 6-3, 6-0.
Stearns, a big hitter who defeated 2017 French Open champion Jelena Ostapenko in the second round, simply could not find her targets on a day with a breeze at about 10 mph and a chill in the low 60s Fahrenheit.
More than half of Kasatkina's 59 points came via the 30 unforced errors by Stearns.
Pegula had similar issues, making 28 unforced errors to only 13 by Mertens.
With Pegula joining No. 5 Caroline Garcia, No. 8 Maria Sakkari and No. 10 Petra Kvitova on the sideline, four of the top 10 women's seeds already are gone. That's part of a pattern this year at Roland Garros: Only 12 seeds made it through two rounds, the fewest in Paris since the field expanded to 32 seeds in 2002.
Sabalenka broke twice in the opening set and raced to a 3-0 lead in the second on a sunbathed Court Philippe Chatrier to set up a meeting with American Sloane Stephens or Yulia Putintseva of Kazakhstan. Sabalenka moved past the third round after losing at that stage in the past three French Open tournaments.
"It feels amazing to feel the support and just to be on this court and have the opportunity to play tennis," Sabalenka said after an encounter that lasted a little over an hour.
The Belarusian player then skipped the traditional postmatch news conference, citing mental health reasons and saying she did not feel safe while being asked about Russia's invasion of Ukraine following her win Wednesday.
In a rematch of last week's final in Strasbourg, Ukraine's Elina Svitolina once again got the better of Anna Blinkova, advancing to the fourth round after recovering from a set down to win 2-6, 6-2, 7-5 over the Russian player.
Svitolina, a three-time French Open quarterfinalist, next faces Kasatkina.
Out in singles, Pegula will now attempt to chase a title in doubles alongside 19-year-old Coco Gauff, whose third-round match in singles comes Saturday against 16-year-old Mirra Andreeva. A year ago, Gauff was the runner-up in singles at the French Open and made it to the final in doubles with Pegula.
"Obviously still a good chance to hopefully kind of go farther than we did last year," Pegula said about doubles, "so definitely I'm going to be focusing on that now."
Reuters and The Associated Press contributed to this report.