Shapovalov, 18, has taken major strides this week to reach his goal of cracking the world's top 100 with wins over Rogerio Dutra Silva and 2009 US Open champion Juan Martin del Potro. He also halted Nadal's bid to regain the No. 1 ranking, which the Spaniard could have achieved by reaching the semifinals.
Shapovalov battled back after Nadal cruised through the first set, then erased a 3-0 deficit in the tiebreaker for the biggest victory of his career.
Shapovalov became the youngest player to reach the tournament's quarterfinals since Bjorn Borg in 1974. With hockey great Wayne Gretzky and Olympic swimming star Penny Oleksiak cheering for him from the seats, Shapovalov also became the youngest quarterfinalist at a Masters Series tournament and the youngest to beat a player ranked in the top two in the world since Nadal beat Federer in 2004 in Miami.
"He played well," Nadal said. "He has a great potential.
"I wish him the best. He has everything to become a great player. He played with the right determination in the important moments."
Federer overcame a weak first set to post a 4-6, 6-4, 6-2 victory over Spaniard David Ferrer in an earlier third-round match.
In Friday's quarterfinals, the second-seeded Federer will face 12th-seeded Roberto Bautista Agut of Spain, who outlasted Frenchman Gael Monfils 4-6, 7-6 (5), 7-6 (2) on center court at Uniprix Stadium.
Unseeded Argentine Diego Schwartzman posted a strange win over American Jared Donaldson 0-6, 7-5, 7-5 to advance to a quarterfinal meeting with Robin Haase, the 52nd-ranked Dutchman who upset seventh-seeded Grigor Dimitrov of Bulgaria 7-6 (3), 4-6, 6-1.
Federer, who breezed past Canadian Peter Polansky in the second round on Wednesday, struggled in the opening set, spraying balls long, wide or into the net, but gradually rediscovered at least some of the form that has seen him earn two Grand Slam wins this year.
Federer is 17-0 in his career against Ferrer, who was ranked third in the world in 2013 but is now at No. 33.
Agut fought off a match point to force a tiebreaker, which the tired-looking Monfils opened with a double fault. Monfils never challenged again.
The unseeded Monfils played his third straight three-set match, including an upset win over fifth-seeded Kei Nishikori in the second round.
Schwartzman saved four match points to upset third-seeded Dominic Thiem in the second round. The win over Donaldson put him in a quarterfinal for the sixth time this year. Schwartzman's only ATP win was on clay in Istanbul last year.
Donaldson, 20, was seeking a first career quarterfinal in a Masters Series tournament, where he is 0-13 in round of 16 matches.