Ottawa won the best-of-seven series 4-1 and will move on for the first time since 2007, when they lost in the Stanley Cup finals to the Anaheim Ducks.
"Finding a way to win two games in this building feels great and so was being able to finish it off on our first chance," Senators captain Daniel Alfredsson said. "Andy made some great saves and we took over more and more as the game went on."
Anderson, who posted heady numbers in the regular season, was just as sharp in the playoffs, holding the usually high-scoring Canadiens to nine goals in five games. His own supposedly low-scoring team piled on 20 against Montreal starting goalie Carey Price and Budaj.
"For our team, Craig Anderson obviously was the MVP of this playoff series," said Ottawa coach Paul MacLean. "He was outstanding in every game.
"He gives us a chance to bend but not break, to recover, get our bearings. And once we do that, we're able to establish our game."
He got no argument from his Montreal counterpart, Michel Therrien.
"He was the player of the series," Therrien said.
Anderson made the difference right off the bat in this one.
Injury-wracked Montreal came out quickly, but Anderson shot out his glove for a brilliant save on Rene Bourque in the opening moments.
At the other end, a clearly nervous Budaj, making his first career playoff start, coughed up a huge rebound on Ottawa's first chance. Zack Smith put it in only 2:17 into the game.
"I think everyone took a good deep breath after that and said, 'We can start playing now,'" Anderrson said of the save. "They came out hard.
"My job is to give the team a chance to win no matter if it's the first minute or the last."
Therrien avoided direct reference to Budaj's play, other than: "We came to play and we worked hard, but every time. ... (the Senators) capitalized on their chances, let's put it that way."
The first playoff meeting in the modern era between the teams only two hours apart on highway 417 featured some nasty moments and name calling. But at the end only kind words were voiced, although the coaches exchanged only the briefest of handshakes on the ice.
The series opener saw Montreal center Lars Eller carried off on a stretcher with the Bell Centre ice covered in blood from a hit by Eric Gryba. MacLean's defense of his defenseman prompted Canadiens tough guy Brandon Prust to call him a "bug-eyed, fat walrus."
Game 3 disintegrated into a full-scale brawl and more verbal exchanges.
None of it seemed to matter when it was over.
"I thought the whole series was fun," said MacLean. "I didn't think it was bitter at any time. At the end of the series it's just 'good job,' 'good job.' I congratulate Michel on his team."
However, he could not hide his glee with what his team accomplished.
"We just beat the Montreal Canadiens in the Stanley Cup playoffs," he said. "I mean, to me that's huge. I'm going to call my mom and say we finally beat her team."
The victory gives Ottawa time to rest up and heal bruises before the conference semifinals begin next week. No other Eastern Conference series is over.
Cory Conacher had two goals while Alfredsson, Kyle Turris, and Erik Condra also scored for Ottawa, the seventh seed in the East but bolstered down the stretch by the return of top defensemen Erik Karlsson and Jared Cowen.
P.K. Subban scored for Montreal, which had five players injured in the series, including starting goalie Price. Budaj made 23 saves in his place.
After Smith's goal, another puck slipped behind Budaj and off a post, where Conacher tapped it in.
Montreal got one back on its first power play with 15 seconds to go in the period, as Subban sent a low point shot to the far corner.
The Bell Centre crowd booed as Turris gave Ottawa a 3-1 lead while shorthanded midway in the second period. The Senators center was pushed into Budaj by Tomas Plekanec and was on his hands and knees in the crease when Condra's shot went in off Turris' arm. The goal stood after video review.
It was quiet as could be at the 21,273-seat rink as the Senators scored on three straight power-play chances in the third period. Alfredsson wristed a shot in, Conacher beat Budaj from the left side, and Condra poked in a loose puck in the crease.
"Yeah we played hard and sometimes we played better than them, but the final score is we lost the series," said Montreal defenseman Andrei Markov. "We can only blame ourselves."
Montreal finished last in the conference last season and rose to second in the lockout-shortened 2013 campaign. The Canadiens looked to have recovered from a late-season funk going into the playoffs, but couldn't find enough goals to win.