The series is tied 1-1. Game 3 is Monday at Washington.
Dan Boyle and Derick Brassard also scored and Henrik Lundqvist made 30 saves in the entertaining game that featured a highlight-reel goal by Alexander Ovechkin that made the final 10 minutes exciting.
Game 2 will be remembered for Kreider's early goal and Ovechkin's spectacular tally in the third period, when the NHL's top goal scorer got past top defensemen Dan Girardi and Ryan McDonagh and beat Lundqvist with a top-shelf shot as he was falling to the ice.
After killing off a late penalty, Washington pulled Holtby and came close to tying the game, but Lundqvist stopped Troy Brouwer in close in the final minute.
The Rangers were looking for a response after losing Game 1 on a goal by Joel Ward with 1.3 seconds left. They only needed 38 seconds to get it and send the sellout crowd at Madison Square into a frenzy.
It was like Game 1 against the Penguins when Brassard scored 28 seconds after the opening-round series started.
This time it was Kreider. He deflected a pass by defenseman Kevin Klein into the Washington zone and then went to the net and fired the rebound of Jesper Fast's shot into an open net for a quick lead.
It opened a dominant first period in which New York outshot the Capitals 15-4, and resulted in a 2-0 lead.
Boyle got the second goal with a shot from the point on a power play with Ward off the ice, and a little help from the officials.
A clearing attempt appeared to hit one of the officials and didn't make it out of the zone. Boyle tracked down the puck at the left point and his shot eluded Holtby, who was being distracted by Rick Nash in the crease.
Holtby was the major reason the Capitals were still in the game after 40 minutes. He stopped 25 of 27 shots, including early second-period stops in close on Fast and James Sheppard that could have given New York a bigger lead.
Kuznetsov scored on a rebound with 6:01 left in the second period to cut the lead to 2-1, but Lundqvist had to be good to keep New York in front.
Earlier in the period, he made a sprawling pad save after Kuznetsov was set up in close by a spinning move and pass by Ovechkin. Lundqvist also showed patience, stopping Brouwer one-on-one in close
Ovechkin narrowed the lead to 3-2 with his fourth goal of the playoffs and second in as many games against the Rangers.
Kreider leads the Rangers in playoff goals (13) since he made his NHL debut in Game 3 of the Eastern Conference quarterfinals at Ottawa on April 16, 2012. ... Capitals C Eric Fehr missed his sixth straight game with an upper-body injury. C Michael Latta was a healthy scratch. ... Rangers RW Mats Zuccarello missed his second straight game after being hit in the head by a shot in Game 5 against Pittsburgh. ... New York recalled LW Ryan Bourque from Hartford (AHL) before the game. He did not play.