It was a celebration worthy of an overtime playoff victory.
For the Devils though, it was simply a first win under new coach John Hynes and they really needed it.
Stempniak finished off a 2-on-1 break with Adam Henrique at 3:07 of overtime and the Devils ended some recent frustrations against the Rangers with a 2-1 decision Sunday.
"It feels great," said Hynes, who lost his first four games, the last in a shootout Friday night. "I am really just happy for the team."
Stempniak, who made the roster after coming to camp on a tryout basis, also got the satisfaction of beating the Rangers, who had sent him to Winnipeg in a trade deadline deal.
"It feels good to beat them, always, but more important, it's a huge win for us as group," said Stempniak, who snapped his winner over the pad of Henrik Lundqvist in ending the 3-on-3 overtime. "Instead of going down 0-5, to get win, pick up a point last game, we're turning in the right direction.
Derek Stepan, who scored for the Rangers in the opening minutes, had hit the crossbar just before New Jersey won for the first time in five games.
"It feels great," said Devils goalie Cory Schneider, who had 26 saves. "Maybe that celebration was a little too much for the fifth game of the year, but you could just tell, everyone, some frustration and anger came out there. It was exciting. It felt really good, but it just one game."
The loss was the third straight for the Rangers, who opened the season with three wins.
"I think we need to be concerned," Rangers defenseman Dan Girardi said. "We're not pressing any panic button by any means but it needs to be addressed right now."
Lundqvist stole one from the Devils with 2:35 left in the extra session, coming across the crease to stop Kyle Palmieri on his doorstep. Greene set up the shot with a great pass.
"I felt like the game was right there for us," said Lundqvist, who had 22 saves. "We had some opportunities late in the game but couldn't cash in."
The Devils had a great chance to take the lead early in the third period when they had a two-man advantage for 1:36 after defenseman Kevin Klein was called for delay of game on a penalty kill. However, the Rangers did not allow a shot.
After being shut out by the Montreal Canadiens on Thursday night, the Rangers scored early. Rookie Oscar Lindberg picked up a loose puck at center ice, skated it into the New Jersey zone and found Stepan alone in front. The center cut across the crease and slipped a backhand between Schneider's pads at 3:12.
New Jersey, which may not have had a quality chance in the first period, tied the game 1:52 into the second period when Henrique stuffed his own rebound past Lundqvist. It was his team-high third goal of the season.
The Devils scratched RW Tuomo Ruutu with an undisclosed injury. ... Backup G Antti Raanta is still waiting to make his first regular-season appearance with the Rangers. ... Klein got the best of Palmieri in a late first-period fight. ... Stepan has 11 goals against New Jersey, the most he has scored against any team. ... New York has outscored its opponents 9-2 in the first period this season. ...The Devils lost all four games to the Rangers last season.