Stroud had chipped in from 51 feet on the 18th hole to force the playoff.
But the 44-year-old Duke made the better approach shot on the second extra hole, bouncing his ball in front of the flag and rolling it close.
"I've knocked on the door a lot and here we are," said Duke, who turned pro in 1994.
He was ranked 144th in the world and was making his 187th start on the PGA Tour.
Canadian Graham DeLaet finished a stroke back in third place. Watson finished fourth, two shots behind, after making a six on the par-3 16th hole.
"You gotta believe in yourself in everything you do," Duke said. "That's why those guys at the top are winning week in, week out because they believe they can do it. It's kind of one of those things once you finally do it it might come easier the next time. That's kind of the way I feel."
Duke wouldn't have been in position to win at all had luck not intervened on the 10th hole, when his ball ricocheted off a tree and onto the green to about 5 feet from the pin, allowing him to make birdie.
After a 17-foot birdie putt on the next hole, he made a 45-footer on the 13th hole, a shot that looked like it might go past the hole to the right, before falling in.
He battled Watson for the lead down the back nine, until the former Masters champion found trouble on the 16th.
Watson put his drive into the water and put his next shot over the green.
He finished two strokes back in fourth place.
"The wind affected the first shot, and the wind didn't affect the next shot," Watson said. "I flew it three feet past the hole, which you can't do right now because the greens are so firm."
Duke looked as though he had the tournament sewn up after saving par on 18, despite a tee shot that went well right and onto a hill, and a second shot that went just over the green. He used a putter to put the ball within 2 feet, then sank the putt as the crowd roared what they thought was a winning shot.
It looked even more secure when Stroud's second shot hit near the stick, but then rolled well off the green. That just set up the dramatic chip shot.
Stroud hit his tee shot over the cart path and 94 yards from the hole on the first playoff hole, while Duke's first shot jumped out of a fairway bunker and into the rough.
Duke bounced his second shot onto the green. Stroud's went into a greenside bunker.
Stroud chipped to 8 feet but had to watch as Green almost sank a long putt that would have ended it.
The two both struck the ball well on the second playoff hole, but Stroud missed a 25-foot birdie putt, and Duke made his short putt.
"I had three shots from 94 yards on 18, the exact same yardage, and I could not figure out a way to stop that ball," Stroud said. "Regulation, luckily, I chipped it in."
Watson, Charley Hoffman and DeLaet began the day tied for the lead, but 21 other players were within five strokes.
Webb Simpson shot a 65 to finish at 271, then headed home immediately after his round despite being just a stroke behind the leaders at the time. He said he knew the score wouldn't be good enough to win.
"I'm itching to get to my family, so I'm going to head to the airport," he said.
Justin Rose followed his U.S. Open win by shooting 6-under par for this tournament. He was in contention, with two birdies on his first seven holes, but didn't get another until the final hole and made three bogeys. He said fatigue was a factor.
"I'm still able to put one foot in front of the other," he said. "I still feel OK, but my guess is there's just a little bit of sharpness that I might be lacking."
No player has gone back-to-back after capturing the U.S. Open since 1997, when Ernie Els won the Buick Classic at the Westchester Country Club in New York.
Rose plans to play next week at Congressional before taking two weeks off to prepare for the Open Championship.
DeLaet a native of, Weyburn, Saskatchewan, said his thoughts this week have been with the people of Alberta, where widespread flooding is blamed for at least three deaths and forced thousands to evacuate.
He had the words "For Alberta" written on his cap Sunday.
The 2009 Canadian Tour player of the year pledged to donate $1,000 for every birdie he made to help the relief efforts.
PGA Tour Canada, a bank and a Canadian businessman all agreed to match the donation. He finished with three birdies on Sunday and nine for the weekend.
"Hopefully it puts a small dent in what they need," he said. "But our hearts are still with them."