Kelly completed a deflected 66-yard touchdown pass and added two more long ones in the fourth quarter, and the Rebels held on to beat No. 2 Alabama 43-37 on Saturday night.
The Rebels (3-0, 1-0 Southeastern Conference) left the topsy-turvy game with their first victory in Bryant-Denny Stadium since 1988, and second straight in a one-sided series. The Crimson Tide (2-1, 0-1) had been 25-1 in Tuscaloosa against Ole Miss and had never lost two straight in the series.
Both games went down to the wire and were settled with the help of Rebels interceptions.
"We certainly think we're at a point now where it's not a shock," Mississippi coach Hugh Freeze said. "I still say it would be an upset in most people's eyes to do it here.
"I don't think it surprises the people in our locker room. Our pregame was as short as it's ever been. I said, 'You're good enough to win.'"
The Rebels made two final defensive stands, the first when Tony Bridges intercepted a deep ball from Jake Coker, who came off the bench to lead a comeback from a 30-10 deficit. Ole Miss scored 24 points off five Alabama turnovers.
Kelly delivered big play after big play following an ineffective first half, none wilder than a fling and a prayer in the third quarter, a 66-yard deflection caught by Quincy Adeboyejo for a touchdown.
Kelly bobbled a high snap and lofted the ball toward a crowd as he was going down before a swarm of Tide defenders. Adeboyejo grabbed a deflected ball and raced toward the end zone in front of Alabama's student section.
"That happened in my prayer time today," Freeze said. "That's the only thing I can say about that."
Kelly said he figured the 6-foot-3 Adeboyejo could "jump the highest out of anybody, and I just gave him a chance."
"Right after that you think, 'Hey, we're winning this,'" he said. "And that's exactly how we all thought, that this was our game."
Alabama had won 17 straight at home coming into the game to tie Baylor for the longest active streak in FBS, another milestone snapped by a feisty Ole Miss team that's established itself as an SEC contender under Freeze.
"We thought last year when we played we gave some things away, and I really don't want to take anything away from Ole Miss," Tide coach Nick Saban said. "They played a really good game, but are you going to be a giver or are you going to be a taker? If you're going to be a great competitor, you've got to take what you want.
"I don't know if there's anybody we can beat if we're going to give away  points, I really don't."
Coker came up with big plays, and big mistakes. He led two touchdown drives in a span of 2:03 late in the fourth quarter, aided by an onside kick. Derrick Henry scored on a 2-yard run and Richard Mullaney caught a 2-yard touchdown pass. Coker also was picked off by Bridges downfield with 2:36 left.
Coker was 21-of-45 passing for 201 yards with three touchdowns and two interceptions. Henry ran for 127 yards.
Coker, meanwhile, ran seven times for 58 yards.
"We had some things in the game plan early that the quarterback was going to pull the ball on and do some things, so we thought Cooper's speed would be sort of a change of pace for them, a little bit of an element of surprise," Saban said.
Alabama got one more chance with 31 seconds left after the defense stuffed Jeremy Liggins on fourth-and-1 from the Tide 31. Replay officials said Jaylen Walton's elbow touched the ground just shy of a first down on the previous play.
Four straight incompletions, including a drop, ended the rally hopes for Alabama.