McIlroy started the last round four shots behind Scott, but drew even when he eagled the seventh and birdied the eighth.
Scott went a shot ahead with a birdie at the ninth. The pair then went shot for shot over the back nine before the dramatic finish.
Scott's approach shot went over the back of the green and his chip went well past the hole, with two putts bringing a bogey. McIlroy hit his approach to 10 feet and sank the putt to win by a shot.
"I wanted to get a win and finally I've been able to get one," McIlroy said. "But more satisfying than that is being able to take one of the best players in the world down the stretch and come out on top.
"Adam is a phenomenal golfer, a great competitor and probably an even better guy and I feel a bit sorry that I was the one that ruined the triple crown for him."
Scott was attempting to become only the second player after compatriot Robert Allenby in 2005 to win his country's Triple Crown by taking the Australian Open, Australian Masters and Australian PGA titles in the same season.
He carried his form from the Masters and PGA tournaments into the Open, shattering the course record with a 10-under 62 in his opening round.
He came into the final day at 16 under, with rounds of 62, 70 and 68, four shots ahead of McIlroy, who was four ahead of the next closest challenger.
That, in effect, made Sunday's final round a shootout between the No. 2-ranked Scott and Northern Ireland's McIlroy, who was the world's top-ranked player last year.
Scott was left to rue a series of missed birdie opportunities, failing to make six putts from within 12 feet during his final round.
"Nothing was going my way on the greens today," Scott said. "I could have put this thing away early on if the putter was behaving how it should have, how it did the rest of the week."
Both players took irons from the tee on the last hole and landed their shots adjacent to each other on the fairway. Scott's approach landed ahead of the hole, but a bounce sent the ball flying through the back of the green.
Rather than pitch and run back onto the green, Scott chose to give his chip some loft. It failed to pull up, rolling onto the lower tier of the green and he did well to salvage a bogey.
McIlroy stood assessing his 10-foot putt during Scott's excursions. After Scott holed out, McIlroy rolled in a slow putt that just made it to the hole.
"I didn't think it was going to unfold the way it did," McIlroy said. "It's hard not to feel some guilt in the way I won it. Having a one-shot lead going into the last having it taken away from you right at the very end, it's tough."
McIlroy finished with an 18-under 270. Scott finished six shots ahead of Australian John Senden (11 under).
Senden and Australians Bryden MacPherson and Rhein Gibson, who tied for fourth at 9 under, all qualified for next year's British Open.