Top-ranked Nadal fended off five break points in the last game before Cilic broke his serve. Then the 16-time major winner went to shake hands with the umpire and his opponent and angrily hurled his headband into his equipment bag.
No. 6-seeded Cilic advanced to his first semifinal in Australia since 2010 with a score of 3-6, 6-3, 6-7 (5), 6-2, 2-0, retired.
Nadal limped into a news conference about a half-hour later, still wincing when he stepped up onto a platform. He said he felt muscle pain in his upper right leg in the third set but played through it. In the fourth set, chasing a drop shot, he felt the pain get worse "but didn't realize how bad."
"Tough moments -- not [for] the first time here," he said. "I'm a positive person, but today is an opportunity lost to be in a semifinal for a Grand Slam and fight for an important title for me.
"It's really tough to accept."
Nadal said he'd have medical scans Wednesday to determine the exact location and extent of the injury, which he could only describe as being high on his right leg but not in the hip.
"I can't tell you exactly the muscle. It's here high," Nadal said. "High on the leg. But I don't want to lie. Tomorrow we're going to communicate what's going on after the MRI. You know, is not the moment to say what's going on or what not going on because we really don't know and the doctor really don't know yet. Is better to wait just a few hours. Give me that time, and tomorrow afternoon we'll let you know."
It was only the second time Nadal had retired during a Grand Slam match -- the last time was also an Australian Open quarterfinal, in 2010 against Andy Murray.
On Tuesday night, Nadal needed a medical timeout after going down 4-1 in the fourth set for treatment on his leg.
Nadal returned but was clearly bothered by the injury, limping and taking as much time as possible as he tried to stretch between points.
He called the trainer again after losing the fourth set, and lasted only two further games.
Cilic had beaten Nadal once in their six previous matches -- in their first match at Beijing in 2009.
"Unbelievable performance from both of us and really unfortunate for Rafa," Cilic said. "He's such an unbelievable competitor. He always gives his best ... it's very unfortunate for him to finish this way."
Nadal had a delayed start to the season because of an injured right knee but appeared to be in good form through the first four rounds. He now hasn't won back-to-back Australian Open quarterfinal matches since 2008 and '09, the year he won his only Australian title.
"I worked hard to be here," said Nadal, who skipped tournaments in Abu Dhabi and Brisbane, Australia, while his knee recovered at the start of the season. "We did all the things that we believed were the right things to do."
His absence also means there's only one of last year's singles finalists remaining in the tournament. Roger Federer, who beat Nadal in five sets last year, is playing Tomas Berdych in a quarterfinal on Wednesday.
Serena Williams didn't defend her title, deciding she hadn't had enough time to prepare following the birth of her first child in September. Her older sister, Venus Williams, was beaten in the first round.
On top of that, six-time champion Novak Djokovic was upset in the fourth round.
There'll be a British man in the Australian Open semifinals for the seventh time in nine years, but it won't be five-time finalist Murray -- who skipped the season-opening tournament to have surgery on his hip.
Edmund had never played in a major quarterfinal, had never won five consecutive matches at tour level, had lost both of his previous matches against Dimitrov and had never beaten a top-five player.
He checked all those boxes on Rod Laver Arena.
"I am loving it right now, just the way I'm playing," Edmund said. "My first Grand Slam semifinal. First time I played on one of the biggest courts in the world. To beat a quality of player like Grigor. They're great feelings. So, yeah, I just try to enjoy it as much as possible."
After breaking Dimitrov's serve in the ninth game of the fourth set, Edmund set up match point with an ace. Then he had to wait before a video challenge confirmed that Dimitrov's last shot -- a floating backhand -- was out.
"I just held my nerve in that last game and prayed that last ball would be out," Edmund said. It was out. So was Dimitrov, who lost a five-set semifinal here last year to Nadal.
"Everything went his way today," Dimitrov said. "It's hard to hide a disappointment. It hurts, and so it should."
Edmund, who upset US Open finalist Kevin Anderson in the first round, is now the center of attention for the tennis-loving British public.
"I know what it feels like to be Andy Murray the last eight years," he said. "It's probably the first time I've done well on my own, so there's more attention there. Of course you take it in stride."
The Associated Press contributed to this report.