Madrid went behind to Santi Comesana's early goal before Luka Modric's penalty and Eder Militao's header put the visitors ahead in Vallecas, before Alvaro Garcia levelled ahead of the break.
A second-half penalty scored by captain Oscar Trejo -- retaken after goalkeeper Thibaut Courtois saved Trejo's initial effort, with defender Dani Carvajal encroaching inside the box -- gave Rayo the win, and left Madrid two points behind Barcelona at the top of LaLiga.
"We weren't convincing in the individual duels," Ancelotti said in his post-match news conference. "Sometimes, when you can't match the attitude, quality alone can't win you the game. We lost a lot of balls and we suffered with their pressing. It was a bad night."
"In terms of quality we dropped off a bit," Ancelotti said. "Rodrygo [Goes] and Vinicius [Junior] missed some chances that they'd usually score. It wasn't a good game. [Marco] Asensio did well, but the others weren't as good as usual."
After a busy spell of 11 matches in 36 days that began in early October, Madrid now have one last game -- hosting Cadiz on Thursday -- before LaLiga takes a seven-week break for the World Cup.
"We lacked some freshness, but that's normal after so many games," Ancelotti said. "If we evaluate this first part of the season, we've done well.
"We lost today and the defeat bothers us, but we're doing well in the league and we're in the Champions League round of 16. We need to finish well against Cadiz and then we need to rest."