John Mozeliak, the team's president of baseball operations, made the announcement Tuesday.
The Cardinals signed Holland to a one-year, $14 million deal on opening day and the three-time All-Star is 0-1 with a 7.36 ERA in 10 appearances. He has one blown save, which came Friday in Pittsburgh when St. Louis was leading 5-2 in the ninth inning.
He pitched in Sunday's loss to the Pirates and gave three hits and one earned run in one inning.
"I don't think he'll be used in the ninth inning at this time," Mozeliak said.
Norris will take over the closing duties for now, Mozeliak said. The Cardinals signed Norris in the offseason to a one-year contract for $3 million.
Asked if he was OK with Norris as the Cardinals' closer, Mozeliak said: "Am I? Perfectly fine."
Norris has five saves in five opportunities with a 2.03 ERA. He has allowed just three earned runs in 13 1/3 innings with 20 strikeouts and just two walks.
In 7 1/3 innings, Holland has allowed six earned runs. He has walked nine, including one intentional walk. Holland's last save was Sept. 22 while with the Colorado Rockies. After signing late with the Cardinals, Holland did not have a chance to take part in spring training. He began the season in Class A and had two appearances.
St. Louis recalled Holland on April 9. He pitched that night in the ninth inning against Milwaukee in a home game. He walked four and gave up a run while getting just one out before being pulled.
"Does it make sense to send Holland down?" Mozeliak said. "I'm not feeling that's probably our next step at the moment, but I don't think anything is off the table. That said, this guy is confident. He believes he can do this."