Victorino hadn't started the last four games. He is hitting .284 with six home runs and 19 RBIs in 39 games.
Brown was hitting .341 with two home runs and 10 RBIs in 11 games for Lehigh Valley. The 23-year-old started the season on the DL after breaking his right hand early in spring training.
"There's still a question of whether he's ready to come up here and play on a regular basis," Phillies general manager Ruben Amaro Jr. said of Brown. "But the circumstances have changed. And I changed my mind. That's all."
Amaro and manager Charlie Manuel said that they met in Manuel's office late Thursday night and talked about the possible options to replace Victorino on the roster. After kicking around ideas, they decided Brown was still their best option, even though he'd played in just 11 games at Triple-A.
"Is it the right thing to do? I'm still not sure," Amaro said. "But it's the right thing to do for this club right now."
Manuel said that while Brown would not play against left-handers at first, this is a different situation than late last season, when he only played sporadically.
"It's different than last year," the manager said. "He'll get to play more than that."
"He's got to play," Amaro said. "It's important for him to play. How much, I think he'll dictate that. ... But Charlie is pretty good at breaking guys into the big leagues. He did it with Chase (Utley). He did it with (Ryan) Howard. And this is a circumstance where we need to break him in."
In other injury news, Amaro said Utley would not be recalled for this weekend's series against the Texas Rangers but made it clear Utley's return is near. Utley played full games in the field back-to-back in Clearwater on Wednesday and Thursday, and "he's doing well," Amaro said. "He hasn't gone backward. That's the important thing."
Utley will take a scheduled day off Friday and then play again Saturday. Earlier in the week, the Phillies had talked about having Utley play three games in a row and seeing how he responded. Now, Amaro said, Utley will play Saturday "and we'll see where he's at after that."
"I don't even know how many times he'll play three games in a row here," the GM said. "He may not do that initially. He might play one game on, one game off, two in a row, then another day off. I think we have to monitor him when he gets here ... until we know he can go out there and play five or six games a week."
Closer Brad Lidge will join extended spring training Sunday night and hopes to return to the Phillies in three weeks.
The Phillies still expect to place pitcher Joe Blanton on the disabled list but didn't do so Friday. Blanton did have an MRI, but results weren't available yet. Amaro said Blanton was reporting "some new discomfort" in his elbow but wasn't willing to speculate on whether that might mean Blanton has a more significant elbow problem. Amaro and Manuel both said it was "likely" that Vance Worley would be recalled to replace Blanton in the rotation, but Amaro said they hadn't made a "final decision" on that yet.
Also Friday, the Phillies optioned right-hander Scott Mathieson to Triple-A and recalled right-hander David Herndon. Herndon began the season with the Phillies and appeared in 11 games before being optioned to Lehigh Valley on May 4.
Information from ESPN.com's Jayson Stark and The Associated Press was used in this report.