Luzardo is out indefinitely after breaking the pinkie on his pitching hand when he thumped a table while playing a video game before his start Saturday.
Oakland manager Bob Melvin said he didn't know how much time Luzardo would miss or whether he would require a cast or splint on the finger. Melvin was hopeful the pitcher could still keep his arm in shape given the location of the break.
"Before the game he was playing a video game and accidentally bumped his hand on the desk as he was playing the game," Melvin said. "He came in, was a little bit sore, training staff checked him out. We threw him in the cage before he went out there, watched him warm up. He was comfortable pitching, the training staff was comfortable with him pitching. After the game, we got an X-ray and there was a hairline in the pinkie finger."
Oakland is getting first baseman Matt Olson back in the lineup Sunday. Olson had missed three straight games after being hit in the left eye by a ball that ricocheted off the L-screen during batting practice Thursday.
Luzardo pitched three innings and took the loss against the Orioles, allowing six runs -- three earned -- on five hits.
He is 1-3 with a 5.79 ERA over six starts this season.
"It just depends on how it heals and how it responds," Melvin said. "As you would expect, it was a little bit worse today, a little bit puffier today, but really have no idea on a timetable yet."
Left-hander Adam Kolarek was called up from the alternate site to take Luzardo's place.
The Associated Press contributed to this report.