Jeter has a sore right oblique and sat out Tuesday night in a 5-1 loss at Toronto. He was hurt last week and played through pain during a weekend series at Baltimore.
"It's really tough because you're not doing anything," Jeter said. "I'm not hula-hooping or anything, you know?"
Manager Joe Girardi said Matsui could pinch-hit Wednesday but said Jeter was only available to play defense.
Jeter hoped to play Thursday but Girardi wasn't convinced.
"I'm not sure he'll play tomorrow," Girardi said. "We'll just have to see how he feels. Maybe we're looking at Friday."
Matsui left Tuesday's game after one at bat with a tight right hamstring.
In issues involving pitchers, manager Joe Girardi said right-hander Chien-Ming Wang will make at least one more minor league rehab start, although the date and location have yet to be decided. Wang (sore hip) allowed three hits in six scoreless innings for Triple-A Scranton/Wilkes-Barre on Tuesday.
Right-hander Brian Bruney (shoulder) threw 25 pitches, all fastballs, in a bullpen session and could return next week, while left-hander Phil Coke, whose back seized up before Tuesday's game, was not available but could return Thursday.
At the Yankees' facility in Tampa, catcher Jorge Posada (right hamstring) took batting practice, threw and ran in the outfield, including partial sprints and agility work. Posada might be ready to play in a minor league game by early next week.
Also in Tampa, right-hander Sergio Mitre, who is coming back from right elbow ligament replacement surgery, allowed one run and three hits and struck out eight over four innings in his third extended spring training game.
Jeter, who is batting .273 with four homers and 13 RBIs, has neither thrown nor swung a bat since Sunday's game for fear of aggravating his injury. He took groundballs during batting practice Wednesday and said he hoped to try swinging on Thursday.
"Until you swing you can't really tell the extent of how it feels," Jeter said. "They just felt it would be better to take that extra day."
Matsui said his hamstring was feeling good enough to play but Girardi didn't want to push him.
"The hard thing is, in a game it's hard to control your running," Girardi said. "Your instincts take over. He could pull it or he could end up hurting his knee."
Matsui is batting .263 with three homers and 12 RBIs. He had surgery on his left knee last Sept. 22, and the Yankees don't expect him to play the outfield until June. Matsui's knee filled with fluid in April, limiting him to pinch-hitting duties for several games.
Wang is 0-3 with a 34.50 ERA in three starts. He was placed on the DL April 24, retroactive to April 19, with muscle weakness in his hip.
"He threw the ball better yesterday," Girardi said. "He's still not where he has to be. His slider was very good yesterday, his sinker was better yesterday. We're going to have him make another start and then we'll make an evaluation after that."
Bruney, out since April 25, is 2-0 with a 3.38 ERA in nine games.
"He'll throw again Friday and then we'll make an evaluation about what's next," Girardi said. "He's definitely on the road to recovery, which is very good. I'd say it's possible he'll be back sometime next week, if everything goes according to plan."