"He's going to be missed more as someone for the younger guys to lean on," O'Neill told ESPNNewYork.com. "He's going to be missed every fifth day, but I still think and feel Andy was the leader and lot of people didn't realize it."
The Core Four is down to three. Jorge Posada, moving from catcher to DH this season, is probably not going to be back in 2012. Mariano Rivera has two more years on his new deal, while Derek Jeter has three and an option for a fourth.
"It is just one more person gone from that special time in New York," said O'Neill, who won four championships with Jeter, Pettitte and Rivera.
Pettitte added a fifth ring in 2009, but the Yankees now go into 2011 in an unfamiliar role -- they are not the favorites to win it all. In fact, they have a rotation short on sure things.
Ivan Nova, 1-2 with a 4.50 ERA in his career, is in position to win the Yankees' fourth starter spot. The fifth slot is up for grabs between Freddy Garcia, Bartolo Colon and Sergio Mitre. The Yankees do feel they are stocked with potential starters in the minors, but the best of the best -- Manny Banuelos and Dellin Betances -- are more likely to arrive in 2012 than 2011.
GM Brian Cashman could add another past-his-prime guy like Kevin Millwood, but to really replace Pettitte, especially after failing to sign Cliff Lee, Cashman will likely need to practice patience. Still, the Yankees will be waiting to pounce on any top-of-the-line starter, via trade, that becomes available.
According to O'Neill, it is hard to replace Pettitte not just because of what everyone saw from him on the mound, but because of what he did away from the diamond.
"Ever since he came back from Houston he's been a leader," O'Neill said.
Pettitte returned to the Yankees in 2007 after playing for his hometown Astros for three years. In Houston, he further cemented his reputation as a big-game pitcher, adding to a major league best 19 postseason wins.
"Usually, someone you consider a big-game pitcher has a fiery personality, but Andy was laid back until he took the mound," said O'Neill, now a Yankees broadcaster on YES. "He has been the staple of that pitching staff. Even with bigger names and everything, people looked to him. I think he is going to be missed in that respect as much as pitching the important games."
Still, on the field, O'Neill knows Pettitte's value when it counted most. O'Neill spoke about Game 5 of the classic 1996 World Series in which Pettitte beat John Smoltz .
"The 1-0 game in Atlanta, that is probably the one game that is etched in my mind," O'Neil said.
O'Neill, like Pettitte, walked away when he could have kept going. He, too, was 38 in his final season. O'Neill wasn't an All-Star then, like Pettitte last year, but he hit a respectable .270 with 21 homers and 70 RBIs.
"It is a big commitment to say I'm playing another year," O'Neill said. "It may not seem that way from a player's standpoint or a fan's standpoint. There are other things that are important in life that he wants to get to."
The Yankees have placeholders to fill in for Pettitte. A big-time, big-game starter is tougher to find. Especially one who might never have led the rotation, but was a leader in the clubhouse.