Fisher wore the jersey Tuesday while appearing at a charity event for Rocketown at Lipscomb University in Nashville with former Colts coach Tony Dungy, among others.
When he introduced Dungy, Fisher took off his jacket and shirt, revealing a Manning Colts jersey underneath, saying, "I just wanted to feel like a winner."
Rocketown is a 40,000-square-foot entertainment complex in Nashville created for teens to gather in a drug- and alcohol-free environment.
The Titans, who were expected to be an AFC playoff contender this season, are a disappointing 0-6. The Colts are 5-0.
The Titans' latest defeat was a 59-0 shellacking at the hands of the New England Patriots.
During his weekly radio show on 103.3-FM in Nashville, Fisher apologized, but made it clear he didn't believe he did anything wrong.
"It was for a very, very worthwhile cause, charity," Fisher said on his show, according to The Tennessean. "I was introducing Tony, just having fun with it and I really apologize if I offended anybody, but if you're offended over the nature of that type of thing, then I think you need to rethink things.
"This is a very worthwhile and needy benefit and I was honored to be there. I'm sorry if I offended anybody but if I had to do it again, I would do it again."