James was flagged for unnecessary roughness after a helmet-to-helmet hit on wide receiver Ashton Dulin.
Dulin made his way to the Colts' locker room following the hit and was ruled out for the rest of the contest because of a concussion.
Chargers coach Brandon Staley said after the game that he did not agree with James' ejection.
"I know Derwin's intent, you have to establish intent, and I thought that he tried to lead with his shoulder," Staley said. "It's a play where [the Colts] obviously laid the guy out in a tough position, which is their fault, not ours, and Derwin made an aggressive play.
"We'll have to look at it, but I know what Derwin's intentions are. I've seen him make a lot of tackles on plays just like that and I know that his target was not above the neck."
Staley said James was placed in the NFL's concussion protocol, but that he was "having a heck of a time in that locker room" during a postgame celebration as the Chargers clinched a playoff spot with the win.
Two plays earlier, James had drawn a face mask penalty when wide receiver Michael Pittman Jr.'s helmet came off following a 7-yard catch.
It was James' first game back after sitting out two consecutive tilts because of a quadriceps injury.
In the first quarter, he intercepted Colts quarterback Nick Foles.
Named to his third Pro Bowl this season, James has registered 109 tackles, four sacks, two forced fumbles and two interceptions.
ESPN's Stephen Holder and The Associated Press contributed to this report.