All of them are looking to bounce back.
Ellerbe, 29, missed 15 games with a hip injury last season, then he was traded from the Miami Dolphins to New Orleans just two years into a five-year, $35 million contract.
Spencer, 31, suffered a knee injury in 2013 that required microfracture surgery. The former Dallas Cowboys Pro Bowler came back last season but wasn't the same player.
The Saints' defense, meanwhile, imploded to 31st in the NFL in yards allowed last season, thanks to a combination of missed assignments, missed tackles, underachieving talent and leadership issues.
Now they're all hoping to rally together.
"[The Saints'] defense is very hungry. Ever since I got here the first day they were talking about how they had a bad taste in their mouth from last year and wanted to play for each other this year and get better," Ellerbe said.
"It didn't really matter where I went, I would've had a chip on my shoulder. I have a chip on my shoulder every year. But the fact that the defense got a chip on their shoulder makes it that much sweeter. We all go together and fight together and have the same goal to be the top defense in the league."
Ellerbe and Spencer were part of the Saints' overall effort to improve their leadership and toughness. They also added veteran cornerback Brandon Browner and defensive lineman Kevin Williams, while parting ways with veterans such as Curtis Lofton, Junior Galette, Patrick Robinson, Corey White and Brodrick Bunkley for various reasons.
It's unclear if either veteran will lock down a starting job. Both of them have just returned to practice this week after missing most of last week with unspecified injuries.
When healthy, Ellerbe has been running with the Saints' first-string defense this summer at Will linebacker. The 6-foot-1, 245-pounder brings an athleticism the Saints have been lacking for years in that spot, with the potential to be an asset both dropping back and coverage and blitzing. And he talked this summer about how much he prefers playing on the weak side after a pedestrian 2013 season as the Dolphins' middle linebacker.
But Ellerbe will have to fend off fellow veteran David Hawthorne to win a starting job, since rookie inside linebacker Stephone Anthony appears to be staking his claim on the starting Mike linebacker job.
Ellerbe said he's eager to measure himself against another team in Thursday's preseason opener -- especially since it's coming against his original team, the Baltimore Ravens. Ellerbe's best season came with the Ravens in 2012, when he helped them win a Super Bowl.
When asked if he feels like he has something to prove all over again after missing last season or if he expects to pick up where he left off, Ellerbe said, "A little of both."
"I know what I can do," Ellerbe said. "When you're gone for a year, people seem to forget. But when you get back out there, it's second nature. We've been doing this all our lives. I just missed a year."
Spencer (6-3, 265) is playing both the "Jack" defensive end position behind starter Cameron Jordan and the Sam strong-side linebacker position, where he's competing with young players Hau'oli Kikaha, Kasim Edebali and Ronald Powell.
If healthy, Spencer can still be a big asset. He started to play his best football toward the end of last season, with his best performance of the year coming in Dallas' playoff win against Detroit. And Spencer was one of the NFL's premier pass-rushers in 2011-12 -- when he last played under current Saints defensive coordinator Rob Ryan.
Spencer said this version of Ryan's defense is "a little more simple," which he said is good since players can react and play faster without thinking so much.
"He knows how to use his players," said Spencer, who had a career-high 11 sacks in his 2012 Pro Bowl season under Ryan in Dallas. "He puts players in good position to make plays."