The Patriots released linebacker Trevor Reilly to create a roster spot for Harrison.
Harrison, 39, visited the Patriots on Tuesday after clearing waivers on Monday to become a free agent.
He posted a photo of himself with 40-year-old quarterback Tom Brady from the Patriots locker room on his Instagram account, joking that he finally has a teammate who is older than him.
Harrison's agent, Bill Parise, told ESPN's Josina Anderson that coach Bill Belichick's reputation among players and the organization's winning tradition factored into Harrison's signing with New England. Parise said Harrison had interest from other teams.
The Patriots have been thin at defensive end and outside linebacker for most of the season. They have Trey Flowers atop the depth chart at their end-of-the-line position, with rookie Deatrich Wise and Eric Lee behind him. The team also uses linebackers Kyle Van Noy, Trevor Reilly and Marquis Flowers at the end of the line, depending on the package.
The Patriots and Steelers are vying for home-field advantage throughout the AFC playoffs and are locked in to the top two spots in the conference. Harrison would get a chance to play against his former team if both teams advance to the AFC Championship Game for a second straight season.
The Steelers released Harrison, the franchise's all-time sacks leader, on Saturday to make room for right tackle Marcus Gilbert, who returned from suspension.
Harrison, 39, has 80.5 sacks in a Pittsburgh uniform, including at least five sacks in each of the past three regular seasons despite playing a part-time role. Harrison also came up big in the Steelers' past two playoff runs with multiple sacks.
But his role decreased as the Steelers transitioned to more pass-coverage work for outside linebackers Bud Dupree and rookie T.J. Watt. Harrison has played 40 snaps through 14 games this season, compared to 587 last year. He has one sack this season.
ESPN's Mike Reiss and Jeremy Fowler contributed to this report.