Johnson will serve as defensive line coach and work under defensive coordinator Jim Schwartz, who was hired last week.
The Bills previously granted the Cleveland Browns permission to interview defensive line coach Anthony Weaver, but the Browns have yet to announce his hire.
Johnson, 49, spent the past 14 seasons as an assistant coach for the Patriots, serving as an assistant linebackers coach (2000), inside linebackers coach (2001-2003), defensive line coach (2004-2011), and linebackers coach (2012-2013). He decided to leave the team earlier this month.
Patriots coach Bill Belichick said in a statement that he was "proud and honored to have spent more years of my career with Pepper Johnson than any other player or coach." Johnson played under Belichick as a linebacker with the New York Giants (1986-92), Cleveland Browns (1993-95) and New York Jets (1997-98).
Schwartz served as a personnel scout for the Browns from 1993-95.
Johnson had expressed desire for a coordinator post in the past and this move might give him a better chance. Schwartz, a veteran coach who was fired after five seasons with the Detroit Lions, could be back in line for a head coaching job as soon as the 2015 season.