Anthony passed James' mark of 273 points. He opened the game by making three 3-pointers and then moved past James, his close friend, with another 3 from the left wing with 1:08 left in the first quarter.
David Robinson (270) and Michael Jordan (256) are third and fourth on the all-time U.S. list, respectively.
Anthony totaled 14 points in the opening quarter and finished with a game-high 31. He has 293 points in his Olympic career.
Brazil's Oscar Schmidt holds the all-time record of 1,093, but unlike Anthony, he didn't have his minutes restricted while playing on powerhouse teams.
"Of course I was conscious of it," Anthony said of the scoring mark. "My teammates was making sure I was aware of the situation, but tonight was one of those nights where I wanted to let the game come to me and take the shots that was given to me, take the open shots and kind of play basketball.
"And at the end of the day I was in my zone. I was playing basketball."
Anthony knocked down nine 3-pointers, one shy of his U.S. record, and finished 11-of-21 from the field. He added eight rebounds and two steals and might have saved the Americans from a loss by scoring 14 points in the fourth quarter.
"Carmelo was magnificent," U.S. coach Mike Krzyzewski said.
Anthony is the first U.S. men's basketball player to compete in four Olympics and has a chance to become the first men's basketball player to win three Olympic gold medals.
A 13-year NBA veteran who has starred for the New York Knicks the past six seasons, Anthony won gold with the U.S. in 2008 and '12. He also won a bronze medal in '04.
"He was wanting that moment," teammate Paul George said after the Americans escaped an upset. "He was special tonight. We joke about it, this being his farewell tour."
Anthony, 32, chose to play with the U.S. team in a summer in which close friends James, Chris Paul and Dwyane Wade either passed on the chance to play or were sidelined by injury. During an interview with ESPN that aired earlier this week, he explained why he decided to play in Rio.
"I enjoy it, I enjoy it," Anthony said. "I like it. I enjoy learning about different players' personalities, not just as basketball players, but as people. I feel like I'm that bridge, that gap, between the veterans and the young guys."
Anthony's Olympic odyssey began with a disappointing performance in Athens, where the Americans staggered to a bronze medal and worldwide embarrassment. But he's been a major part of the U.S. program's resurgence.
"His leadership has really blossomed big time," said Jerry Colangelo, USA Basketball's director. "I would say if Carmelo played on any team long enough he would be the leading scorer. That's the kind of scorer that he is."
In addition to holding the U.S. men's all-time scoring mark, Anthony has also played in a record 26 Olympic games. He trails Robinson by 31 rebounds for the all-time lead and trails Jordan by 20 field goals for the all-time field goal attempts mark.
Information from The Associated Press was used in this report.