Betsy Hassett grabbed a consolation goal for New Zealand, but USWNT produced a stunning performance to earn its first win at the Olympics.
The U.S. went into the game looking to recover from the shock loss against Sweden in the opening match, which ended a 44-game unbeaten run.
"Sweden was a very good team and we didn't play our best, and when you do that up against a top opponent, they're going to punish you," Crystal Dunn said after the match.
"So that wasn't our best performance. I think we came into Game 2 knowing that we don't go from being a really great team two days ago to not being a great team anymore."
Lavelle got USWNT off to the perfect start after nine minutes after she received Tobin Heath's through ball and produced a composed finish.
Horan thought she had doubled the lead after 35 minutes, but her strike was ruled out by VAR for offside.
Just before the break, Horan scored from close range with a header, with Megan Rapinoe providing the assist. It was Horan's 23rd international goal and it came on her milestone 100th appearance for the national team.
With the team leading 2-0 at the break, U.S. first lady Jill Biden arrived in time to watch the side put the game away in the second half.
It could have been worse for New Zealand but the U.S. had four goals disallowed goals for offside in the first half.
Nine minutes later, New Zealand capitalized on some poor defending to score through Hassett.
However, USWNT responded with late strikes from Press and Morgan, plus a C.J. Bott own goal, to seal a comfortable win.
"I think my approach going into this game -- obviously it's in the back of your head that you're getting your 100th cap -- but I didn't want that to be a factor today," Horan said.
"I think we wanted to get the job done and my focus was doing whatever I possibly could to help the team win. I'm happy to get a goal and yeah, it's nice to have a fan in the stands, too."
Dunn added: "I think we were a little bit more composed, a little bit more patient on the ball this game, and we know it's going to be challenging chasing that gold medal. So we're not taking anything for granted."
The U.S. will now meet Australia in the final match of Group G on Tuesday.
Information from Associated Press was included in this report.