Pogba quit Old Trafford in 2012 after becoming disillusioned with a lack of first-team opportunities before returning to the club in a world-record deal four years later.
The midfielder's United future is up in the air again amid interest from Real Madrid and Juventus, and he has revealed his first move away was partly inspired by members of his family.
"My brother, the United fan, told me he was really angry -- even more than me -- when I didn't play," Pogba told The Official Manchester United podcast.
"He told me, 'No, go. You can play in this team. If they don't want you here, go somewhere else. They will see.'
"My mother always told me, 'You will go somewhere but come back.' She always said this. I was like, 'We will see,' but you know mothers and the things she said: 'You will come back here, don't worry,' and that was just after. She said, 'You will come back to Manchester, don't worry,' and I did."
Pogba, 27, has won the League Cup and the Europa League since returning to United, adding to the four Serie A titles he won at Juventus.
He also lifted the World Cup with France in 2018 and, fit again after spending most of the season on the sideline because of injury, says he has the ambition to achieve much more before the end of his career.
"When you have a dream, just go for it," he said. "A dream is free. You wake up after a dream but, at least when you dream, you can dream as big as you want.
"You still dream and want to achieve it. What my goals are, I always want more, we can always learn at this age or 10 or even 40 or 50.
"You will always learn something. I love to learn new things. I love to try things and stuff."
Pogba has made only eight appearances in all competitions this season after suffering metatarsal and ankle injuries, and he said he's been frustrated at how long it is taking to make his return, even with the league suspended due to the coronavirus pandemic.