Brown made his feelings known on CJ McCollum's "Pull Up" podcast, when asked about Boston's championship chances.
"Oh, we're getting to the Finals. No question about it," Brown said.
"I hate how everybody is like, 'Oh, LeBron's gone in the East,'" Brown said. "I know he did have a strong hold on the East for the last seven years, but he barely got us out of there this year. And our mindset was like, 'Man, he's not beating us again.'"
It was James and the Cleveland Cavaliers that won the last two games in the Eastern Conference Finals to take the series in seven games and send Boston home for the summer.
The 21-year-old Brown has increased his scoring average by almost eight points a game from his rookie campaign to last season and says the team can only get better.
"We got like a lot of -- we got a lot, a lot of talent on this team. And we've got some real good mindsets, and it's a pleasure playing with them all. But we have to be on one page if we want to be successful," he said.
The Celtics will start the regular season against the Philadelphia 76ers on Oct. 16.