Brown retired from football in his prime at age 30.
In 1966, he was in London making a movie, "The Dirty Dozen." It just so happened that boxing promoter Bob Arum was in town, preparing for Muhammad Ali's fight against Brian London in August.
Brown was a long-time friend with Ali, and had played a key role into getting Arum into promoting fights. For some reason, most likely because he was uber competitive, Brown got the idea that he could beat Ali in a fight.
This was, remember, Ali at his peak before he was banned from the sport for refusing induction into military service.
Many who saw him for his entire career, including Arum, say he was at his best from 1964 through 1967.
"He was an artist in the ring," Arum said on Friday, the day Ali died from respiratory failure at 74 years old. "You guys couldn't hit him. He'd be standing there in front of you, and he was so damned fast and had so quick reflexes, that he was impossible to hit.
"So many of the young guys that came after him and looked up to him like Sugar Ray Leonard, emulated him. Sugar Ray had that style, and I guess you could say he passed it onto [Floyd] Mayweather. But he was a genius, and the best guys in the world couldn't do a thing with him. Jimmy was great, but he wasn't a fighter."
But Arum was friends with Brown, so he felt he owed it to him to at least ask.
Arum went to see Ali and told him what Brown was proposing. Ali and Brown were friends, and Ali knew that someone who had not boxed previously wouldn't be competitive.
He knew, though, the fire that fueled Brown, so he would need to prove to him it was a bad idea. He told Arum to bring Brown to Hyde Park.
Brown was a big, strapping powerful man who was 6-feet-2, an inch shorter than Ali but at 230 was about 15 pounds heavier than Ali at the time.
They met and Arum said Ali stood with his hands at his side and told Brown to hit him as hard as he could in the face.
"Jimmy throws a big punch at him and it misses," Arum said. "And then he throws another and another and before long, he's thrown a lot of punches and he hasn't hit him. And Ali's hands are at his side.
"And so as he slows down, Ali kind of hits him about 10 or 12 times in like two seconds. And Jimmy just looked at us and kind of shrugged. Ali had proven his point."
The two icons never did fight, but they remained close until Ali's death Friday.