After upsetting Ivanovic at the Auckland Classic for her first win over a top-20 player, friends of Britain's Broady struggled to place the former world number one.
"Some of my friends don't follow tennis at all," Broady, who comes from Stockport near Manchester, told the New Zealand Herald.
"I have messaged them 'I won, I won' but it's like eating humble pie... I have to explain to them that I have won a good match and they just don't get it."
She added: "They are more bothered that I played Schweinsteiger's girlfriend I think. They don't get tennis whatsoever."
Broady, 25 and ranked 122nd, came through qualifying to make the main draw in Auckland where she beat Ivanovic, the Serbian second seed, 7-5, 6-4 in the first round.
Broady fired down 14 aces to overpower Ivanovic, and another 21 when she beat Latvian Jelena Ostapenko in the second round 4-6, 7-6 (7/4), 7-5 on Wednesday.
Broady said that before playing Ivanovic, she was inspired by the 18-year-old Russian Daria Kasatkina's win over top seed Venus Williams in the first round.
"My coach said before the match the biggest thing would be believing you could win," Broady said.
"The fact that the young girl went out before me and beat Venus gave me a boost. If she could take out Venus, why couldn't I take out the number two seed?"