Djokovic came in with 11 straight wins over Tsonga but was nowhere near adding a 12th.
''Just nothing was going,'' Djokovic said. ''No baseline, no serve, no return. So just generally (a) very bad day, very poor performance.''
Tsonga's serve gave Djokovic plenty of trouble. The Frenchman had only eight aces, but Djokovic continually struggled to keep his returns in play.
Serving on match point, Tsonga won when Djokovic's return went wide. Tsonga did some shadow boxing and jumped up and down before waving to a receptive crowd.
''It's not every day you get the rewards of what you're doing every day during the year,'' he said. ''You leave your family, practice every day hard, suffer a lot on the court. And when you win a match like this, you just feel good.''
Djokovic never looked comfortable in Toronto, his first tournament since winning Wimbledon last month. Four days later, he married his longtime girlfriend.
On Wednesday, Djokovic needed three tough sets to outlast Gael Monfils 6-2, 6-7 (4), 7-6 (2) in the second round.
''I wasn't hitting the ball clean and a lot of unforced errors,'' said Djokovic, who beat Tsonga in the 2008 Australian Open final for his first major title. ''It's maybe (a) lack of matches on this surface and, well, it takes time. Let's just say it's going to be better, I'm sure.''
Tsonga will play eighth-seeded Andy Murray in the quarterfinals. Murray advanced when third-round opponent Richard Gasquet withdrew because of an abdominal strain.
Djokovic's loss was the second big upset of the day. Third-seeded Stan Wawrinka lost 7-6 (8), 7-5 to Kevin Anderson.
Anderson will play seventh-seeded Grigor Dimitrov, a 7-5, 5-7, 6-4 winner over Tommy Robredo.
Fifth-seeded David Ferrer also was pushed to three sets in a 1-6, 6-3, 6-3 victory over Ivan Dodig. Ferrer will face Federer.
Canada's Milos Raonic also advanced, beating Julien Benneteau of France 6-3, 4-6, 6-4.
The sixth-seeded Raonic, coming off a tournament victory Sunday in Washington, will play Feliciano Lopez in the quarterfinals. Lopez beat fourth-seeded Tomas Berdych 3-6, 6-3, 6-4.