The 29-year-old Cilic, who reached the finals of two of the last three Grand Slams, dropped to a record of 10-4 on the season.
Kohlschreiber advances to the fourth round where he will play unseeded Pierre-Hugues Herbert who won his third round match when fellow Frenchman Gael Monfils retired in the second set with a lower back injury.
Cilic blasted five aces but couldn't overcome Kohlschreiber's strong serving game. The German won 76 percent of his first serve points and broke Cilic's serve twice, once in each set.
Kohlschreiber is now 7-4 overall against Cilic. He has won three out of their last four meetings, including a straight set victory in the final in Munich on clay in 2012. He won their first meeting on hardcourt in Moscow in 2007.
Cilic, who is considered to be one of the most consistent players on the Tour, claimed his 17th career title in Istanbul last year. He also reached the final at Wimbledon.
In January, he lost in five sets in the final of the Australian Open to Roger Federer, the third Grand Slam final of his career.
In the first set Kohlschreiber got the key break he needed in the third game to go up 2-1. He managed to hold serve for the rest of the set and take the final game on four straight points.
He got another break of serve in the seventh game of the second set to pull ahead 4-3 then held serve in the next game to go up 5-3.
Kohlschreiber closed out the contest on match point when Cilic's final backhand shot hit the net cord and fell back into his side of the court.