Eden Hazard, symbol of Chelsea's slump from champions to a 10th-place finish last season, opened the scoring at Stamford Bridge with a penalty early in the second half, before James Collins equalised in the 77th minute.
Costa arguably should have been shown a second yellow card after catching West Ham goalkeeper Adrian with a late challenge, but he stayed on to steer a dramatic winner into the bottom-left corner from 20 yards.
"I think the team played a good game with a great intensity and spirit," Conte told Sky Sports.
"It is important to find the right spirit and intensity. We know we can improve and we must improve a lot, but today I am pleased for my players."
Asked about Costa's challenge on Adrian, Conte replied: "I did not see it."
The bruising Costa's goal made it a positive opening night for Conte, who handed a full debut to former Leicester City man N'Golo Kante and saw another debutant, substitute Michy Batshuayi, set up the winning strike.
But it was a bitter defeat for West Ham manager Slaven Bilic, who lost debutant Andre Ayew to a thigh injury in the 34th minute.
On Costa's challenge, Bilic said: "We are not happy with that. I don't think it was reckless, definitely not.
"It was not deliberate. But it was very late. Nine times out of 10, when you see that kind of challenge on a goalkeeper, it's a booking. Unfortunately this was the 10th time."
He added: "We were so close against a top side. But they deserved it."
- Antonio storms off -
With both managers spending the entire match on the edge of their technical areas, suited and gesturing animatedly, the game was characterised by hasty, imprecise football and full-blooded tackling.
Kante started at the base of midfield in the only change to the Chelsea team that started last season's opening game -- a 2-2 draw with Swansea City that gave the first hints of the rot that was about to set in.
Bilic fielded two debutants in Ayew and left-back Arthur Masuaku, but the former was forced into a premature exit after injuring himself chasing after the elusive Hazard.
The hosts' high defensive line prevented West Ham bringing lone striker Andy Carroll into the game and the first-half chances were all Chelsea's.
Branislav Ivanovic drew a save from Adrian at his near post, Hazard curled a shot a foot wide following a probing run and Willian obliged Adrian to palm a dipping free-kick over the bar.
In seeking to atone for his error, Antonio succeeded only in blundering into Azpilicueta from behind, obliging referee Anthony Taylor to point to the spot.
Hazard thundered the penalty into the roof of the net and within four minutes Antonio had been replaced by Sam Byram, the former Nottingham Forest player storming past Bilic and straight down the tunnel.
With West Ham offering next to no attacking threat, Chelsea continued to attack, Willian working Adrian with a crisp shot and John Terry heading over.
But the hosts should have been reduced to 10 men when Costa, booked in the first half for dissent, caught Adrian with an ugly, high challenge as the West Ham goalkeeper dallied over a clearance.
West Ham looked to have pulled off a smash-and-grab with 13 minutes remaining when Chelsea failed to clear a corner and Wales international Collins lashed a left-foot half-volley past Thibaut Courtois.
But with full-time looming, former Marseille striker Batshuayi flicked on a long ball to Costa, who steadied himself before guiding a shot between Collins's legs and beyond Adrian's despairing drive.