I would keep him. I would also keep Cano, Perez, Chapman, and Clippard.
You have solid closers and those can be hard to come by in a new draft, but I could see a good argument for Cain. If Anderson can stay healthy next year he could be a stud in Oakland.
Sabathia is getting old, as are many of your offensive players, Suzuki, Hunter, Beltran, and Teixeira.