You're not actually his trump card. The fact he's playing as if you are lets them know that without you, he's a lot weaker. When you don't start showing up when he's in trouble, they'll get bolder. Not that they need the excuse, but when they have him, it's going to be a lot worse now that they think sex is his strongest weapon.

Did you save him?