Withholding Sex As Punishment

At what point does it become punishment because I personally won't sleep with someone if I'm angry with them. It's not about "punishing", it just means I'm too cross to find you sexually arousing right now.