I always figured that anyone hurting Dean's little brother would be a deal-breaker for him, especially when Cas knew how important it was to keep those memories of hell walled off. I agree that it's a totally separate thing from the personal crap that Sam and Dean had to work through over Ruby