This is an interesting thread... I think that people are going to be jerks and call you out no matter what the circumstances. I've just purchased my first reps so I do not have any confrontation stories... However -- even if you look the part, some negative punk will find something to hassle you about.
For an example, I'm a student: young and often run around on the weekends and after work in shorts and a t-shirt. I can't tell you how many times people comment negatively on the cars I drive in. He's young so it must be his dad's car... If I was a minority, I'd be a drug dealer. If I drove a cheap car with a nice watch, it's gotta be fake or stolen.
It depends on the situation but there will always be something because some unhappy people get the attention they seek by calling others out. I can tell you that if I wore a rep, no matter how excellent or crappy it was... Even if I'm rolling in a 100K plus ride, someone will try and make me look bad. Sure the watch is real but he couldn't have paid for it himself...
I've just learned to live and let live. They have a problem with what you cherish? Who cares. Hopefully you're wearing it for yourself because you enjoy wearing it so what they say about you shouldn't really matter anyway. Also, if you present yourself as the better man, show your confidence and pride, you'll come out looking good while he appears as the pessimist ass that he is.