Traditionally, with a few notable exceptions (Arab sheiks, rap artistes, etc), Rolex & Patek tended to be worn by those whose habiliment was more stylishly conservative. Old money, if you will. While those who wore larger, more modern watches tended to be more fashionable. The nouveau riche.
Today, with Rolex chasing (& often designing for) the Asian market, which tends to overcompensate for decades of communist-imposed anti-displays of wealth, the demarcations are no longer so clear.