Yeah, you'll have that with any hobby. High-end Audio gear is WAY worse. I've spent $800 on a power-cable for some stupid reason.
To avoid this, only 2-3 of the 15 "rep" watches I own aren't what most would consider "super-franken" so that I can always part them out if I need to.
I have about $5K into my 16520. Now, I did it "right" and have a watch that would have cost $10K on a good day. If I lose interest, I can ALWAYS sell off the Dial, Hands, Movement, Bracelet, Pushers, and Crown/Tube on eBay and break even. Hell, I may make some $, as I got good deals on everything.
It's all about your pain threshold.