It may be cultural. I always pay the fee because I feel the same way you do.
But from the other point of view. The seller is asking you to pay a certain amount for an item, say $100. If I the buyer agree to purchase the goods for $100 that's what I'd expect to pay, not $101. Fees would be on the seller. As a business owner, I charge a a certain price for my product. When the buyer pays me with a credit card I pay a percentage of that sale to the credit card processor. That's just how it works. It's the cost of doing business... I guess technically I build that fee into my sales price, which is probably what we as sellers accepting paypal should do. Maybe we should build the price of the transaction fee into the selling price...
This may be a totally American, free enterprise kind of mindset. What's the going mindset in the Land of Oz or Europe? Interesting topic actually!