I have flown fixed wing, but not rotary. Flying itself is relatively easy. I have only ever done flying in the military scenario, not much civvie stuff. There is a lot to take onboard, but its all about a learning curve. If you can fly fixed wing, you can fly rotary and vice versa. I wouldn't say one is harder, just that they are both different.
For each there are things that are easier, and things that are harder. Obviously one of the hardest things in rotary is to get the thing to hover, but it is comparible to riding a bike, once it clicks, you can do it. Flying can sometimes feel unnatural to people at first, but it's not something that should put you off. If it is something that you have a passion for, then go for it.
My recommendation would be to go and have a few trial lessons, in both fixed wing and rotary, and see what you like best. I have always fallen on the side of fixed wing, whilst rotary is great fun at low level, you can have a lot of fun flying fixed wing at higher altitudes, more specifically aerobatics.