Pure speculation here but if you were to remove the insert and find something in hard thin plastic (but that could also flex a little) that had the same diameter as the distance between the bezel and case.
Then try to push it on the case so that it deforms and push back the spring in the bezel, wouldn't it come off easily then?