I don't think anything prevents unsetting handlers at runtime, nor setting them, actually. I don't think it was particularly well defined before, but it's not now.
In theory one could unset handlers. However there is only a difference if we are removing on objects which used to be processed later and aren't brothers. One should question though if the original behaviour is right: the event was originally send when the object was in the list, so one might expect the event to be received. This actually seems to make it rather more well-defined than less.