you are probably accidentaly double clicking the mouse, and there is no way I know of to stop that happening

I do it more as i get older and just hit the V key to switch back

I agree it is annoying