Check your head at set-up. Is it cock-eyed? If so, it is most likely messing with your backswing which in turn messes with the downswing.