It is very easy. You want to stop save your marriage. Just ignore him and do not allow him to come near you. Why should you worry about your marriage. It will be done when the time is ripe. All living beings, even the worms are endowed with the sexual pleasure by God for the sustenance of this world. You need not give undue importance to carnal desires for a happy life.