If I were you, I will try to avoid the temptation and stay away from the guy, be it quitting my job or change department. I will not want to destroy my own marriage life because of unable to control my own temptation. Try to appreciate of what you already have with you now and don't make mistake and destroy a family.
I have a friend who can't resist the temptation and ended up with a divorce and separated from her kids. She is very regret now but nothing can be done since the mistake has already made.
Read more