To prevent any difficulties face at work, I rather much quit the job. I did once and it was for my well-being. Selfish as it seems but just know that it is either you being difficult to your co-workers or they being difficult to you. So pick your option wisely.