Aside from playing sports, are there any other scenarios where you have to remove your wedding rings?

I think it's better to remove it whenever you do chores at home, when you shower or any activity that involves soap and water (even when you put lotion on) just so to avoid losing it. You may opt to remove it when you are also going to the gym or doing any sports (especially swimming).