I think the best way to approach this would be through your Husband. He needs to talk to them, and explain that you and him are a family unit, and not that they aren't welcome, but a phone call beforehand would be nice in case you guys are busy or are out.