Swaddle Or Not?

My baby (currently 3 weeks) always struggle whenever we swaddle him. There was a period when we released his swaddle during the day and he seemed to sleep better. Do you think I should stop swaddling him at night?