They are annoying. That's the point. They react realistically. They don't have any idea what he is doing. The audience knows what more about the context and actions than the family.

It is at least worth watching until the end of Season 4, which actually ends in such a way that you feel concluded, instead of on a cliffhanger.