If it annoys you ( as it would me) then you should it address it now. You have let them do it, so they do! Don't open your door until exactly opening time. They will stop showing up early when they can't get in.

If you let it go it will just eat away at you and you will resent this family.