But I let them all the time.

No one should be able to take away from you the things that you enjoy.