I don’t even have a fan, because of the same reason, I sleep with my room on lockdown and I wear earplugs and sometimes that doesn’t even help.
I wouldn’t suggest just to remove it, but that’s your choice :3
Or wear earplugs, they are life saving, trust me.
If you ever want to talk, just tell me 😊