Why.. do you think that? Maybe you need to ask yourself where you got that idea in the first place? You've already decided that it's not really true. At least that what you seem to be implying in your post.

Decide that only you can access your inner mind. Check out some websites for psychic protection. Some might offer some exercises for protection. Just doing these exercises can help you gain control. We often experience frustrations when we feel we cannot control something. So it doesn't matter if people are accessing your mind or not, you feel a loss of control and therefore annoyed. Just doing something to protect yourself will help you move past your annoyance.

Alex