Dear Grant,
If you can honestly recognize that your inner voice is telling you one thing about yourself, even though someone else says differently, then what does it really matter what that other person thinks or says?
Of course, if what that person says is objectively true and was offered in a helpful spirit to help you recognize a blind-spot with the intention to help you grow and patch up that deficiency, then it would be wise to consider the external feedback. We are all growing, after all.
I can fully appreciate your situation, though, because I grew up in a fundamentalist, Christian family who placed heavy reliance on the voice of spiritual leaders. My folks had the unfortunate habit of patterning their lives on what the "prophets" in our church had to say about them. In a sense, they were allowing the Church to direct details of their lives, and my folks sincerely believed they were following the will of God as revealed through His prophets. This fit into their world-view just fine, and they are very happy and satisfied to have this kind of guidance in their lives. But to me, this flew against the face of all reason. When I matured and gained independence, I had to decide whether to be true to myself or to break with the mind-set of my core family (and thus officially label myself as an outcast along with all of its consequences.) Well, I chose the route of integrity, because I hear my own inner voice, and I sense that it is right (for me.) At first, this caused quite a stir in my family, but over time, things settled down, and I believe we have all grown through the experience. My family still worries about my soul, though, ha ha.
Anyway, back to your situation ... it really boils down to rising to the level of becoming self-actualized (a.k.a. Maslow). When we reach this higher level of living, we walk according to how our hearts direct, and it doesn't matter what others think or say. In fact, we don't even notice.
Yes, we all have a long way to go before attaining such an enlightened mode of living, but as long as we strive toward this end, we are on the right path.
Hope that helped a bit, and do let us know how you're progressing,
HF