Well, it's true some verbs are reflexives, but a verb is reflexive when the person who speaks it's the same who recieves the action of the verb.
In the sentece: "haberte contestado", i speak, but i don't recieve the action of the verb. This is not a reflexive sentence.
"Te" is the person who recieves the action, in this case, you (2nd). Sometimes, this pronoun (unstressed pronoun) is put next to the verb. If you change the pronoun and the person, you can't put the pronoun with the verb. For example: "tenía que haber contestado a la llamada de tu hermana" = "tenía que haberla contestado".
Saludos
P.S.: i have an exam tomorrow about this 
P.S.2: sorry for my english