I have warts on my penis for years but they never grow any bigger they're just there, what can it be?

Condylomata. Condyloma can stay the same size for years. Molluscum can also do this. Uncircumcised men can also get warty lesions on the crown that are just keratin overgrowth.
Warts. As you mentioned, these can be warts. There are probably over 60 different strains of the virus that causes genital warts and some of these viruses are slow growing and may not change over time. These lesions can also be some other type of benign lesion as well. For definitive diagnosis, a biopsy is recommended. If these are warts, they are transmissable to your partner, even if they don't change.