Most likely not.
The second testicle (if normally functioning) is able to "pick up the slack" and produce adequate testosterone
without a change in your "manhood".
However, if there is already an issue with the testicle (such as it is small or has sustained prior injury) then it may not produce normal amounts of testosterone. All things equal though with a healthy remaining testicle, there should be no changes.