Hard to find info...
Hard to find good, consistent info on penis sizes, but several doctors on healthtap have offered opinions.
It looks like the average erect one is 5-6 inches, so if one is worried about being less than 4 inches, he can talk with his doctor. When non-erect, perhaps under 1.5 inches would be small. In the end, function will trump size.