I have a shriveled than normal part of my scrotum, what's wrong?

Normal?: The testes don't swell up when in an unaroused condition and thus the scrotum is usually somewhat shriveled. However if this is a new finding, check with your doctor to find out if you have testicular atrophy. I presume you are under age 40 but if you were over 50, testes may tend to shrink. Taking testosterone usually reduce testicular size as well.