Since plasma is straw-yellow in color when no red blood cells are present, wouldn't it be true to say that red blood cells (and the hemoglobin contained within them) give human blood its red color?

Yes that's true: Hemoglobin and myoglobin (a related compound found in muscle tissue) are both red.

Yes.: The red pigment is hemoglobin which is contained in the red blood cell.

Yes.: The red color is indeed from the hemoglobin in red cells. The yellow color of the plasma is from the proteins contained in plasma. How red the blood looks depends on the oxygen content of the hemoglobin. Oxygenated HGB is bright red, but deoxygenated blood, what you see in veins, is darker, and can be almost purple if there is very little oxygen. Hgb poisoned by carbon monoxide also is red.