My 5 yr old son has never been vaccinated other than what was given at birth, yet he's never been sick other than an occassional cold. Is this safe?

Not recommended . Vaccines are extremely safe where the benefits far out weight the risks of vaccination. It would be devastating to lose a child from a preventable illness just because a vaccine was not given. Nonetheless, oat physicians will also respect the personal convictions of their patients (as long as the risks and benefits have been fully explained to you.).
Well... You and him are walking a fine line. The fact he's been healthy may stem out of the fact that he's been around children/folks who are vaccinated, so the prevalence/incidence of a particular disease is small in that community. If he were to happen to be in a situation where there are a lot of unvaccinated folks like him, he may contract a potentially preventable disease and only god knows outcome.