I have a hard time looking sideways and my neck is stiff. What's wrong?

Spondylosis. In part, it depends on your age. As we get older, we develop "wear and tear" of our cervical discs - the shocks in our neck. Bone develops around the areas of strain, which are often called spurs, and over time, we develop restriction in our neck movement. Plain cervical spine films can determine if you have this and to what degree. Physical therapists are often helpful.
Neck stiffness. Neck stiffness commonly is from degenerative changes involving the spine or the muscles associated with it. Less common causes include life-threatening conditions, such as meningitis or aneurysm rupture, which almost always include a lot of other symptoms as well.