How do I know if my elbows are just really sore or if I have tendinitis?

Have those examined. Lateral epicondylitis (tennis elbow) and medial epicondylitis (golfer's elbow) are among the most common causes of elbow pain. The pain will be around the bony prominences called epicondyles. Frequently patients will have pain in their elbow when they make a fist. It's common in epicondylitis. You may try ice, nsaids (ibuprofen/naproxen), a tennis elbow band. If none works, see your doc.
Both will cause pain. The pain is usually worse during and after activity with tendonitis. Tendonitis usually happens around the tendons area.