Could a shoulder dislocate and go right back in?

Yes, very common: You describe a shoulder subluxation, whereby the shoulder joint transiently moves out of place enough to give a sensation of pain (sometimes) and discomfort. A shoulder that slips out of place and requires a healthcare provider to relocate the joint is call a shoulder dislocation. Shoulder subluxations are much more common than actual dislocations.