Is double vision after wearing goggles anything to worry about?

Unlikely. Double vision lasting only a few moments and specifically related to a particular situation - in your case, wearing goggles - is unlikely a significant problem. The goggles may be fitting too tightly or may unintentionally have a mild prescription in them that is straining your eyes. If the double vision persists or occurs at other times, you should have your eyes examined.
Could be. I am not sure, but if it is only after the goggles and goes away quickly-you probably don't need to worry. Quit wearing those goggles. Are they swimming goggles? With a prescription in them? You need an exam to be sure.