Is polycythemia something that will go away on its own eventually?

It depends: If polycythemia occurs in a newborn, either as the result of maternal diabetes, placental insufficiency, or maternal-fetal transfusion at delivery (e.g., delayed cord clamping) then the extra red blood cells will slowly be removed naturally by the infant's body. There is no therapy required. Rarely, polycythemia is associated with an underlying hematologic problem, in which case it will persist.

Unlikely: Primary polycythemia, also called polycythemia vera does not go away by itself. Secondary polycythemia, often due to chronic respiratory disease is also unlikely to go away, short of a lung transplant.