How much should I worry about trouble breathing and fever?

Fever and dyspnea. A combination of fever and breathing difficulty is a serious problem. There may be underlying infection that can, and should be treated. I would encourage you to visit your family doctor or, if you don't have a family doctor, go to a local hospital emergency room. I am not so sure that a minor er will be suitable, since you may require hospitalization if the infection is serious wnough.
Breathing problems. If you are experiencing shortness of breath, you should seek help promptly. If it is associated with significant fever, you may very well have pneumonia, and that will require an antibiotic. Other potential causes include a blot clot in the lung, tumor, and other inflammatory conditions of the lung. So you should worry.