We should all worry about mercury as it is a toxin.
Swordfish and mackerel are two that are known to be pretty high in mercury as is tuna. So the it is best to eat these kinds of fish no more than once a week. Try and get wild caught fish as much as possible. Look for fish that are not known to be high in mercury. Salmon is still a good bet as is sea bass, snapper, trout, grouper.