I am currently 23 weeks pregnant and have only gained 4 pounds. Is this ok for me and the baby?

Depends. That depends on your starting weight and nutritional status. Underweight women may need to gain extra. Overweight women may need to gain none. Individualized counseling is needed. Ask your doctor what appropriate goals and expectations are for you.