Is it wrong? No. That means that you wish you were doing it.
Most of us should feel that way. I feel a little guilty leaving the house every day. Should you be disabled by it? No, not necessarily. If you have to work, or have important tasks at hand, help is fine as long as you are giving them the attention and love they need, and making sure the other caregivers are also giving that love.