Is it a math learning disorder, something else or am I just not smart enough to remember numbers?

Hard to say. Not remembering numbers probably has more to do with short-term or working memory. A math learning disorder is diagnosed when someone has much more problem with basic math than would be expected for their level of intelligence--enough so to interfere with school or work performance--despite adequate instruction.