For a migraine, is it ok for me to take baby aspirin if that's all I have at home?

Yes to prevent. Yes it may prevent, but you may have to take 8 pills to treat.
OK but won't help. If you're trying to treat the pain of a migraine (?), you'd have to take an awful lot of baby Aspirin to get where you want to go! (at least 12 of them). Why not buy some Ibuprofen - it's not expensive.