Was wondering if I only wear my top retainers (invisalign), will it affect my bottom teeth?

It could. Well aligned teeth tend to support each other. So if your lower molars and bicuspids fit tightly together when you close (with the aligner out), keeping the uppers in place may help keep the lowers in place as well. However, better to have a retainer in the lower.
Top and Bottom. When wearing a top retainer only , you cannot prevent your bottom teeth from shifting. By wearing both retainers top and bottom, you will be rewarded with a beautiful smile for a lifetime.
Wear both retainers. You spent a great deal of time, effort, and money to correct your malocclusion. Why risk relapse? Because clear aligners do not achieve the same level of root alignment and bone support as braces, the intensity of retention is even more important.
Depends. To ensure teeth stay in place I would suggest wearing both retainers. I personally had a glued in "permanent" retainer for 7 years, and when mine was lost the teeth began to shift within 3 years. So I would suggest at least a few nights a week, wear both.
Possibly. The lower teeth will function on the plastic. That's not a good fit. However , if worn only at night, there may be no effect.