Much of this depends on you, the patient. But the majority of the decision falls on the expertise and experience of your plastic surgeon.
I ask patients to bring photos of women from magazines who are too small, just right, and too large for the patient's taste. This gives us both a visual representation of what the patient is looking to become. I also examine the patient, take measurements, and evaluate the patient's overall frame. Based upon these results, there are a limited range of sizes of implants that can be placed within any given patient that will achieve aesthetically pleasing results. It's impossible to go from a very small a cup to a d cup safely because the skin can only handle a certain amount of stretch at once. Too much of a size change is safely achieved with staged surgeries.
You should determine what type of look you're wanting to achieve. I've heard it said that there are three questions to ask a potential breast augmentation patient. Do you want to be noticed from across the table? Across the room? Or across the street? Comical as this may seem, announcing your choice also helps me as a surgeon to steer the patient in the right direction.
Your level of physical activity does impact the size of the implants you may choose. Runners may not easily become accustomed to the weight of large implants.
Most importantly is that the patient is happy, but the surgeon should be happy with his or her results, too. Plastic surgeons would likely agree that if a breast augmentation patient were to be unhappy with the size of her breasts after surgery, she usually wishes she would have gone with larger implants. But a good plastic surgeon can lead you to the ideal size for you when taking into account all of the factors.