Would being abused as a baby have any effect later in life, even if you were too young to remember it?

Yes: Child abuse is "remembered" as a "file" in the "hard drive" of the brain below conscious level.It can influence behavior through life. A preventive approach, strange as it may seem is through proper attention to the "fuelling" of the brain through pristine diet.

Could affect you.: A history like that could have long-term ramifications in your adult life and should be brought to the forefront through psychological counseling. Those memories do get burned into your brain, but could be hidden deep within and won't go away unless you address them.