Is it possible for liver cancer tumors to be too big for a transplant?

Milan criteria. Most transplant programs adhere to some form of the milan criteria that restrict transplants to persons with one tumor<5 cm, up to 3 tumors<3cm, no extrahepatic disease, and no vascular invasion. Size is a marker for vascular invasion. Some programs have adjusted these criteria. Some tumors not amenable to transplant may nevertheless be resected (removed) surgically.
Try to shrink tumors. Transplant is the best treatment for certain liver cancers. Though all centers have criteria for acceptable tumor size in liver transplantation, there are mechanisms to improve the chances of approval for transplant. Liver directed therapy can be used to shrink tumors - embolization, radiation, etc. If they shrink, you may be reconsidered for transplant at some centers if within criteria.
Yes. According to the current tnm/uicc staging system for hepatocellular carcinoma most consider stages iiib, iiic, and stage iva, as well as stage ivb, disease to be incurable by resection.