Are allergies or colds more often the cause of swelling of the nasal passages?

Colds. Both allergic rhinitis (hay fever) and a viral upper respiratory tract infection (a "cold") cause nasal congestion. Almost all people get colds; about 30% of the population suffers from allergic rhinitis. With allergies you never get a fever and any mucus drainage is clear. With both you get sneezing. Allergies typically cause more itching of nose and eyes.
Colds. Colds are the most common cause of swelling of the nasal passages, but persistent symptoms that do not start to improve after 10-14 days may represent a bacterial infection of the sinuses or allergies. Nasal symptoms that occur after exposure to pets or dust, or recur at the same time of the year, every year, may be allergies.