My varicocele got bigger, so should I do something about it?

No, as long as... Do nothing as long as being asymptomatic and having no concern about male infertility; it will not affect your quality of life and longevity.
Maybe. Would fix only if symptomatic causing terrible pain or fertility issues or if it bothering you cosmetically.