I had cat scan showed I had a tumor on adrenal gland 5-6mm in size is that something to worry about?

Not to worry but. This size is not too large to be overtly concerned as incidental adrenal masses are common nowadays as more people are subjected to ct scans for various reasons. However, i don't think this should be ignored and you will need basic testing to make sure this "incidentaloma" is in fact the case and to rule out any "hyperactivity" coming from the adrenals such as excess aldosterone or cortisol.
Check the report. If the tumor was really only 5 millimeters, it is probably nothing to worry about; however, I am guessing it was actually 5-6 centimeters. If that's the case, then the first step is to have blood tests done to determine if it is a functioning tumor, meaning does it secrete high amounts of adrenal hormones. Even if it does not, there are some ct features that favor removal at this size.