See below. You are an adult. Why would you use a pacifier.
YES! Any pernicious habit (chewing on pencils, fingernails, your hair, etc) can damage braces and/or interfere with tooth/jaw movement. I don't understand why a 32 year old would evidence the behavior of a toddler. This is distressing. Perhaps you should look into why you are still behaving this way, and get help addressing the problem. Don't ignore this.
Why? If you're wearing braces to straighten your teeth, why would you want to stop the straightening process? Counterproductive, isn't it?