Wear and Tear. I think adding weight to the lower extremity (which is already bearing the entire body weight) is increasing wear and tear on the joints. Also additional pressure would be applied to the neurovascular structures on the inside of the ankle which could cut off circulation and increase the likelihood of tarsal tunnel syndrome.
Depends on what u do. If you are using them as part of an exwercise routine to strengthen your legs (sets/repetition type) usually no problem. Pedestrian (walking around) activity to increase caloric expenditure ok if comfortable fit. It is vigorous activity where it can become a problem firstly chaffing the skin, increasing likely hood of stress reactions and irritating tendons or nerves in the foot/ankle.
Injury. If you use too heavy of a weight you could hurt the muscles and ligaments.