What you do with your feet can be unconscious but quite distracting. In his book Power Speaking; The Art of the Exceptional Public Speaker Achim Nowak lists eight different types of footwork: the shuffle, the sway, the cha-cha, the lean, the lift-off, the cross, the bop and the step-away. In the sway the speaker rocks from side to side, as if he was standing aboard a ship that is rolling in the midst of a gathering storm at sea. You can see descriptions of the other seven on page 27, if you look in Google Books and search under footwork.
What you do with your hands can be even more distracting. You can click a pen, or twirl it like a baton. Hair can be adjusted. Women may twirl it, or smooth it over an ear. Men can also stroke their beard or mustache. Women may fiddle with their necklace or bracelets. Men can adjust their tie or rotate their cufflinks. They also may put both hands in their pockets and jiggle keys and change. Chris Farley (as motivational speaker Matt Foley) kept adjusting his belt.
When I was young, I wore glasses with plastic frames. Even though I added foam pads they still tended to slip down on my nose. Every so often I would push them back up. Long after I had switched to wire frames with adjustable nose pieces that didn’t slip, I still subconsciously tried to readjust them.
The first step to fixing these habits is recognizing them. Either videotape yourself and view your speech, or have someone else watch you. You probably will be surprised by finding habits you had not noticed.