A toggle button is a button which stays down when clicked once and goes up when clicked a second time.
When to use
Use a toggle button to indicate a state.
Example: A word processor should use toggle buttons to indicate the state of "Bold", "Italic" or "Underline" formatting.
When not to use
Do not use a toggle button to indicate an action.
Example: A music player should not use a toggle button to implement a Play/Pause button. It should use a normal button and adjust the icon and label to represent the action which would be performed when clicked.
Icon and label
Never change the icon or label of a toggle button. A toggle button is very similar to a check box. It should thus be used in the same way.
The fact that it is pressed down is enough to indicate the current state of the element it represents.