How about a "vote of confidence"? If a certain percentage of the constituents sign a petition for a vote of confidence, they hold a special election to see if the candidate can get enough "yes" votes. If not, they lose their seat mid-term and get replaced by a second special election.
