Right to remain silent beyond the matter at hand

If stopped by police for a traffic stop or other incidental contact, you often see the officer asking a slew of questions not related to the reason for the stop. (Where are you coming from?, what are you doing?, why are you here?, etc)

Does a person have to answer these ''fishing'' questions? And if not, what is a non comfortational way to state ones desire not to talk about matters unrelated to the specific reason for the contact?