You can limit the workflows context settings and such to make sure it's only getting triggered in certain ways, but there's no way to explicitly say "only via N/task"