As a martial artist, angler and wargamer I have been raked over the coals by the sandal & lentils brigade before. The answer is to extol the positive side of your hobbies, ones I have used are:
Fishing - patience, environmental awareness, ability to adapt to different situations and ability to manufacture solutions.
Martial Arts - self-reliance, character judgement, health benefits, ability to cope with hardship and stress, not put off by scale of tasks.
Wargaming - strategic thinking, dealing with people in an adversarial context, attention to detail, decision making.
One person once told me to my face that violent people like me should be put down to which I replied (rather smugly) that one sentence showed he was many times more violent than I am. I got a written apology from the company several days later but I didn't get the job. Principles are worth more than a job anyday.