I can feel your pain. I'm a guy who knits (although I'm better at crochet) and I get teased about it all the time. I work in an office and all the guys there tease me. I get looks from people at the craft stores, because here is this guy with tattoos getting excited over yarn sales and new cool magazines. My wife loves it though. All of our friends call me the male Martha Stewart, and I don't know if that is a good thing or not.
Keep it up though! It's fun stuff.