I'm suprised at the number of people who say they knit in class. I can't imagine that if I'd tried that, my teachers wouldn't accuse me of not paying attention. Not to mention the fact that it's tough to take notes while you're knitting!
Yeah, as a university instructor, I would ask someone to leave if they pulled out knitting in class. To me, it would be the same as texting or browsing the internet - I would interpret it as disengagement with the rest of the class. Knitting while sitting around chatting with friends is one thing (I do it, and know that many people don't have a problem dividing their attention in that way), but I expect my students to be listening, thinking, and participating very actively in my classroom.