I worry a little about the "like a food allergy" point of view...since I have friends with life-threatening food allergies, I'd rather just say that I'm a vegetarian, and that "anyways, I'm here for the company, not the food". I almost always eat before I go anywhere unless I'm sure that there will be vegetarian food available.
Like someone else mentioned, I do worry about being seen as picky, but lately I've been trying to speak up and remind people ahead of time if food is being ordered at work or suggest alternatives to going to a steak house for lunch. I wish someone had told me long ago that no one takes it personally and that it's better to say something up front instead of groaning while looking at the menu and being stuck with the only veggie option.
And I do wish someone had said it's more than just removing the meat and being left with the veggies and potatoes. It's literally a whole world of new food.