Have you considered asking the minister at your church how he/she feels about it? Since he/she is the one doing the speaking they might have an opinion that you should consider. I think it may depend a lot on the minister and type of church. I know my minister does not mind and there are a handful of people who regularly knit or crochet during the service, but I attend a pretty liberal, easy going, Unitarian Universalist church. I can think of other churches that I have visited where I would not consider knitting during the service. I seems like something that would be different for each congregation.