A fellow quilter told me that using this mechanic's handsoap called "goop" or "orange goop" would take pencil lines out of fabric. It's designed to help mechanics get grease off their hands. Usually can be bought at Farm & Fleet, Menards, Home Depot, etc. I actually found mine at the dollar store. It's not the gritty kind - it's like a smooth cream.
I can't say I've tried it yet for pencil (since that quilt isn't done yet for washing) but I have used it for other gooey price tag removal, greasy spots, stains, etc. and they all come out beautiful. At minimum it shouldn't hurt the fabric. I trust it with my quilts. Good luck!