Thanks you guys!
Okay, here's what I'm trying: I switched to size 6 needles, and started the back of the sweater again. I wanted a size medium, so I still stuck with that. But, after 40 rows, I spread it out and tried to measure the width (should be 25 inches) and it's not even 20 inches!
What should I do, just start over according to the directions for a size large? In that case should I stick with the size 6 needles?