Maybe let the knitting lie in your lap as you go so you don't feel like you have to "hold" all the needles. Remember, you are only using two at a time.
The hardest part of using dpns is the first few rounds, before the knitting is strong enough to support the needles in a round shape. But even on those rows, you don't need to hold all the needles at once. Just use two, like you're knitting flat, and let the other two dangle. If you push the stitches on the two "resting" needles to the center of the needles, they should be fine all by themselves.
It sounds like you're using the "backwards loop" cast-on. Yeah, it has that problem, where the cast-on starts to stretch out as you knit with it. You can either use a more structured cast-on, like the long-tail cast-on, or you can just ignore the tail, push the two needles together, knit the stitches, and move on. At the end, you can go back and tighten up the cast-on edge.
I think the poster of this thread
had the same problems as you did. Although her pictures don't show up any more, the advice might help you, too.Here's another one, with pictures.
You're not alone!