Jersey would be crazy tricky to get to keep the shape. You'd have to interface it with something super stable. Like canvas. Or steel.
I've got superboobs too, so I fear the keyhole top, but I agree it's adorable. Maybe you can do it up in a heavier cotton, so it would still be pretty comfy, but keep its shape. You could still use jersey for the band/sash and the skirt.
Oh, I'm already drawing it up in my head! With piping! And interfacing. Lots of interfacing...