I just wanted to chime in that I hardly ever consider time when determining swap size. If I make a simple quilt for a swap it's for a random swap where there are no sizes or points. That solves that for me lol. Usually I start a medium and by the time I'm done I realize it's probably more of a large.
On the other end there are also things that take no time at all but you know your partner will love. I usually try to judge by amount of time on those and how many would equal a small. Simple polymer clay and simple earrings, things like that.
I also err on the side of spoiling.