Mad alchemist at'cher service! Let's see, here...
I used oranges we grew in our back yard and I did it according to the directions on this site:
Unfortunately, the site's not loading for me, so I have no idea what technique you used... but usually, there are three major ones...
* steam distillation
Used in industry, this is the really nifty one that uses a lot of test tubes.
* solvent extraction
Make tincture, evaporate alcohol. Simple!
* simple boiling.
'cause oil 'n water just don't mix... err, usually.
It smelled like wet dog AND I didn't get the layer of oil that it says I should see. Can anyone let me know if I did something wrong?
heh... well, *that* sure makes it sound like you used "simple boiling," which works... but I'm not sure.
Usually, there's a few simple things that can get in the way.
* how much orange peel did you use to how much water? In a *lot* of oil extractions, it's a whoooole lot of source to make very little oil.
* what's the pH of the water? Saponifiable (read : mildly acidic) oil + base (e.g. lye) = soap... which is *much* harder to isolate from water.
To fix the latter, try to dump in a little vinegar - it should crash the oil out of emulsion.
To fix the former... well, the *easy* solution is to use a *lot* more orange peels... but if you're still having difficulty, there's a few chemists' tricks that can help.
* transfer everything to a tall, thin column. A layer of oil that's less than 1mm thick when it's floating on the top of your industrial-scale cookpot can become several inches thick when it's floating on top of a 8-foot-long, 1cm column. For smaller jobs with higher yield, cylindrical flower vases work pretty well. For that matter, so do the necks of wine bottles...
* ablation. Ablation, ablation, ablation. Favored (and invented) by ~12th century alchemists, the original involved brushing feathers across the top of the cauldron to wick up the oils by surface tension. Currently, some cooking shops sell ablating wands for defatting which allow you to wring the fat out with a simple motion - probably requires higher yields than you're getting, but ya never know...
...soak the feathers in alcohol first (to get rid of any existing oils), wick up the product of a batch, stuff it in an empty, washed 'n dried wine bottle, preferably clear, 'n stick the cork on. Repeat until full... and then pour your boiling water in the wine bottle to the bottom of the neck or so, and suck up 50 batches at once in your eyedropper, rather than the pittance from just one or two.
...and, last but not least, you could always cheat - just float a bit of vegetable oil on the top, and infuse *it* with your essential oils. Not as good, and you'll have its residue left over if you try to use it as perfume or incense - but if you're cooking with oil of orange, it works pretty well, and it's still too easy to get it overconcentrated enough to be dangerous...
...the 'wet dog' smell you mention is probably due to short-chain, water-soluable organic acids (*most* things we recognize as "stench" are, in fact)... you can pull those out by taking your oil up in (dry) alcohol, running it over baking soda (which won't dissolve without water), filtering out the baking soda, and crashing out the oil from the alcohol with a little vinegar... stench-inducing organic acids bind to the baking soda (which is why it's used to deodorize), alcohol dissolves in the vinegar, and you're left with nice, clean oil, if you need it...
Hope this helped.