Payday loan debt is regarded as the bottom of the barrel. Most legitimate collection agencies won't touch it. Even a lot of Bottom Feeder third party collection agencies won't touch it. The collectors who do are usually the worst of the lot -- the criminals, drug abusers, high school drop-outs, etc. My guess is that when they initially contacted you they probably broke the law in one way or another.
Might be hard, if not impossible, to get your money back now that you've given it to them.
Your first course of action should have been to dispute the debt and ask for proof in writing. Debt collectors often call this a formality and pretend it isn't necessary or that they don't have to provide validation for a debt. Well, its not a formality, its your right.
As far as what you can do now... First file complaints with your local attorney general and the FTC. Most Attorney General's offices have a Payday loan division, or someone dedicated to dealing with Payday loans because abuse is rife within the industry. You may also want to contact the Better Business Bureau.
Lastly I'd contact http://www.budhibbs.com/ He works with consumers who have been ripped off by bad bill collectors and may be able to refer you to an attorney in your area.