Capital One Late Fees, Membership Fees City Of Industry, California

 I had applied online for a Capital One credit card and was told that I was approved, and I would receive the terms with my card in the mail. The first correspondence that I received wasa bill from Capital One credit cards. It was informing me that I had a Capital One credit card with a $500.00 limit, a $39.00 annual membership fee, and a $15.00 late fee.
When I called their 800 number to decline the card, I was told that it had been longer than 30 days, and that even if I closed the account, I would have to pay the membership fee. I informed them that the first time I had seen any terms, was when I received the bill notifying me that I was past due. I was told to check with the post office to see if they had a record of my card being delivered (bogus) and told to send a letter to Capital One to see if they would reverse the membership fee, and close the account.
I do not want past due payments on my credit history, so I paid the $39.00 membership fee, and they waived the $15.00 late fee, but how do I get my money back and close the account?