You probably owe the money.
Your initial payments were calculated on you paying your payment on or before the due date each and every month of the loan.
There is no Grace Period for interest assessed. Never being more than 30 days late is not good enough. If you pay 3 Days late you will be charged 3-Days of additional interest. There are also late fees, most loans give you anywhere from 5-10 days before they assess a late fee on top of the additional interest.
While $4,900 does seem high, since you probably had a very high interest rate(being a sub-prime loan), it may be quite possible. Especially if you were late more than a few days, several times during the loan.
If you trace back ALL of your payment dates, balance, and interest rate and plug them into an amortization table you will probably find that everything does match up.