What did the original contract say? Depending what you signed, you might owe them.
A big point though, in that letter it says they needed your color selection for roofing and siding and they needed to measure for windows. That says whatever you signed didn't cost them a penny.
That is an important point in whatever you do. If they had ordered products and were stuck with them then for sure you owe. Because that didn't happen...
What did the contract you signed say?
A funny thing I noticed. The letter says the color selection must be "signed off by both of you". Did both of you sign the original contract?