The FSA said it had found “serious failings” in the sale of the products, designed to protect firms taking out loans against rising interest rates.
The FSA said it had reached agreement with Barclays, HSBC, Lloyds and RBS over providing “redress”.
The mis-selling is the third case of serious malpractice at the UK’s banks.
This comes after manipulation of lending rates and the PPI insurance mis-selling.
The FSA said it believed the swaps had had a “severe impact on a large number of these businesses”.
It did not say how much money would be necessary to compensate the businesses involved.
Sandy Chen, from Cenkos Securities, estimated that the total cost to the banks collectively would be between £1.1bn to £1.4bn, with most of that falling on Barclays.
Around 28,000 interest rate protection products were sold to thousands of small businesses, starting in 2001.
The businesses affected should now be contacted by their bank to instruct them whether they are included in a review of these sales. Those which were found to have been the victims of mis-selling will eventually be offered compensation.
The banks all released responses in the wake of the FSA announcement, saying they had co-operated with the FSA and would continue to work with it to resolve the matter. They said they had agreed to carry out a thorough assessment of sales of these products to certain customers.