I am not concerned about credit card fraud. You have 100% protection in US, especially with Amex.
The biggest issue is identity theft. I have seen people suffered from identity theft and the their credit reports are ruined.
Scammers could have used your identity to get a new credit card. You just won't notice if you don't check your credit report frequently. It is a big hassle to dispute your credit report.
I don't even use my billing address as my shipping address. I don't think our dealers will sell our information, but you just never know how safe these websites are.
Better safe than sorry.
FYI: you can get free credit report here:
https://www.annualcreditreport.com/cra/index.jsp