Enslaving people because of their race in a country that was otherwise prosperous, and hiring children in an economy and society where any thing else is nearly impossible are two entirely different things.
Slavery was the primary economic engine for a long, long time in US and its originating colonies and it was a primary reason for the economic prosperity and viability of colonies.
I agree with you that they are not the same, but both political and social wrongs that can and should be addressed (yes slavery still exists in the US, both de facto and actual i.e. human trafficking).