“That's what America is about. A land of dreams and opportunity. There were other immigrants who came here in the bottom of slave ships, worked even longer, even harder for less," said Carson, who is black and ran for the Republican nomination in last year's presidential contest, during ad hoc remarks at the department.
"But they, too, had a dream that one day their sons, daughters, grandsons, granddaughters, great grandsons, great granddaughters might pursue prosperity and happiness in this land," added Carson, who was confirmed by the Senate only last week.
Slaves, of course, had no say in coming to the U.S., nor were they able to imagine any prosperity for their children who would be born into the now outlawed institution.
Carson's department denied he intended to liken slavery to unforced immigration.
"This is the most cynical interpretation of the Secretary's remarks to an army of welcoming HUD employees," it wrote on Twitter. "No one honestly believes he equates voluntary immigration with involuntary servitude!"
Nonetheless, Carson's comments are likely to feed into public discontent among some parts of America that feel alienated or targeted by the nascent Donald Trump administration.
Hollywood star Samuel L. Jackson was unforgiving.
"OK!! Ben Carson....I can't! Immigrants ? In the bottom of SLAVE SHIPS??!! MUTHA****A PLEASE!!!", he wrote in a Twitter post retweeted tens of thousands of times.