You do know the story behind that verse, right?
The British commander, Alexander Cochrane, in the campaign that witnessed the bombardment of Ft. McHenry and the burning of the White House had tried to hire Americans to fight for the Crown and had issued a proclamation that any U.S. slave who left his owner and joined the british forces would be emancipated.
Just like Governor Dunmore had done in Virginia in 1776 and Abraham Lincoln had done in 1862.
People living in a slave society, whether they own slaves or not, do not take kindly to people trying to stir up a slave insurrection because slave insurrections tend to be indisciminate.
Hugh Forbes, one of John Brown's associates in 1858, had warned John Brown that a slave insurrection would “leap beyond his control, or any control,” because a slave insurrection was “from the very nature of things deficient in men of education and experience.” F. S. Key and his neighbors probably had similar concerns in 1814.