Well, I often take the side of the tough-minded, but I have to agree with the tender-hearted this time. The phrase "inmates running the prison" is a terrible metaphor in context. Inmates running the asylum is the phrase, and even then it might be insensitive. Inmates running the "prison", for a group of black men protesting, are you kidding me??! This is a gaffe beyond measure. Like jbama said, with prison being how it is, that just piles on. It is more than stupid. It is racist. Not racist = lynching. But racist = privileged superiority, yes, it is. It is a hostile phrase in context. I'm sure the man didn't "mean" it that that, but that callousness and lack of awareness of the meaning is pretty much what people mean by "soft" racism. As a person who is not pro-kaepernick and not a fan of kneeling during the anthem, I can say this comment is bad and should bother objective people.
Edit: I want to add, about the OP question about inferring meaning, I agree with the general point. We should definitely put things in context. This time is no different. It's just that the more you infer meaning, the worse this one gets. I want the world to have more space for a good gestured joke, some clever humor, and biting political sarcasm. But to preserve the space for serious and friendly humor we do have to draw the line somewhere when the time comes, and I would draw it here. This was not funny, witty, clever, or poignant. It was mean and stupid.