HAPEVILLE, Ga — A restaurant owner is flying a controversial banner on top of his restaurant after he was denied an alcohol permit for reasons he calls “racially motivated.”

“They’ve called my customers thugs, gangsters, low life, no class,” says Joshua Patton, owner of The Castle restaurant in Hapeville, Georgia. The new sign hanging above his restaurant has the names of City Council and Alcohol Review Board members beneath the message, “These citizens don’t want black people in Hapeville!”

“They flat told me we don’t want ‘you or your kind’ in this city and what I meant is African-Americans,” Patton says. He adds most of his customers are African-American and he’s upset about the things said about them.

CBS reporters say the Alcohol Review Board claims they denied The Castle’s permit last year for reasons including material misrepresentation. The Board says it was not convinced on a traffic issue and that granting the license would not be “in the best interest of the health, safety, or welfare of the city.”

Patton says he will continue fighting and is not scared to be vocal. “It’s wrong and that’s the reason I’m fighting, I’m fighting for my customers.”