Pope Francis said Thursday there are limits to freedom of speech, especially when it insults or ridicules someone’s faith.

Francis spoke about the Paris terror attacks while en route to the Philippines, defending free speech as not only a fundamental human right but a duty to speak one’s mind for the sake of the common good.

But he said there were limits.

By way of example, he referred to Alberto Gasbarri, who organizes papal trips and was standing by his side aboard the papal plane.

“If my good friend Dr. Gasbarri says a curse word against my mother, he can expect a punch,” Francis said half-jokingly, throwing a mock punch his way. “It’s normal. You cannot provoke. You cannot insult the faith of others. You cannot make fun of the faith of others.”

His pretend punch aside, Francis by no means said the violent attack on Charlie Hebdo was justified. Quite the opposite: He said such horrific violence in God’s name couldn’t be justified and was an “aberration.” But he said a reaction of some sort was to be expected.

Many people around the world have defended the right of satirical magazine Charlie Hebdo to publish inflammatory cartoons of the Prophet Muhammad in the wake of the massacre by Islamic extremists at its Paris offices and subsequent attack on a kosher supermarket in which three gunmen killed 17 people.

Others, though, have noted that in virtually all societies, freedom of speech has its limits, from laws against Holocaust denial to racially motivated hate speech.

Recently the Vatican and four prominent French imams issued a joint declaration that, while denouncing the Paris attacks, urged the media to treat religions with respect.

Francis, who has called on Muslim leaders in particular to speak out against Islamic extremism, went a step further Thursday when asked by a French journalist about whether there were limits when freedom of expression meets freedom of religion.

“There are so many people who speak badly about religions or other religions, who make fun of them, who make a game out of the religions of others,” he said. “They are provocateurs. And what happens to them is what would happen to Dr. Gasbarri if he says a curse word against my mother. There is a limit.”

In the wake of the Paris attacks, the Vatican has sought to downplay reports that it is a potential target for Islamic extremists, saying it is being vigilant but has received no specific threat.

Francis said he was concerned primarily for the safety of the faithful who come to see him in droves, and said he had spoken to Vatican security officials who are taking “prudent and secure measures.”

“I am worried, but you know I have a defect: a good dose of carelessness. I’m careless about these things,” he said. But he admitted that in his prayers, he had asked that if something were to happen to him that “it doesn’t hurt, because I’m not very courageous when it comes to pain. I’m very timid.”

He added, “I’m in God’s hands.”