لَن يَضُرُّوكُمْ إِلَّآ أَذًى ۖ وَإِن يُقَـٰتِلُوكُمْ يُوَلُّوكُمُ ٱلْأَدْبَارَ ثُمَّ لَا يُنصَرُونَ Qur’an Aal-Imran (3:111)Lan ya d urrookum ill a a th an wain yuq a tilookum yuwallookumu aladb a ra thumma l a yun s aroon a
As is obvious from the opening sentence of verse {110}, this promise to the followers of the Qur'an is conditional upon their being, or remaining, a community of people who "enjoin the doing of what is right and forbid the doing of what is wrong, and [truly] believe in God"; and - as history has shown - this promise is bound to lapse whenever the Muslims fail to live up to their faith.
i.e., they can only annoy you with their words.