To do a "Not all" is to make the claim that not all people in a certain category is in a certain way. While this claim is true for all categories except those for which it is tautologically false, the claim can still be anywhere between being an reasonably on topic objection to being an unreasonable level of derailment.
While a "Not All" can be used in regards to any category, the most famous version is "Not All Men".
Below follows three ways in which doing a "Not All" can be bad, good, or ambiguous - each of them using "Not All Men" as an example.
"Not all" as only unjustified derailment
When someone is bringing up an actual problem without generalizing about any category, others can pointlessly bring the conversation down by arguing as if generalizations had been made.
Example: After a woman has been sexually harassed by two different guys, her friends argue that everyone should respect each other and that those who go over other people's boundaries should stop doing that. Some guy finds this argument to be very offensive, and angrily retorts that "not all men are like that" - acting as if the friends had been arguing that all men are always sexual harassers and that no women are ever sexually harassers.
"Not all" as only reasonable objection
When someone is doing monolithization or spreading stereotypes, others can call them out on it and argue for more nuance. Those who are doing the monolithization or similar can then use the thought-terminating cliché of dismissing valid criticism as "not all X", as if that would somehow invalidate the criticism.
Example: A man who sometimes sexually harass women tries to justify his bad behavior by arguing that "this is just how we men are". A woman who grew up in a misogynistic family and is really tired of certain men who has been close to her agrees with him. Someone who's trying to stand up against this normalization of sexual harassment points out that most men don't behave like that, but gets dismissed by a "not all men" said in a mocking tone.
"Not all" in ambiguous situations
Situations are often much more complex than in the two examples above.
Example: A rant against sexual harassment has a lot of good points, but also uses an oversimplified rhetoric which could be interpreted as if all men would be guilty and as if all women would be innocent. Thus it becomes an issue of balance between upholding nuance and upholding momentum.
Identity fragility intersecting with the "Not All" fallacy
Lets take a story in three steps, with the character labels in italic...
Step 1.) Some people are doing categorist things against people who belong to a certain category of people, for example the local managers in a corporation refusing to give a certain employee a raise or promotion simply because this employee happens to be black or white or homosexual or heterosexual or Athist or Christian.
Step 2.) Someone speaks up against the discrimination in step one, this speaker tells about how the acts of the local managers (and of those who behave in the same way) are unfair both against the employee personally and to people of the employees category in general.
Step 3.) Some bystander feels offended by the speaker. This bystander is not employed at the corporation and doesn't know any involved people personally. Hoever, the bystander shares a category with the local managers, and feels that the speaker has somehow offended everyone in the category: THis bystander treats the speaker as if pointing out the unfairness of these local managers and of other people who behave in the same way would somehow mean that ALL managers/whites/blacks/heterosexuals/homosexuals/christians/atheists are mean people like that. So now the discussion is all about how the speaker is the villain and about how the bystander is the real victim here.