In most C-like languages that would be a syntax error. E.g. in C and C++ as a rule you tokenize "greedily", "1--2" would be tokenized as "1", "unary decrement operator", "2", which is illegal because you're trying to decerment an rvalue.
Python doesn't have "--", which allows the tokenizer to do something else.