Perhaps you are confused about the constitutional right to free speech. It does not give people the right to give insult to others without receiving insult back. The constitutional right to free speech protects people from repression by government.
It's Nazis who disregarded agreed upon entry and exit paths to the park, making it impossible for the police to keep them separate from counter-protesters. The Nazis sought confrontation. It backfired on them when their actions were met with effective defense and caused the police to end their protest.
I'm not making this up. The captain of the Charlottesville police force said as much.