From what I understand, both extreme leftists and extreme righties want the same thing: a person's freedom to do what they choose as long as it doesn't infringe on the rights of others. The difference comes in the execution: ELs feel it's a general consensus that determines those freedoms, and as such laws and acts must be in place to ensure the consensus is upheld. ERs feel it's an individual's choice, and believe laws and acts should only come into play when another person's rights are directly violated.
Bringing it back to pot, ELs feel that there is a consensus in this country that dictates this is a substance that should be legalized, thus legislation is needed to make this so. ERs would state consensus doesn't matter; if an individual wishes to smoke pot, so be it, and any laws pertaining to its consumption should be limited to consequences that would affect others' rights (such as DUI).
So in a sense, OP, leftists ARE trying to "take away" freedoms, in so much as they believe (whether mistakenly so or not) that as a Nation, the United States must create and follow laws that it's people deem appropriate. Meanwhile, righties ARE trying to maintain a "static and unchanging" view of the Constitution because, in their view, a simple document (and as such one with fewer laws) provides a greater degree of freedom for the individual.
I could wax on forever, but really what it comes down to is compromise instead of name calling and idea bashing. Cause in the end we all just wanna chill on the couch and get high as shit.