Depends on what you exactly mean with "rolling back segwit". 

- Counting witness bytes as 1 vB instead of 1 wu would be a soft fork. 
- Forbidding segwit outputs and/or inputs would be a soft fork.
- Removing the segwit rules and returning segwit outputs to being anyone_can_spend would be a hard fork.

Murch

bitcoin

Powerful arguments for the 'meaning' of Bitcoin - What is going on?

028559d218

> Clients that had not upgraded would treat new segwit transactions @ prior costs as valid, would consider them part of longest chain.

Fees and tx standard rules such as `OP_RETURN` size and count are simply relay/inclusion policy, not block consensus. So you need just a single path to a similarly configured miner to push and get it included. You'll wait longer of course if only a subset of miners accepts it under their policy. This is why someone running Peter Todd's libre relay fork can actually push a bigger `OP_RETURN` today. F2pool seems to allow these as they mined [the taunt](https://mempool.space/tx/7776c1ffd8747a5ee6199974e473709607f8fab009c58fffc8a1d30b49f6a691) the other day. The same goes for changing fee calc.

> this is why it's considered impossible to roll back segwit without a hard fork.

This part is correct. 