Zephyr supports supervisor mode. I want to nullify it at software level.
Where did you find this information? For RISC-V SoCs, the supervisor mode is only used when the base architecture has the 'S' extension. Since there is no Supervisor enabled SoCs supported by Zephyr so far, the core RISC-V code doesn't care about it.