No, you can't. You can change some of the parameters of the current VEX compiler scheduler, but if you want to write a new scheduler from scratch, you're out of luck...
You might be interested in researching about the ST200 compiler (or better the so-called LAO optimizer) from ST which was released in the open source (GPL) and might give you something to start from. The ST200 code is not the same as VEX code, but they're close relatives... Start from the http://www.stlinux.com page and work your way from there.
-- Paolo