Same experience here, although I wouldn't laud flash's performance on Windows either. Silverlight runs better than flash most of the time, and Silverlight is pretty terrible.
Compared to what? Silverlight uses a bit more CPU time when playing MP4 videos compared to Media Player or VLC but about the same as Google Chrome. It may be different on OS X. How does Silverlight's video performance on OS X compare to native Quicktime for example (which is relatively slow on Windows)?