RMSC - stands for RingModulationSideChain

It's a procedure where the the Signal gets reduced by the exact Amplitude Value of the Sidechain Signal. This provides that the sum of both signals will never exceed the maximum Amplitude and won't start clipping.
It's typically used in addition to Sidechain Compression/ Volume Shaping to gain some more headroom in the mix. If you want a deep dive on how this works I recommend the tutorials of Borrasca, scarlvtt or Virtual Riot.

Backwards compatible with older Ableton Live Versions