Hey Gabor,
I am a bit lost as to what you are trying to connect. The only RCAs on the Mixage are the outputs. And the only place your outputs should go is to your speakers (Rokit 5s in your case).
I have no clue what you are trying to accomplish, but running any kind of cable from the output RCAs to your laptop is bound to be trouble.
The Mixage connects ONLY with a USB cable to the PC/Laptop.
Hope that helps. Or give us more (precise) information so we can help you better.
Greetinx.