On the DJM800 plug into the RCA jacks marked Rec Out, not into the master or booth out.
If you don’t have a compatible soundcard on some laptops the mic input jack will also accept a line signal. You could try to connect that with an RCA to mini-jack cable. If that works then you are in business with free software like Audacity.
If that doesn’t work, you could either look at getting a soundcard (doesn’t need to be a DJ card) or what I would personally recommend is buying a pocket recorder. I own the Zoom H1 (https://www.long-mcquade.com/10403/Pro_Audio_Recording/Multi-track_Recorders/Zoom/H1_Handy_Recorder_-_Black.htm) and can plug it into anything I might want to record, then I just copy the mp3 file over to my computer later. It is not an expensive little gadget, and comes in handy all the time.