DJM 900 nxs 2 Record over USB

I am not particularly knowledgeable about OBS per se, but in general you need to follow the routing starting with the source (or the final signal, but source is usually easier). Leave your windows default to whatever it was before, you shouldn’t need this changed for your purpose.

You said nothing about players, so I am guessing you are doing everything inside the laptop.

In that case your source is the laptop. You can set the audio output from you DJ software to correspond with channels 1-4 (setting under the input tab of the DJM utility software and in the output section of the DJ software).
At this point you would hook up speakers to the main output of the DJM and be up and running. So, this would still be the first step. Fire up the DJ software, drop tracks in all decks you intend to use and hit play on them. Set trim/gain, listen for a signal in your headphones by cueing one of the channels. Check that all switches at the top of the channels are set to USB. By turning up your mixer channel faders and master level you should now have sound over your speakers. Or you can set your headphone cue/master mix to master if you don’t want to hook up your speakers.

If this works, the first part of your routing is ok. You get the sound from your DJ software to your mixer in the correct channels and can mix with it through the faders.

Now you should set rec out in the output tab of the DJM utility (which you say you have done). Effectively I would think (not sure though) that this will send the same record out (stereo) to each of the 4 available return channels. It COULD be that you get 4 different record feeds (one for each channel) but I don’t think so.

Now you should go into the OBS software and as input pick channels 1&2 from the DJM. If you are on Windows that would be an ASIO driver. Remember that record outputs are normally pre-fader (master) outputs. This means they will react to the channel fader levels, but are independent from the master settings. This is handy, cause you can turn the master up or down while listening (or answering a quick phone call) without affecting the level of your recorded mix.
So, play a track on deck 1 (corresponding with USB 1/2) and open the channel fader, again for monitoring purposes you can open the master as well, and check if you have incoming signal in OBS. Once that works, turn the channel 1 fader down and repeat for channel 2. If this also works everything is set up correctly and you should be able to play tracks in your DJ Software decks, mix them on your DJM and send out the final mix live to/through OBS.

Hope that helps some.