With the worse mod you're using more GPU power as that mod enables many effects that were previously disabled so it isn't really surprising seeing drop in performance.
The audio is bad because it only records it at 96kbps which is ridiculously low, so it's not really surprising it's as bad as it is.
As for the FRAPS go, it will decrease your FPS by far much than any other recording software. You need to have extremely fast hard drive (Preferably ssd) and CPU (4 cores, 8 threads, 4+ghz) because of it's heavy cpu encoding it does.
If you don't find Shadowplay good to record with I recommend you try using MSI Afterburner for many reasons:
1) It's free
2) It uses CPU encoding (The opposite of Shadowplay that uses GPU)
3) It's much better than FRAPS in view of CPU encoding as you have more options on how to record, at what quality, resolution, etc.
4) It doesn't fill up your disk as fast as other programs
If you don't like MSI Afterburner you still have Bandicam (Which I use as well in combination with Shadowplay as I can use any codec there is avaible and tons of options for customization),then you have Dxtory and that's where the list of decent recorders end.
Hope this helped.
Cheers
Asus Rog G750JZ
Screen: 1920x1080 17.3'
CPU: Intel® Core™ i7-4700HQ (3.4Ghz @ -70mV)
GPU: GeForce GTX 880M 4GB (OC +125Mhz Core, +600Mhz Memory @ 1.062V)
RAM: Samsung 32GB 1600MHz
Storage #1: Samsung Evo 840 250GB
Storage #2: Samsung Evo 850 500GB
Storage #3: Seagate Barracuda 2TB
Storage #4: Samsung Evo 128GB MicroSD