I had the same problem.. and my case is miserable for airflow ( Coolermaster Cosmos SE )... I simply changed the fan settings and now it's awesome... and the fan isn't really that loud on the GPU...
in fact, if I run it for mining I set the fan manually to 85%. keeps it great.. VRAM gets hot while mining, and I mean really hot... but that's about it.
side note... what temps are ok and expected for VRAM?