I'll direct you to here.
Slide the voltage to 100%, slide the power target to 120%.
Start with the Core Clock.
Open a game or a benchmark, ALT+Tab out of the game/benchmark and slide the core clock up 10MHz and click apply, Alt+Enter to return to the game, play for a minute to see it's stable. Keep doing this until it crashes or artifacts then back it off 10MHz - 20MHz. Most if not all GTX 1080's can hit 2000MHz, the good ones can overclock to 2100MHz - 2150MHz.
Then do the Memory.
The memory overclocks very far on Pascal you should be able to overclock the memory anywhere from 11000MHz - 11600MHz, do just like above in 100MHz increments then back it off 10MHz until it stops artifacting/crashing. Some games/benchmarks will vary on the overclocks.
Let me know what you end up with.
Do you have the 7700 or 7700k ?