I had to switch to a X570-F Board (coming from a Intel-machine). Board and CPU are running fine, the RAM is running at 2133 MhZ and I dont have any idea how I have to configure it in Bios for at least running at 3600 MhZ. The RAM is G.Skill F4-4266C17D-32GTZRB. When setting DOCP to Auto and using the 4266-Preset the board does not boot.
Do not try to use the D.O.C.P settings, the voltage will not be high enough to get the 3600 MHz or 4266 MHz speeds. You will need to set the RAM settings to Manual and then select what speed you want under RAM Frequency and then set FCLK to 1/2 of the RAM speed you selected. You might also have to go under the Advanced -> Overclocking section and set the RAM Frequency there as well to match the AI Tweaker section's values. Leave everything else as Auto except the DRAM Voltage near the bottom of the AI Tweaker section. I eventually had to set my DRAM voltage to 1.5 V to get my G.Skill TridentZ 4000 MHz to consistently boot at that speed. My RAM's D.O.C.P. settings are 17-17-17-35 1.35 V and could never get those settings to work at all, just like the problems you experienced with your RAM. I looked at the newest specs of the AMD specced G.Skill RAM of mine and now they are setting the D.O.C.P voltage to 1.5 V and the CL timing at 28. That is where I came up with the DRAM voltage of 1.5 V to try on my RAM and it has been working without crashing for several weeks now. Give this a try and see if you can at least get the 3600 MHz speed to work for you.