Undervolting server CPUs
I got quite some CPU cycle requirements. For some time now I’m waiting for AMD to deliver sub 100 Watt (e.g. 95W) CPUs, but unfortunately the 95W editions of the Phenom II X6 1055T and 1065T are nearly impossible to come by (except you are an OEM ordering thousands of them). So I got a test system with the regular, 125W 1055T, and it just keeps running too hot.
While I’m still waiting for the Bulldozer to appear, and in a 95W flavor no less, I decided Sunday is a great day to experiment with undervolting the aging, hot silicon I got at hand. Actually it turned out the beast of a silicon I got can be undervolted by huge margins, from the factory settings:
NbVid NbDid CpuVid CpuDid CpuFid UNb CpuMult UCpu PCore
P-State 0: 32 0 6 0 17 1150.0mV 16.50000 1475.0mV 23305mW
P-State 1: 32 0 14 0 12 1150.0mV 14.00000 1375.0mV 20625mW
P-State 2: 32 0 18 0 6 1150.0mV 11.00000 1325.0mV 15370mW
P-State 3: 32 0 22 1 14 1150.0mV 7.50000 1275.0mV 10838mW
P-State 4: 32 0 26 1 0 1150.0mV 4.00000 1225.0mV 6738mW
NbVid NbDid CpuVid CpuDid CpuFid UNb CpuMult UCpu PCore
P-State 0: 36 0 26 0 17 1100.0mV 16.50000 1225.0mV 19355mW
P-State 1: 36 0 30 0 12 1100.0mV 14.00000 1175.0mV 17625mW
P-State 2: 36 0 34 0 6 1100.0mV 11.00000 1125.0mV 13050mW
P-State 3: 36 0 40 1 14 1100.0mV 7.50000 1050.0mV 8925mW
P-State 4: 40 0 52 1 0 1050.0mV 4.00000 900.0mV 4950mW
I’ll continue to stress this system over the next week, and hope I do not have to update this post due instability, … :-)
Update: So far stable for some 8 days compiling stuff. I now try with even lower voltages:
NbVid NbDid CpuVid CpuDid CpuFid UNb CpuMult UCpu PCore
P-State 0: 36 0 32 0 17 1100.0mV 16.50000 1150.0mV 18170mW
P-State 1: 36 0 34 0 12 1100.0mV 14.00000 1125.0mV 16875mW
P-State 2: 36 0 40 0 6 1100.0mV 11.00000 1050.0mV 12180mW
P-State 3: 36 0 46 1 14 1100.0mV 7.50000 975.0mV 8288mW
P-State 4: 38 0 54 1 0 1075.0mV 4.00000 875.0mV 4813mW
Looks like I effectively made a 95W CPU (I was longing for) out of it, … !1!
And this is a hexa-core: all those power consumption total multiplied by six! Imagine how much waste energy could be saved if the CPU vendors (AMD, ARM, Intel, Via, …) would program their final QA run minimally required voltages into the CPUs! Looks like this friendly silicon can run with about 20% less (waste) energy, …
PS: I’m using k10ctl for the live system MSR fiddling.
Update 2: Ok, with the last values I got rare “Internal Compiler Error”s (ICE) from GCC. So I spent some more time fine tuning this a little more automagically, Google’s stressapptest is a great help for it. Just limit your max CPU frequency scaling value, and up it up each strep will running it until you get either mis-compares, kernel MCE warnings, or sporadic reboots. Another note: Do not alter the northbridge voltage (too much, and only at the end of all test runs), it is not worth it and greatly contributes to instability. Without the (aggressive) northbridge tweaking I could even go further down with the CPU core voltage!
NbVid NbDid CpuVid CpuDid CpuFid UNb CpuMult UCpu PCore
P-State 0: 34 0 30 0 17 1125.0mV 16.50000 1175.0mV 18565mW
P-State 1: 34 0 36 0 12 1125.0mV 14.00000 1100.0mV 16500mW
P-State 2: 34 0 44 0 6 1125.0mV 11.00000 1000.0mV 11600mW
P-State 3: 34 0 52 1 14 1125.0mV 7.50000 900.0mV 7650mW
P-State 4: 34 0 64 1 0 1125.0mV 4.00000 750.0mV 4125mW
Oh, and note that P-State 0 is the Core Performance Boosting state, so make sure you tinker with state 1 (and best core boosting temporarily turned off) to find out the minimal voltage under max load of all cores, …
Update 3: So the final tweaked voltage values for now, that survived a T2 reference build without a glitch:
NbVid NbDid CpuVid CpuDid CpuFid UNb CpuMult UCpu PCore
P-State 0: 32 0 28 0 17 1150.0mV 16.50000 1200.0mV 18960mW
P-State 1: 32 0 32 0 12 1150.0mV 14.00000 1150.0mV 17250mW
P-State 2: 32 0 40 0 6 1150.0mV 11.00000 1050.0mV 12180mW
P-State 3: 32 0 50 1 14 1150.0mV 7.50000 925.0mV 7863mW
P-State 4: 32 0 60 1 0 1150.0mV 4.00000 800.0mV 4400mW
Lessons learned: tinkering with the north-bridge voltage appears to be a waste of time, and it is best to start with a reliable system load test and systematically lower the voltages and making proper notes while doing so. Saving a lot of energy, heat, and noise is easily possible.
Linux somebox 3.1.0-dist #1 SMP Tue Oct 25 15:50:26 UTC 2011 x86_64 T2/GNU/Linux
14:10:00 up 9 days, 2:35, 3 users, load average: 0.16, 0.30, 0.54
model name : AMD Phenom(tm) II X6 1055T Processor
Update: An older 905e I can not undervolt too much, so far down to:
NbVid NbDid CpuVid CpuDid CpuFid UNb CpuMult UCpu PCore
P-State 0: 36 0 34 0 9 1100.0mV 12.50000 1125.0mV 16313mW
P-State 1: 36 0 36 0 3 1100.0mV 9.50000 1100.0mV 13640mW
P-State 2: 36 0 43 1 12 1100.0mV 7.00000 1012.5mV 11441mW
P-State 3: 36 0 48 1 0 1100.0mV 4.00000 950.0mV 7505mW