To change the resolution of a Windows Server 2008 Core, you have to change two reg-dwords.

On cmd type regedit.exe

Nav to HKLM\System\CurrentControlSet\Control\Video

Change following values for the graphiccard (Attention: Values are default in HEX, change to decimal)
DefaultSettings.XResolution
DefaultSettings.YResolution