What roles will the server run? If it's Microsoft-based roles, or roles using software applications which support Server 2008, then I'd suggest you go for Windows Server 2008. There is no major reason to go for an older version of Windows; in fact, installing the latest OS will put that particular server in good stead without the need to upgrade for several years.
>> If I do go with 2008, what will I need to do to prepare the domain for a mixed environment? This server will be a DC.
Simply prepare the domain prior to installing the server as a DC. ADPrep /forestprep followed by ADPrep /domainprep /gpprep from the ADPrep tool on the Server 2008 DVD media. You can then install the server as a DC, just like a 2003 server, and it will readily integrate itself into the Server 2003 network. No further work required to get it working as a DC.
>> The old server held the Operation Master roles, SQL databases, and Backup Exec to disks...all of which will need transferred to the new server. What's the best way to "replace" a server, including the DNS name/IP address? Software on all our clients is set to point to the old server name. Can I somehow use the same name?
You'll need to transfer the FSMO roles per
http://www.petri.co.il/transferring_fsmo_roles.htm over to the new server. The SQL Server databases would need to be taken down and migrated, as will the Backup Exec databases.
Only once the old server is demoted and removed can you consider referring to the new server using the old name. I never rename any server after it's promoted; I'd simply map a CNAME of the old name to the new name in DNS, add the old IP address as an additional IP on the new server, and add the old name as an additional NetBIOS name:
http://thelazyadmin.com/blogs/thelazyadmin/archive/2006/07/18/Configuring-Additional-NetBIOS-Names.aspx. This is sufficient and will enable the new server to still be contacted using its old names/IPs.
-Matt