You're getting these numbers from task manager? I would think add the virtual memory column and show all processes would indicate the memory hog. We had a couple sql boxes that seemed like sql server was reluctant to yield memory to the OS while set on autopilot, so the OS was causing heavy paging when something else ran. Just changing the sql limit to a few GB less than the physical memory helped in that case.
