You can freeze to a specific version of Rails.
If you've got shell access and subversion, the easiest way is to export the version you want into /vendor/rails:
svn export
http://dev.rubyonrails.org/svn/rails/tags/rel_1-2-5/ ./vendor/rails
Run that command from the root of your project.
Or, download it locally and upload it.
When the rails server starts up, it looks for that folder and uses the rails version in it if it exists.