This isn't really about DNS, but about how to have an application running on a custom port respond on port 80.
You have two options:
- Make your redmine installation respond on port 80 and serve incoming requests directly.
- Use a reverse proxy to forward incoming requests on port 80 to redmine running on port 5200.
Option 1 isn't doable if your web server already listens on port 80.
For option 2, the DNS entry for subdomain.domain.com should be configured with the server's public IP adress. On your web server, you should have an (empty) web site responding to the subdomain.
I never used IIS as a reverse proxy, but I'm sure this can be set up relatively easily. On IIS6 I'd recommend IIRF which can do reverse proxy.
Otherwise you may also use apache as a reverse proxy, and you may consider spinning up multiple redmine instances as well as serving static content directly from the webserver with aggressive cache headers. Here's an example apache configuration (from here):
<VirtualHost *:8080>
ServerAdmin admin@domain.local
ServerName redmine.domain.local
DocumentRoot "C:/redminepath/public"
<Proxy *>
Order allow,deny
Allow from all
</Proxy>
<Proxy balancer://redmine_cluster>
BalancerMember http://127.0.0.1:8081
BalancerMember http://127.0.0.1:8082
BalancerMember http://127.0.0.1:8083
</Proxy>
ProxyPreserveHost On
<DirectoryMatch "/(javascripts|images|stylesheets|plugin_assets|themes)">
<FilesMatch "\.(ico|pdf|flv|jpe?g|png|gif|js|css|swf)$">
ExpiresActive On
ExpiresDefault "access plus 1 month"
</FilesMatch>
</DirectoryMatch>
<FilesMatch "favicon\.ico$">
ExpiresActive On
ExpiresDefault "access plus 1 month"
</FilesMatch>
# Let apache serve the static content
ProxyPass /images !
ProxyPass /stylesheets !
ProxyPass /javascripts !
ProxyPass /favicon.ico !
ProxyPass /plugin_assets !
ProxyPass /themes !
# Proxy all other requests
ProxyPass / balancer://redmine_cluster/
ProxyPassReverse / balancer://redmine_cluster/
</VirtualHost>