For a project I’m currently working on I needed a dynamic robots.txt. Because our test environment is public facing we want to keep it from being indexed by Google etc. It took a bit of Googling to find a solution that worked, but in the end it was actually pretty simple.
Also need to add the following to your web.config so that the robots.txt file can processed by the routing handler. Without this IIS will attempt to serve it as a static file and will return a 404 when it’s not found:
<system.webServer><modules><!-- ... --><handlers><addname="RobotsTxt"path="robots.txt"verb="GET"type="System.Web.Handlers.TransferRequestHandler"preCondition="integratedMode,runtimeVersionv4.0"/><!-- Other handlers --></modules><!-- ... --></system.webServer>