For a project I’m currently working on I needed a dynamic robots.txt. Because our test environment is public facing we want to keep it from being indexed by Google etc. It took a bit of Googling to find a solution that worked, but in the end it was actually pretty simple.
Also need to add the following to your web.config so that the robots.txt file can processed by the routing handler. Without this IIS will attempt to serve it as a static file and will return a 404 when it’s not found:
<system.webServer><modules><!-- ... --><handlers><addname="RobotsTxt"path="robots.txt"verb="GET"type="System.Web.Handlers.TransferRequestHandler"preCondition="integratedMode,runtimeVersionv4.0"/><!-- Other handlers --></modules><!-- ... --></system.webServer>
As you have probably already guessed, I’m using devise to handle my authentication. Unfortunately this seems to be overriding some of my routes whenever it co-exists with another form. Luckily there exists a very simple, although hard to find solution – simply add a forward slash before the controller definition:
I managed to break my app again while playing around with the routes and a few redirects:
No data received
Unable to load the webpage because the server sent no data.
Here are some suggestions:
Reload this webpage later.
Error 324 (net::ERR_EMPTY_RESPONSE): The server closed the connection without sending any data.
This wasn’t one I had encountered before and unfortunately Google couldn’t give me anything to cheat off. Luckily, a quick review of the server log revealed the following message just before it closed the connection:
[2011-12-03 21:09:05] ERROR URI::InvalidURIError: the scheme http does not accept registry part: 192.168.1.3:3000locations (or bad hostname?)
From this error message it became clear that I needed to place a forward slash before my redirect path:
#Redirect to edit location path
Hopefully this’ll be able to help someone else out there, Good Luck!