edd47877c2
We were using the same logic in many different places, so we're simplifying the code. I'm not convinced about the method names, though, so we might change them in the future. Note using this method for the default tenant in the `TenantDiskService` class resulted in a `//` in the path, which is probably harmless but very ugly and it also generates a different key than the one we got until now. I've added an extra test to make sure that isn't the case.
18 lines
447 B
Plaintext
18 lines
447 B
Plaintext
# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
|
|
#
|
|
# To ban all spiders from the entire site uncomment the next two lines:
|
|
# Disallow: /
|
|
|
|
User-agent: *
|
|
Disallow: /users/
|
|
Disallow: /comments/
|
|
|
|
Disallow: /*?*locale
|
|
Disallow: /*?*order
|
|
Disallow: /*?*search
|
|
Disallow: /*?*locale-switcher
|
|
Disallow: /*?*filter
|
|
Disallow: user_id
|
|
|
|
Sitemap: <%= "#{root_url}#{Tenant.path_with_subfolder("sitemap.xml")}" %>
|