The robots.txt file

The robots.txt file is a good way to help search engines index your site. Sharetribe automatically creates this file for your marketplace.

Thomas Malbaux avatar
Written by Thomas Malbaux
Updated over a week ago

Where is the robots.txt file?

You can find the file at the root of your marketplace address. For example:

  • If your marketplace address is https://example.mysharetribe.com, your robots.txt file can be found at https://example.mysharetribe.com/robots.txt.

  • If your marketplace address is https://www.example.com, your robots.txt file can be found at https://www.example.com/robots.txt.

What does the robots.txt file contain?

The robots.txt file for Sharetribe marketplaces allows Google (and other search engines) to index all your marketplace pages. Some bots might be restricted for performance reasons.

Your robots.txt file would look something like this:

#
# Hello bot! There are some routes that need authentication. You should avoid them :)
#

User-agent: MJ12bot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: *
Disallow: /profile-settings
Disallow: /l/new
Disallow: /l/*/checkout
Disallow: /l/*/draft
Disallow: /l/*/pending-approval
Disallow: /l/*/new
Disallow: /l/*/edit
Disallow: /inbox
Disallow: /order
Disallow: /sale
Disallow: /listings
Disallow: /account
Disallow: /reset-password
Disallow: /verify-email
Disallow: /preview
Crawl-Delay: 5

Sitemap: https://example.mysharetribe.com/sitemap-index.xml

Is it possible to edit the robots.txt file?

If you customize your marketplace with code, yes. Otherwise, it is impossible to edit the robots.txt file for a specific marketplace with the no-code solution.

Did this answer your question?