What is Robots.txt?

Robots.txt is a file that helps search engine crawlers (web robots or bots) crawl pages on your website. It tells search engines which pages to index in SERPS and avoid the ones you don’t want them to see.

Robots.txt is an essential part of SEO and can help you boost your website’s rankings in search engines.

If you use WordPress, you can install Yoast SEO Plugin. When you add new pages, posts, or products, it will automatically create and update a robots.txt file.

Shopify automatically generates a default robots.txt file that works for most stores. But you can create a custom robots.txt.liquid file to make customizations.

Though it’s not sustainable, you can manually create a robots.txt file. You’ll need to use a text editor (like Sublime Text) and type in some simple instructions. But you’ll need to manually update the file when you add new pages, posts, or products.