Before creating reboot.txt you should know why it is important to use it in your site. Earlier search engines were very slow and had a lot of trouble getting search results because search engines had a hard time telling a site which pages, posts, tags, labels and categories to show and which not to show. In this problem the crawl time of the site was running out due to which the site was not showing any results and therefore it was very difficult to rank in the search engines but since Google introduced robot.txt. Search engines don't have to face so much trouble, rather they easily know which pages to search and which ones not to search. And that's what robot.txt does, where you put everything on your site that you want to show up and things that you don't want to show up in search engines. Then you write Allow with the things you want to show and Disallow with the things you don't have to show, and that's why it's easy for Roborts to know what to show and what not to show.
Create Robot.txt For Blogger:
All you have to do is paste your site URL in place of this yellow and you will get a nice robot.txt for your site.
No 1. Custom Domain:
User-agent: *
Disallow: /search?q=
Allow: /
Sitemap: https://www.Your domain.com/sitemap.xml
Sitemap: https://www.Your domain.com/sitemap-pages.xml
No 2. Free Domain:
User-agent: *
Disallow: /search?q=
Allow: /
Sitemap: https://www.Your domain.blogspot.com/sitemap.xml
Sitemap: https://www.Your domain.blogspot.com/sitemap-pages.xml
How to Disallow a particular post or page:
If you want to Disallow a post or page you have to do two things
No. 1 You have to put this ( Disallow: ) in front of Disallow in your Robot.txt.
No. 2 You have to paste the URL of that post or page in front of it.
example
User-agent: *
Disallow: /search?q=
Disallow: The URL of your post
Disallow: The URL of your page
Allow: /
Sitemap: https://www.Your domain.com/sitemap.xml
Sitemap: https://www.Your domain.com/sitemap-pages.xml
How to Disallow Labels and Categories:
If you want to Disallow a Labels or Categories you have to do two things
No. 1 You have to put this ( Disallow: ) in front of Disallow in your Robot.txt.
No. 2 You have to paste the URL of that post or page in front of it.
User-agent: *
Disallow: /search?q=
Disallow: Your label and category
Allow: /
Sitemap: https://www.Your domain.com/sitemap.xml
Sitemap: https://www.Your domain.com/sitemap-pages.xml
Note! Whether it is a custom domain or a free domain, the procedure for all domains is the same.
المشاركات
