Hello friends, once again welcome to our new blog. In this blog, we will learn what a Robot.txt file is and what is its use in SEO and how we can create a Robot txt file and use it.
Robot.txt is a text file in which we give instructions to Google crawlers, bots, and internet programmers, which page has to crawl or not where we just use some Allow & Disallow Code. Robot.txt is a file that tells search engine crawlers to not crawl certain pages or sections of a website.
or
Robots.txt is a text file through which we give instructions to Google crawlers or search engine bots as to which of our pages should be crawled and which pages should not be crawled. Usually, we use it to disallow any page.
Why Robots.txt is important?
If there are some pages on your website that you want Google’s bots or crawlers not to crawl that page, then today we are talking about the same method which is robot.txt.
Robot.txt therefore becomes necessary in some places like if we have a bank website where we have some documents and also our details, it is personal information for us whereas for any bot or robots it is just a If there is a web page, it can also be crawled and people can also get our personal information. In such a situation, we use robots.txt file to inform them not to crawl that page.
How you can see robots.txt file -
If you want to see the robots.txt file of any website, then we type robots. txt along with its domain name. Example : { www.yourdomain.com/robots.txt }
How to create robots.txt file ?
Let us know how to create a robot txt file. A robot txt file looks something like this.
If we are using any SEO plugins on the website, like Yoast or Rank Math, then you can create it very easily, but if you are using it on a coding website, then you can create it in this format. First of all, after user agent *, write your allow or disallow, then copy the slug of the page you want to disallow and paste it. In this way, you can create the code for robot.txt. And if you are creating it for coding websites, then you can upload the file to its root folder. Let us know how you can create robot txt file with the help of plugins.
First of all you have to come to your dashboard.
After that you have to go to the tools option in Yoast SEO plugins, there you get the option of file editor.
After clicking on the file editor, you get the option to create a robot.txt, and when you click on create a robot.txt file.
Your robots.txt filler is created through Yoast, then the slug of the page you want to display. Copy and paste it in front of Disallow or Allow.
Then, after that, click on the save option and save it. After saving, you can check your robots.txt file again.
Now we know how robots upload test files to coding websites. First of all, you have to create a txt file,and for this, we first login to Cpanel.
After that, we go to the file manager
Then we go to the public HTML file
Then, from there, we click on the upload option,
and we uploaded the created robots.txt file. Then we start showing the robots.txt file in our root file.
Why we use [ * ] in robots.txt file ?
We use asterisks, then. When we don’t have a fixed name here, in place of *, the name of the boat for which you want to stop crawling will be written. If it is Google’s search engine, then it will be Google’s bot, and if it is Yahoo’s search engine, then it will be Yahoo’s bot. By using this *, we have told you that no matter what the bot is, this page should not be crawled.
If we want to disallow this page only for Yahoo Bot, then we write Yahoo Bot in front of the user agent.
Same, if we want to allow or disallow any page for any search engine, then we write the name of the bot of that search engine.
Friends, we hope that you have understood what a robot.txt file is, what its purpose is, and how you can create it. If you liked the blog, please leave a lovely comment. And you can also tell us on which topic you want a blog, we will definitely create a blog on your given topic.
