One of the most essential parts of a site’s SEO optimization is configuring robots.txt. In this learning part of magneto2 on the Wegento site, we describe the setting up of robots. If you want to config the Robot file, do the following settings.
Go to the Admin Panel:
navigate to Content >Design> Configuration.
From the left side, Edit the Global Design Configuration.
Related posts: What is Magento?
Scroll down and find the search engine robots section.
Related posts: What is the Robots.txt file?
Choose the Default Robots you want to be enabled on your website.
In the default robot section are several options you can choose from. Every option does a different job. In the following we describe them.
INDEX, FOLLOW: Robots index a page and follow the links on that page
NOINDEX, FOLLOW: Robots don’t index a page but follow the links on that page
INDEX, NOFOLLOW: Robots index a page but don’t follow the links on that page
NOINDEX, NOFOLLOW: Robots don’t index or follow the page
We click on the “INDEX, FOLLOW” option by default.
To see if these settings apply, go to the home page and apply the control + u key combination. Then press control + f and write the word “robots”. Now you can see a meta tag under the name “robots” that its content shows “INDEX, FOLLOW”.
After that, we return to the setting page and in the Edit custom instruction of the robot.txt File section, write any instruction that you want to be searched on your website based on it. In this box, we determine that robots have access to all sections and enter the default file, and Sitemap.
To save the changes click on the Save Configuration button. If you do not want to save the changes, click on the Reset to Defaults button.
Finally, to view the commands we entered, type “robot.txt” in the address bar after the name of the site.
As you see in the picture the instructions are shown and Google robots can have access to our site. This action is so useful for the SEO of the site. I hope our tutorial was useful for you.
