Register now or log in to join your professional community.
Both are supported by all crawlers which respect webmasters wishes. Not all do, but against them neither technique is sufficient.
You can use robots.txt rules for general things, like disallow whole sections of your site. If you say Disallow: /family then all links starting with /family are not indexed by a crawler.
Meta tag can be used to disallow a single page. Pages disallowed by meta tags do not affect sub pages in the page hierarchy. If you have meta disallow tag on /work, it does not prevent a crawler from accessing /work/my-publications if there is a link to it on an allowed page.
On the most basic level, neither the meta robots tag or the Robots.txt has authority over the other – but rather the “noindex” request has authority over the “index” request.
What I’ve suggested to development teams I’ve worked with in the past is to set the Robots.txt file to allow all crawlers, then use the meta robots tag to request that a page not be indexed on a page by page basis. This is easy to remember, easy to update, and makes it easy to avoid conflicts and confusion.
Meta Robots tag is much better as it helps in forcing the search engine crawlers not to index and display the hidden pages in your server.
Robots.txt is the file to block certain pages/content from search engines. If you want to block folders or any section of website robots.txt file is the better option.
If you want to block some specific pages you can use meta robot tags. In some situation you really need to use these tags for eg. listing pages.
There is no real "MAJOR" difference between the files. They both do the same work however the META tag is not as reliable as the .txt This is because I think (could be wrong) that the Robot.txt is a seperate file. The intention of the file,tag is to inform the search engines ROBOT not to index certain pages. IE Contact us ETC ETC... Hope that helps...
Personally I think the meta tag is even more useful than the robots.txt file.
The robots.txt file is intended to PREVENT bots from spidering files in the first place.
The robots meta tag can in some ways be used to get indexed files deindexed. And it
is more versatile as a tool to 'guide' bots through your website without having them
index all the files.
I always put up a robots.txt file BEFORE I upload any of the other files.
Then I fine-tune using the meta tag.
You can have any one but if your website has plenty of web pages then robots.txt is easy and reduces time complexity
Actually both works differ from each other to some extent & it all depends on the priorities to define the Google Crawler which one to give more priority to the index pages, which one not to.
Both serve the same purpose, and it is up to the site owner to chose whichever is convenient. robot.txt would be easier if you want to block specific directories etc. Meta tag could be better if you have lots of pages to be blocked from different directories / without a pattern. Adding a meta tag in the page itself would be easier in such occasions.
+ robots.txt can stop a robot from attempting to crawl certain URLs
- robots.txt has to have each URL to block listed inside, with some support for wildcards
+ <meta> tag can be customized for each page
+ <meta> tag offers more detailed instructions about whether to allow indexing or to allow following links
- <meta> tag has to be managed on each page
- <meta> tag requires the bot to crawl the page so it cannot prevent that from happening
Do you need help in adding the right keywords to your CV? Let our CV writing experts help you.