岑辉宇SEO博客:分享搜索引擎优化的心得和经验,充当新手SEO道路上的照明灯,让SEO不再没方向,欢迎加入SEO交流群:241865235,微信公众号:qq945748441(每天前两名关注公众号的朋友,免费分析网站)

robots协议

seo百科视频 贵阳SEO 2154℃ 0评论

robots协议

spider在访问一个网站时,会首先会检查该网站的根域下是否有一个叫做 robots.txt的纯文本文件,这个文件用于指定spider在您网站上的抓取范围。

robots.txt文件放在哪里?

robots.txt文件应该放置在网站根目录下。
首先会检查该网站中是否存在http://www.abc.com/robots.txt这个文件
robots格式
User-agent: *
Disallow:
allow:

“*”和”$”的含义

Baiduspider支持使用通配符”*”和”$”来模糊匹配url。
“$” 匹配行结束符。
“*” 匹配0或多个任意字符。

转载请注明:岑辉宇SEO博客 » robots协议

喜欢 (0)or分享 (0)
发表我的评论
取消评论
表情
(38)个小伙伴在吐槽
  1. Nice post. I find out some thing tougher on various blogs everyday. Most commonly it is stimulating to learn to read content from other writers and exercise a specific thing at their store. I’d would prefer to apply certain while using the content on my own blog whether or not you don’t mind. Natually I’ll supply you with a link on your own internet blog. Thank you for sharing.
    匿名2019-07-10 03:41 回复
  2. Somebody necessarily help to make seriously articles I might state. That is the very first time I frequented your web page and to this point? I surprised with the research you made to make this actual put up amazing. Fantastic task!
    匿名2019-07-08 23:04 回复
  3. However, erectile dysfunction (ED) is only considered a concern if satisfactory sexual performance has been impossible on a number of occasions for some time.
    匿名2019-07-08 03:36 回复
  4. thank you for sharing – Gulvafslibning | Kurt Gulvmand with us, I conceive – Gulvafslibning | Kurt Gulvmand really stands out : D.
    匿名2019-07-08 00:09 回复
  5. We would like to thank you just as before for the gorgeous ideas you offered Janet when preparing her post-graduate research as well as, most importantly, pertaining to providing each of the ideas in a blog post. Provided that we had been aware of your website a year ago, we’d have been saved the unnecessary measures we were having to take. Thank you very much.
    匿名2019-07-08 00:00 回复
  6. Youre so cool! I dont suppose Ive read anything such as this just before. So nice to find somebody by original thoughts on this subject. realy thanks for beginning this up. this site is something that is required over the internet, someone with some originality. beneficial problem for bringing a new challenge towards world wide web!
    匿名2019-07-07 09:52 回复
  7. This is a great blog, will you be involved in doing an interview about how you developed it? If so e-mail me!
    匿名2019-07-03 23:37 回复
  8. I am curious to find out what blog platform you happen to be utilizing? I’m experiencing some minor security problems with my latest site and I’d like to find something more risk-free. Do you have any solutions?
    匿名2019-07-02 23:35 回复
  9. Muchos hombres experimentarán problemas de erección al menos una vez en su vida. Un estudio ha demostrado que los tratamientos con receta como Viagra pueden mejorar los casos de disfunción eréctil en varones deprimidos y, hasta cierto punto, aliviar sus síntomas depresivos.
    匿名2019-07-02 15:49 回复
  10. Can i get thumbs up never did? before (please ). … I love this song
    匿名2019-07-01 22:24 回复
1 2 3 4