岑辉宇SEO博客:分享搜索引擎优化的心得和经验,充当新手SEO道路上的照明灯,让SEO不再没方向,欢迎加入SEO交流群:241865235,微信公众号:qq945748441(每天前两名关注公众号的朋友,免费分析网站)

robots协议

seo百科视频 贵阳SEO 1806℃ 0评论

robots协议

spider在访问一个网站时,会首先会检查该网站的根域下是否有一个叫做 robots.txt的纯文本文件,这个文件用于指定spider在您网站上的抓取范围。

robots.txt文件放在哪里?

robots.txt文件应该放置在网站根目录下。
首先会检查该网站中是否存在http://www.abc.com/robots.txt这个文件
robots格式
User-agent: *
Disallow:
allow:

“*”和”$”的含义

Baiduspider支持使用通配符”*”和”$”来模糊匹配url。
“$” 匹配行结束符。
“*” 匹配0或多个任意字符。

转载请注明:岑辉宇SEO博客 » robots协议

喜欢 (0)or分享 (0)
发表我的评论
取消评论
表情
(30)个小伙伴在吐槽
  1. i love baby gifts and i love to give baby gifts to my baby and also the my sister’s baby`
    匿名2019-07-01 02:31 回复
  2. High quality info here! Keep up the great work. I love the feelings being expressed.
    匿名2019-06-30 11:34 回复
  3. Pero nos paramos a pensar en lo que hace el cuerpo automáticamente, que es vivir, si, vivir; respirar, hacer la digestión, que nos crezcan las uñas, el pelo, parpadear, son algunos ejemplos de nuestro sistema nervioso automático. La Fosfodiesterasa Tipo 5 (PDE5) es una sustancia que actúa eliminando el óxido nítrico de la circulación sanguínea, disminuyendo la cantidad de sangre en el cuerpo cavernoso.
    匿名2019-06-22 10:56 回复
  4. What are you stating, man? I recognize everyones got their own opinion, but really? Listen, your website is awesome. I like the work you put into it, especially with the vids and the pics. But, come on. Theres gotta be a better way to say this, a way that doesnt make it seem like everybody here is stupid!
    匿名2019-06-17 23:43 回复
  5. there are tanning lotions which are very hard on the skin, i always use organic tanning lotions,,
    匿名2019-06-17 05:56 回复
  6. then obviously it will be more than you expected. There is nothing in this world which comes easy. Every thing in this world is very tough to get and the competition for getting the things has become very much tough.
    匿名2019-06-16 23:36 回复
  7. Comfortably, the post is during truthfulness a hottest on this subject well known subject matter. I agree with ones conclusions and often will desperately look ahead to your updates . Saying thanks a lot will not just be sufficient, for ones wonderful ability in your producing. I will immediately grab ones own feed to stay knowledgeable from any sort of update versions. Amazing get the job done and much success with yourbusiness results!
    匿名2019-06-16 22:49 回复
  8. I simply want to mention I'm newbie to blogging and seriously enjoyed you're blog. Probably I’m going to bookmark your site . You absolutely have superb articles. Cheers for revealing your web site.
    view publisher site2019-06-13 19:19 回复
  9. You may need to make a few appointments. This type of hair loss usually affects just the scalp, but it sometimes also occurs in beards or eyebrows. Treatment is often entirely behavioral.
    匿名2019-05-31 03:41 回复
  10. Es un problema más común de lo que creemos: la DE moderada o completa (no un “pinchazo” de vez en cuando) afecta al 8% de los hombres de más de 40 años, y a un 40% de quienes han cumplido ya los 60, según International Journal of Impotence Research.
    匿名2019-05-30 02:59 回复
1 2 3