Webtrees and search engines4/17/2023 My ISP (TalkTalk) has an annoying habit of following me everywhere. So, I set that particular UA and IP range to "allow", so that they can see the same page that a real visitor would see. It sends the UA "W3C_Validator/1.3" and uses lots of different IP addresses, but tells me that they are all owend by MIT, and are part of the netblock 128.30.0.0 to 128.30.255.255. It needs to be able to see all the pages, and I trust it just to visit pages that are specifically requested. I use the w3validator service to check that webtrees is generating error-free HTML. So, here are a few examples of rules that I have created. By following the links, they would end up with an almost infinite number of pages. For example, we don't want them to have access to the calendar page. They only tend to cause problems when they visit too frequently, and consume too many server resources.īut it is important to identify them. Anyone can send a request with a UA string of: Remember also that user-agent strings are easily faked. What you do with this information depends on how aggressively you want to restrict access to the site. It was designed so that webtrees will work perfectly if you just ignore it.īut if you want to actively manage things, you'll want to start by finding out a little more information about each of your "unrecognised visitors":ġ) Use to find out who owns the IP addressĢ) Use to find out a little more about the user-agent string.ģ) Check your webserver logs to see if it visits too frequently, and what sort of requests it makes. Secondly (and importantly!), there is no requirement to do anything. It will probably stay in the code - but it may be removed if it causes too many problems. It was written because I get lots of emails from people who ask me to update webtrees to block a certain robot that is causing problems on their site. Firstly, this is an experimental feature.
0 Comments
Leave a Reply.AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |