+- Sinisterly (https://sinister.li)
+-- Forum: Coding (https://sinister.li/Forum-Coding)
+--- Forum: Python (https://sinister.li/Forum-Python)
+--- Thread: How does google know my bot is a bot? (/Thread-How-does-google-know-my-bot-is-a-bot)
RE: How does google know my bot is a bot? - m0dem - 07-03-2016
Anyways, Google is probably changing there bot detection techniques every day! So its probably a combination of many things.
RE: How does google know my bot is a bot? - Satan - 07-03-2016
Surprised nobodies said anything yet about it measuring the webpage frame to see if it's even in a browser.
Think they call it canvas? Idk.
RE: How does google know my bot is a bot? - Despised - 07-03-2016
(07-03-2016, 02:43 PM)m0dem Wrote: Anyways, Google is probably changing there bot detection techniques every day! So its probably a combination of many things.This.
Might want to go for a different search engine if the problem persists.
RE: How does google know my bot is a bot? - Equinox - 07-09-2016
(07-03-2016, 09:02 AM)Eclipse Wrote:(06-12-2016, 01:26 PM)Stocking Wrote: Google does more user checking than a user agent alone. However, if it's something such as a search, just use Google's API. APIs exist for more than one Google service, in fact most of them, so if it's something that can use an API, in this case, it should.
As far as what else it does, I'm not sure. I myself have never been able to quite "crack the code" on what Google does. Perhaps it's worth a Google.
Google's search API is deprecated and it was rate limited anyway.
Deprecated doesn't mean unusable. It can still be used (Just a note here, I was incorrect, I've just double checked and Google has completely removed all possible usage of the API), and it's not hard to bypass rate limiting on, well, anything. Besides that, if you're searching for something, and you get the RL response, chances are you don't have good intentions to begin with.
RE: How does google know my bot is a bot? - pvnk - 07-09-2016
I learned a lot while lurking on this thread. Thank you for the knowledge!
RE: How does google know my bot is a bot? - Eclipse - 07-09-2016
(07-09-2016, 12:57 AM)Stocking Wrote:(07-03-2016, 09:02 AM)Eclipse Wrote:(06-12-2016, 01:26 PM)Stocking Wrote: Google does more user checking than a user agent alone. However, if it's something such as a search, just use Google's API. APIs exist for more than one Google service, in fact most of them, so if it's something that can use an API, in this case, it should.
As far as what else it does, I'm not sure. I myself have never been able to quite "crack the code" on what Google does. Perhaps it's worth a Google.
Google's search API is deprecated and it was rate limited anyway.
and it's not hard to bypass rate limiting on, well, anything.
I'd disagree. Google uses machine learning algorithms and it's vast amount of data to map how a bot looks and behaves and then serves anyone that fits the bill a captcha. It even flags up your real browser of you behave like a bot. Feel free to attempt it.
RE: How does google know my bot is a bot? - Equinox - 07-09-2016
(07-09-2016, 08:34 AM)Eclipse Wrote:(07-09-2016, 12:57 AM)Stocking Wrote:(07-03-2016, 09:02 AM)Eclipse Wrote: Google's search API is deprecated and it was rate limited anyway.
and it's not hard to bypass rate limiting on, well, anything.
I'd disagree. Google uses machine learning algorithms and it's vast amount of data to map how a bot looks and behaves and then serves anyone that fits the bill a captcha. It even flags up your real browser of you behave like a bot. Feel free to attempt it.
Homemade proxies are a thing. I was referring to the APIs. The only reason you wouldn't be able to bypass the rate limiting now is because you would use the same API key, however, if you have multiple already, you're good to go.
RE: How does google know my bot is a bot? - Eclipse - 07-10-2016
(07-09-2016, 11:50 PM)Stocking Wrote:(07-09-2016, 08:34 AM)Eclipse Wrote:(07-09-2016, 12:57 AM)Stocking Wrote: and it's not hard to bypass rate limiting on, well, anything.
I'd disagree. Google uses machine learning algorithms and it's vast amount of data to map how a bot looks and behaves and then serves anyone that fits the bill a captcha. It even flags up your real browser of you behave like a bot. Feel free to attempt it.
Homemade proxies are a thing. I was referring to the APIs. The only reason you wouldn't be able to bypass the rate limiting now is because you would use the same API key, however, if you have multiple already, you're good to go.
No idea how homemade proxies come into this.
RE: How does google know my bot is a bot? - Equinox - 07-11-2016
(07-10-2016, 12:56 PM)Eclipse Wrote:(07-09-2016, 11:50 PM)Stocking Wrote:(07-09-2016, 08:34 AM)Eclipse Wrote: I'd disagree. Google uses machine learning algorithms and it's vast amount of data to map how a bot looks and behaves and then serves anyone that fits the bill a captcha. It even flags up your real browser of you behave like a bot. Feel free to attempt it.
Homemade proxies are a thing. I was referring to the APIs. The only reason you wouldn't be able to bypass the rate limiting now is because you would use the same API key, however, if you have multiple already, you're good to go.
No idea how homemade proxies come into this.
It was previously mentioned in the thread that using proxies would be essentially useless as Google would recognize them. Not entirely true, and hence I brought up said homemade proxies.
RE: How does google know my bot is a bot? - 0M4R - 07-22-2016
It could be the web client, i'm not sure if this is the same with python as i have very little knowledge in that. But with vb if you use the internet explorer client it asks you for the verification however with google chrome it does not.