Advanced searching 11-01-2012, 06:17 PM
#1
What?
Yes, you read that right. Searching. No, it's not an obscure hacker term you haven't heard about. I'm talking about what you think I'm talking about - looking for things on the internet, usually using search engines like google.
Why?
The little known art of searching
Searching is the single most important skill a hacker can have. And believe me, most of people (yes, even hackers) can't REALLY search. Knowing how to use google dorks or how to dox someone is barely scratching the surface of all the things you can find if you know how to look for them.
What can you find
If you know how to find stuff, you can do a lot. I'll give you some examples:
- creation of correct googledorks for the vuln you've found
- finding misconfigurations
- doxing
- fingerprinting without direct interaction
- finding backdoors
- easier and faster search for interesting data, especially (but not limited to) scientific
- exploration of deepweb, darknets and web archives
- finding stuff that can help in social engineering, password guessing, data gathering and enumeration
The basics
Before we go into the more difficult stuff, here are things you need to know. Those things are the introduction to advanced searching.
Efficient use of search engines
As most of you know, google has a lot of advanced commands - and so do other popular search engines. They can help you limit search results to those that interest you the most by limiting extensions, searching in URLs, titles and text, excluding words and phrases, applying wildcards etc. There is no point in listing all the commands for all the engines so just grab those links: http://www.googleguide.com/advanced_operators.html#info: http://help.yahoo.com/kb/index?page=cont...ions=false http://msdn.microsoft.com/en-us/library/ff795620.aspx . I'll just say that you'll mostly need file extensions, intext/inurl, searching specific site and wildcards. But you'll use the other ones sooner or later, believe me.x
Use of different search engines
Yes, this is also a thing. There are different rules in robots.txt and different policies of those who own search engines. The results in every engine will vary a little. Basically, if google doesn't give you what you want, try Yahoo or Bing.
Catalogs and toplists
Let's face it, if you are searching for something very obscure, whatever you look for it still might be on page over 9000 in the results. If you need something from a small, old website, try to search for catalogs and toplists on related subject. This is not a very efficient now as most of that kind of things are either for SEO purposes (so you won't find things you can't find on google there) or taken over by spambots.
Specific search engines
Searching in blogs or searching the usenet is sometimes a good idea. Popular search engines often have sections for those. Use them.
Beyond the indexed
Mainstream search engines don't have everything. Sometimes, you'll want to look deeper.
Deep web
Most of the internet hasn't been crawled by the popular engines because many things on the internet are too obscure to matter, restricted to members or not placed on HTTP webservers. Fortunately, there are search engines that specialize in looking for info there. Most of them are specific to certain category (i.e. pubmed for medical resources) and you'll find them easily so I'll just give you links to some good general purpose deepweb searchers: http://www.mamma.com/ http://clusty.com/ http://lookahead.surfwax.com/index-2011.html http://www.search.com/ http://deeperweb.com/ http://www.touchgraph.com/seo/ http://infomine.ucr.edu/ http://vlib.org/ . Get to know them, they are very helpful.
The archives
Some things just aren't there anymore. Unfortunately, some of those things could have helped you with what you're currently doing. Fortunately, there are internet archives for them. First of all, most of search engines have cache which stores previous version of the website. This cache is helpful for bypassing blocked or finding removed stuff, provided it's relatively recent as the cache is temporal. If you can't find what you want to find in cache, you can always try the wayback machine at http://archive.org/ . It is a great tool with really amazing numbers of old versions of websites.
Darknets
Some parts of the net are purposefully hidden from the normal users. Sometimes it's a simple password protection, sometimes it's a more complicated system. Now I'm talking about the latter. Things like TOR Hidden Services or Freenet.
As this kind of darknets are meant for anonymization of both the users and the administrators, they are volatile. Search engines are small and usually very oldschool (catalog-based, not crawler-based), the addresses are everchanging etc. The best way to find things in the darknets is to get into their communities. Find related wikis on the surface web, ask around on the Usenet and the imageboards. Get into a realtively big community (but beware of malicious websites, pedophiles and fraudsters - there a lot of them there), get to know people there. Find the relatively up-to-date catalogs and hop from one website to the websites linked there, like in web1.0 times. It's really the only way to explore darknets.
Non-HTTP resources
There are p2p networks with good content. I reccommend Soulseek because the community is nerdy enough to be relevant in hacking world. There are IRC channels - Anonnet is a good place to start if you don't mind the fact that a lot of noobs are there. There are public and semi-public FTP servers - there are databases for them, but most of them are not up-to-date but worth trying anyway. http://www.mmnt.net/ is mostly good and http://www.ftp-sites.org/ is not bad for something that hasn't been updated for 10 years.
Final thoughts
Additional tips
- don't be afraid to ask others for help in your search, community is always important
- learn how the engines work to know how to use them the right way
- plan ahead
Further reading
I've just outlined the idea of advanced searching here. You NEED to know more and your best resource is http://www.searchlores.org/ , possibly the most important website on the whole internet. Trust me on this one. Read it and learn it, you'll be a better hacker after it.
Yes, you read that right. Searching. No, it's not an obscure hacker term you haven't heard about. I'm talking about what you think I'm talking about - looking for things on the internet, usually using search engines like google.
Why?
The little known art of searching
Searching is the single most important skill a hacker can have. And believe me, most of people (yes, even hackers) can't REALLY search. Knowing how to use google dorks or how to dox someone is barely scratching the surface of all the things you can find if you know how to look for them.
What can you find
If you know how to find stuff, you can do a lot. I'll give you some examples:
- creation of correct googledorks for the vuln you've found
- finding misconfigurations
- doxing
- fingerprinting without direct interaction
- finding backdoors
- easier and faster search for interesting data, especially (but not limited to) scientific
- exploration of deepweb, darknets and web archives
- finding stuff that can help in social engineering, password guessing, data gathering and enumeration
The basics
Before we go into the more difficult stuff, here are things you need to know. Those things are the introduction to advanced searching.
Efficient use of search engines
As most of you know, google has a lot of advanced commands - and so do other popular search engines. They can help you limit search results to those that interest you the most by limiting extensions, searching in URLs, titles and text, excluding words and phrases, applying wildcards etc. There is no point in listing all the commands for all the engines so just grab those links: http://www.googleguide.com/advanced_operators.html#info: http://help.yahoo.com/kb/index?page=cont...ions=false http://msdn.microsoft.com/en-us/library/ff795620.aspx . I'll just say that you'll mostly need file extensions, intext/inurl, searching specific site and wildcards. But you'll use the other ones sooner or later, believe me.x
Use of different search engines
Yes, this is also a thing. There are different rules in robots.txt and different policies of those who own search engines. The results in every engine will vary a little. Basically, if google doesn't give you what you want, try Yahoo or Bing.
Catalogs and toplists
Let's face it, if you are searching for something very obscure, whatever you look for it still might be on page over 9000 in the results. If you need something from a small, old website, try to search for catalogs and toplists on related subject. This is not a very efficient now as most of that kind of things are either for SEO purposes (so you won't find things you can't find on google there) or taken over by spambots.
Specific search engines
Searching in blogs or searching the usenet is sometimes a good idea. Popular search engines often have sections for those. Use them.
Beyond the indexed
Mainstream search engines don't have everything. Sometimes, you'll want to look deeper.
Deep web
Most of the internet hasn't been crawled by the popular engines because many things on the internet are too obscure to matter, restricted to members or not placed on HTTP webservers. Fortunately, there are search engines that specialize in looking for info there. Most of them are specific to certain category (i.e. pubmed for medical resources) and you'll find them easily so I'll just give you links to some good general purpose deepweb searchers: http://www.mamma.com/ http://clusty.com/ http://lookahead.surfwax.com/index-2011.html http://www.search.com/ http://deeperweb.com/ http://www.touchgraph.com/seo/ http://infomine.ucr.edu/ http://vlib.org/ . Get to know them, they are very helpful.
The archives
Some things just aren't there anymore. Unfortunately, some of those things could have helped you with what you're currently doing. Fortunately, there are internet archives for them. First of all, most of search engines have cache which stores previous version of the website. This cache is helpful for bypassing blocked or finding removed stuff, provided it's relatively recent as the cache is temporal. If you can't find what you want to find in cache, you can always try the wayback machine at http://archive.org/ . It is a great tool with really amazing numbers of old versions of websites.
Darknets
Some parts of the net are purposefully hidden from the normal users. Sometimes it's a simple password protection, sometimes it's a more complicated system. Now I'm talking about the latter. Things like TOR Hidden Services or Freenet.
As this kind of darknets are meant for anonymization of both the users and the administrators, they are volatile. Search engines are small and usually very oldschool (catalog-based, not crawler-based), the addresses are everchanging etc. The best way to find things in the darknets is to get into their communities. Find related wikis on the surface web, ask around on the Usenet and the imageboards. Get into a realtively big community (but beware of malicious websites, pedophiles and fraudsters - there a lot of them there), get to know people there. Find the relatively up-to-date catalogs and hop from one website to the websites linked there, like in web1.0 times. It's really the only way to explore darknets.
Non-HTTP resources
There are p2p networks with good content. I reccommend Soulseek because the community is nerdy enough to be relevant in hacking world. There are IRC channels - Anonnet is a good place to start if you don't mind the fact that a lot of noobs are there. There are public and semi-public FTP servers - there are databases for them, but most of them are not up-to-date but worth trying anyway. http://www.mmnt.net/ is mostly good and http://www.ftp-sites.org/ is not bad for something that hasn't been updated for 10 years.
Final thoughts
Additional tips
- don't be afraid to ask others for help in your search, community is always important
- learn how the engines work to know how to use them the right way
- plan ahead
Further reading
I've just outlined the idea of advanced searching here. You NEED to know more and your best resource is http://www.searchlores.org/ , possibly the most important website on the whole internet. Trust me on this one. Read it and learn it, you'll be a better hacker after it.
(This post was last modified: 11-01-2012, 07:34 PM by digitkharel.)
![[+]](https://sinister.li/images/modern/collapse_collapsed.png)
