Differences

This shows you the differences between two versions of the page.

--- hacker_forum_search_-_surface_web [2023/07/06 11:24]
kaduuwikiadmin [Are there any limitations?]
+++ hacker_forum_search_-_surface_web [2025/01/11 14:51] (current)
kaduuwikiadmin
@@ Line 15: / Line 15: @@
   * Network information: Hacker forums may contain information about network vulnerabilities, such as open ports and misconfigured servers, that can be exploited by attackers.
-===== How do we search forums? =====
+===== Live vs Database Search =====
-In this deep-web search, we log in to +50 known hacker forums with various accounts and submit the keyword that is entered in the search mask of Kaduu. For example, you can enter your company name or a brand to see if people are talking about it in the forums. If there are results for the search term, we show them in a link as a download. The corresponding pages are saved as a screenshot and also as a web page. We focus on the most popular forums in English, German, French and Russian language.
+The Kaduu dashboard offers a sophisticated darknet search tool designed to navigate and extract information from the elusive corners of the internet, commonly referred to as the "darknet." This search tool is vital for security professionals and researchers who require access to real-time and historical data related to cyber threats, data leaks, and hacker forums.
+==== Technology and Background ====
+The darknet is a part of the internet hosted within an encrypted network and accessible only through specialized means, such as specific browsers and configurations. Due to its secretive nature, standard search engines do not index these spaces, necessitating specialized tools like ours.
+==== Components of the Darknet Search ====
+The Kaduu darknet search tool comprises two main components:
+**Database Search:**
+  * Automatic Crawlers: Our system utilizes automatic crawlers that navigate and extract data from structured marketplaces and forums that allow for such operations. This data is then standardized and stored in our database for easy access.
+  * Programmatic Search: For forums lacking a clear structure, our tool programmatically uses the forum's search function, which often requires bypassing captchas and bot protection. This is achieved through authenticated accounts.
+  * Analyst Team: Some forums cannot be indexed or searched programmatically. In these cases, our analyst team manually explores these forums to extract valuable data, which is then saved into the database.
+==== Live Search: ====
+This feature is intended for in-depth, occasional searches in real time. It is limited to prevent account bans and IP blocking by robust botnet protection systems like Cloudflare. Live search is complementary to database searches and is particularly useful when specific, up-to-date information is required.
+==== Handling of Data Leaks ====
+  * Freely Available Leaks: If hackers publish data leaks on forums without cost, we index the complete data set.
+  * Paid Leaks: Most cybercriminals attempt to sell stolen data (e.g., credit card details, personal information). We index only the advertisements of these leaks, not the leaks themselves.
+  * Ransomware Data: Due to its size, structure, and the slow download speed from threat actors' sites, we do not store ransomware-related data. Instead, we provide links to these external sites for users to access directly.
+==== Usage and Restrictions ====
+  * **Database Access:** Clients can make extensive use of the database with minimal restrictions, subject to a maximum of 100,000 API calls per month. This ensures access to continuously updated information without significant delay.
+  *** Live Search Limitations:** Due to operational risks, the number of live searches is restricted. This feature should be considered a secondary tool, used sparingly to complement the comprehensive data available through the database search.
+==== Recommendations and Extended Use ====
+While the database search covers approximately 90% of relevant forums through automated crawlers and our analyst team, live search serves as a supplementary tool for deeper investigations. Should clients require more extensive use of live searches, we offer extended licenses that include the creation of additional forum accounts.
+===== How do we search forums in the live search? =====
+In this deep-web search, we log in to +50 known hacker forums with various accounts and submit the keyword that is entered in the search mask of the hacker forum. For example, you can enter your company name or a brand to see if people are talking about it in the forums. If there are results for the search term, we show them in a link as a download. The corresponding pages are saved as a screenshot and also as a web page. We focus on the most popular forums in English, German, French and Russian language.
+===== Can other users see our search? =====
+No - we only use the generic search field form the forum. Only the system admin could see in the log files the queries.
+===== How to perform a search? =====
+Lets say your company name is "Bank24 LdT" and your domains are "bank24.com and bank24.us". We recommend that you use a more generic search approach. The recommended query in the abive example would be "bank24" without the domain or legal specification.
 ===== How do we present the data? =====
@@ Line 34: / Line 78: @@
 ===== Are there any limitations? =====
-A search can take up to 30 minutes. Please be patient. We also only allow maximum 5 searches per client per day and maximum 20 per month because otherwise our authenticated accounts will get flagged.
+A search can take up to 60 minutes. Please be patient. We also only allow a maximum of 5 searches per customer per day and a maximum of 20 per month, otherwise our authenticated accounts will be flagged. If you want to search for leaks without restriction, you can use the expert leak search mode https://wiki.kaduu.ch/doku/doku.php?id=leak_search, which searches for data that has been leaked in the past. The difference is that hackers usually do not publish recent data leaks for free, but sell them. So, if you want to find more recent data leaks that are being sold, you need to perform a live search.
 ===== Why do I see so many errors in the forum search? =====
 Hacker forums are not professional websites in terms of stability, accessibility and functionality. So it happens again and again that they are temporarily offline. Sometimes they disappear completely and then reappear under a new domain. All hacker forums that have relevant data have in common that you have to log in with a user, solve a captcha and also very often have to pass an anti-bot check (Cloudflare). These components change frequently (e.g. the type of captcha) and thus automated scripts that are tuned to a forum quickly get problems when such a change takes place. But also changes to the HTML code within the forum can lead to errors if our tools are programmed to search for data in very specific areas of the forums. Another problem is that hacker forums do not tolerate automated bots like ours. So if a forum detects our activity, they will block our account or IP address. In summary, the technology will never be 100% reliable, there will always be changes and errors in individual forums. If there are several hundred forums, these errors will naturally accumulate and in kaduu only the errors are displayed on the very first page, which gives the impression that many forums have errors. But this is not the case and we are working daily on the optimization of the crawlers.
+===== Why is the number of live searches restricted? =====
+When performing a live search through Kaduu, we programmatically log in to various forums. This process includes solving captchas, bypassing anti-bot protections such as Cloudflare, and authenticating with our account. Once logged in, we use the forum’s search field to locate your search term.
+Throughout this process, we aim to minimize the number of queries we make. Allowing clients to perform multiple simultaneous live searches could trigger bot protection mechanisms or lead forum administrators to block or ban our accounts. Since many forums require paid accounts, account bans can result in significant costs.
+Unlike database queries, each live search query incurs substantial costs on our end. For example, we rely on paid residential proxies to bypass anti-bot protections. Additionally, many forums employ captchas that can only be solved manually, requiring us to engage paid external support services.
+Due to these constraints, we recommend using live search as a premium paid feature rather than for daily monitoring. If you require additional search credits, we can set up a second account for an additional fee.

My DokuWiki

User Tools

Site Tools

Differences

Page Tools