Table of Contents
Open-Source Intelligence (OSINT) is a method of gathering information from public or other open sources, which can be used by security experts, national intelligence agencies, or cybercriminals. OSINT leverages advanced technology to discover and analyse massive amounts of data, obtained by scanning public networks, from publicly available sources like social media networks, and from the deep web—content that is not crawled by search engines, but is still publicly accessible.
History of OSINT
The term OSINT was originally used by the military and intelligence community to denote intelligence activities that gather strategically important, publicly available information on national security issues. In the Cold War era, espionage focused on obtaining information via human sources (HUMINT) or electronic signals (SIGINT), and in the 1980s OSINT gained prominence as an additional method of gathering intelligence. With the advent of the Internet, social media, and digital services, open source intelligence grants access to numerous resources to gather intelligence about every aspect of an organisation’s IT infrastructure and employees.
How Attackers and Defenders Use OSINT
There are three common uses of OSINT: by cybercriminals, by cyber defenders, and by those seeking to monitor and shape public opinion.
How Security Teams Use OSINT
For penetration testers and security teams, OSINT aims to reveal public information about internal assets and other information accessible outside the organisation. This includes information such as open ports, unpatched software with known vulnerabilities, publicly available IT information such as device names, IP addresses and configurations, and other leaked information belonging to the organisation. Websites outside the organisation, especially social media, contain huge amounts of relevant information, especially information about employees. Vendors and partners may also be sharing specific details about an organisation’s IT environment.
How Threat Actors Use OSINT
A common use of OSINT by attackers is to retrieve personal and professional information about employees on social media. This can be used to craft spear-phishing campaigns targeted at individuals who have privileged access to company resources. LinkedIn is a great resource for this type of open source intelligence, as it reveals job titles and organisational structure. Other social networking sites are also highly valuable for attackers, as they disclose information such as dates of birth, names of family members and pets, all of which can be used in phishing and to guess passwords. Another common tactic is to use cloud resources to scan public networks for unpatched assets, open ports, and misconfigured cloud datastores.
OSINT Gathering Techniques
Here are three methods commonly used to gain open intelligence data.
Passive Collection
This is the most commonly used way to gather OSINT intelligence. It involves scraping publicly available websites, retrieving data from open APIs such as the Twitter API, or pulling data from deep web information sources. The data is then parsed and organised for consumption.
Semi-Passive
This type of collection requires more expertise. It directs traffic to a target server to obtain information about the server. Scanner traffic must be similar to normal Internet traffic to avoid detection.
Active Collection
This type of information collection interacts directly with a system to gather information about it. Active collection systems use advanced technologies to access open ports and scan servers or web applications for vulnerabilities. This type of data collection can be detected by the target and reveals the reconnaissance process. It leaves a trail in the target’s firewall, Intrusion Detection System (IDS), or Intrusion Prevention System (IPS). Social engineering attacks on targets are also considered a form of active intelligence gathering.
Artificial Intelligence: The Future of OSINT?
OSINT technology is advancing, and many are proposing the use of artificial intelligence and machine learning (AI/ML) to assist OSINT research. As AI/ML techniques become available to the private sector, they can help with improving the data collection phase, improving the data analysis phase, and improving actionable insights.
OSINT Tools
Here are some of the most popular OSINT tools.
Maltego
Maltego is part of the Kali Linux operating system, commonly used by network penetration testers and hackers. It is open source and provides built-in data transformations, the ability to write custom transformations, and built-in footprints that can collect information from sources and create a visualisation of data about a target.
Spiderfoot
Spiderfoot is a free OSINT tool available on Github. It integrates with multiple data sources and can be used to gather information about an organisation, including network addresses, contact details, and credentials.
Spyse
Spyse is an “Internet assets search engine” designed for security professionals. It collects data from publicly available sources, analyzes them, and identifies security risks.
Intelligence X
Intelligence X is an archival service that preserves historical versions of web pages that were removed for legal reasons or due to content censorship. It preserves any type of content, no matter how dark or controversial.
BuiltWith
BuiltWith maintains a large database of websites, which includes information on the technology stacks used by each site. It can be combined with security scanners to identify specific vulnerabilities affecting a website.
Shodan
Shodan is a security monitoring solution that makes it possible to search the deep web and IoT networks. It provides information on devices operating on protocols like HTTP, SSH, FTP, SNMP, Telnet, RTSP, and IMAP.
HaveIbeenPwned
HaveIbeenPwned is a service that can be used to identify if an individual email address was compromised in any historical breach. It also checks accounts on popular services for exposure to past data breaches.
Google Dorking
Google dorking is a technique commonly used by security professionals and hackers to identify exposed private data or security vulnerabilities via the Google search engine. It uses advanced search operators to identify content that can be useful to attackers.
Open Source Investigation Best Practices
Here are best practices that can help you use OSINT more effectively for cyber defence.
Distinguish Between Data and Intelligence
To successfully practice OSINT, focus on identifying the data needed for a specific investigation and refine your search to retrieve only the relevant information. This will let you derive useful insights at lower cost and with less effort.
Consider Compliance Requirements
Most organisations are covered by privacy regulations, so collecting, storing, and processing personal data can create a compliance risk. Be aware of the legal requirements for exposing criminal intent discovered through OSINT.
Be Ethical
Limit data collection to a minimum that can help you meet your goals without violating the rights of employees or others. Ensure data collection is controlled by humans and collaborate with all stakeholders to avoid privacy issues and other ethical concerns.