What is Domain Discovery?

The term “domain discovery” refers to the act of mapping out the namespace of a domain by finding subdomains. This can be achieved using a variety of different methods. The goal is usually to identify issues and vulnerabilities and to correct them before the bad guys get a chance to find them.

Many organizations have policies and practices in place to help them to manage the security of their internal networks and services, but they often forget what’s “outside the firewall”

DNS is one of the key services that’s poorly managed, with mismanagement so widespread that it’s even become a meme to say “it’s always DNS” when something goes wrong with an internet service. You can learn more about why Namespace Security is important at NamespaceSecurity.com.

It’s not uncommon for corporate and public sector namespaces to have hundreds of thousands of subdomains. Parts of these namespaces are often delegated to different teams and third party companies.

Because of that, namespace owners often face significant challenges when keeping track of what the various teams and third parties are doing in their namespace. There’s often a lack of lifecycle and general domain management, as well as a lack of clarity when it comes to who’s actually responsible and empowered to manage the namespace or subdomains within the namespace.

Responsibility is often fragmented between the branding, legal and technology teams within an organization. To make matters worse, the scope of the challenge is so large that it’s difficult for companies to wrap their heads around it.

So how big is the internet, really?

The honest answer to this is that no one knows for sure. At the last count, domaindiscovery.uk had observed 3,285,889,131 subdomains. Measuring in at over three billion as of December 2021, that’s obviously a heck of a lot, even when you consider that a number of these domains are by now probably inactive.

We can take the number of top-level domains (364.6 million) as published by Verisign in The Domain Name Industry Brief and the number of subdomains that we discovered to figure out that the average top-level domain name has around ten subdomains.

We should also remember that the average is dragged up significantly by the leading players. For example, amazonaws.com alone has over 67 million subdomains, and so it’s likely that the median number is a little lower than the average.

We explored the largest namespaces on the internet in a recent article, so click here to check that out.

Discovering domains

It’s mostly straightforward to figure out the size of various Top Level Domains (TLDs). There are a number of services where you can find out the size of a TLD. For new Top Level Domains (nTLDs), you can use ntldstats.com or one of the many other services that publish the size of zone files online. You can also read the excellent Domain Name Industry Brief that’s published by Verisign.

These services often determine the numbers by analyzing the zone files of a TLD as published by the Registry Operator or by the “regulator” of the internet DNS, ICANN. If you feel so inclined, you can analyze zone files yourself by getting them from the ICANN CZDS service or by contacting the various registry operators directly.

For ccTLDs, this process is more complicated because many don’t make their zone files available to third parties. One exception in the ccTLD world is the .se registry, which makes the .se and .nu zones files available for download.

According to the latest Domain Name Industry Brief, the third quarter of 2021 closed with 364.6 million domain name registrations across all top-level domains. This is a huge number, but it only tells part of the story. While very large this is just the number of domain registrations at the “top level” and including domains like ibm.com, centralnic.com and google.de. It doesn’t include domains further down the namespace like googlemail.l.google.com or support.ibm.com.

When you move down the namespace from the “top level”, everything gets a lot more complicated.

There’s no single registry where you can get an authoritative list or number for these subdomains. As an example, ibm.com is one domain in the .com zone file as operated by Verisign Inc, but when you look up the ibm.com namespace at domaindiscovery.uk, you can find over 137,920 observed subdomains as of December 2021. Another large namespace is amazonaws.com, where domaindiscovery.uk has located over 67,742,771 observed subdomains.

To be able to find these subdomains, domaindiscovery.uk deploys a number of active and passive techniques to observe domains being used on the internet which are then cataloged and tracked to generate a map of what’s often called the public core of the Internet.

You can find the size of the observed namespace for any organization by going to the domaindiscovery.uk website and searching for a domain name.

Why is domain discovery performed?

Your namespace is a virtual representation of your organization, so it’s important to know what your online estate looks like.

It’s like keeping track of your equipment and your inventory, except that your domain is public-facing and much more vulnerable to attack. Your online namespace is often the first place that malicious actors will try to attack.

Many organizations have thousands of domain names in their namespace, often with no lifecycle management in place. This represents a significant risk because domains can keep pointing to suppliers and other third parties after they or the services they point to have been decommissioned, making them vulnerable to attacks.

We’ve found many large public and private entities that have shockingly little control over their namespace and who don’t know who operates their domains at the various levels. We often see lacking lifecycle management and poor hygiene in namespaces that are operated by entities who ought to know better.

This can be an issue not only to the namespace owner, but also to its customers and users.

If an entity loses control of a domain in their namespace, it can be used in many different ways to compromise the business, its partners or users.

Sometimes a domain can be fully taken over, which can give a malicious third-party total control over it. We call this a complete domain hijack. This can be used to point the domain to a webpage or email service of the attacker’s choice and can also be used to get valid SSL/TLS certificates.

In other cases, a domain can’t easily be fully taken over, but it can point to a decommissioned service that can be compromised. We call this a partial domain hijack. This is what you often read about when well-known brands have domains pointing to web pages containing malicious material that isn’t endorsed by the entity.

A partial domain hijack can also include domains pointing to email services that can be taken over.

A number of well-known partial domain hijack cases have involved domains pointing to Amazon S3 buckets that have been decommissioned. This hijack takes advantage of poor domain hygiene where appropriate lifecycle management for the subdomains has been overlooked.

Both of these issues are serious, but a domain that can be fully compromised is by far the worst as an attacker can set up services which can’t be distinguished from the real thing.

We’re not going to discuss the techniques that are used to achieve this except to say that there are a number of vectors that can be deployed to take control of subdomains within a namespace. We often see well-known domains from global entities that have vulnerable subdomains.

You may have heard about people being fooled by phishing messages that come from lookalike domains. These attacks become even more effective when the message comes from one of a company’s actual domains and with all of the usual email security methods being valid.

The domain in the email may have valid SSL/TLS certificates, SPF, DMARC and DKIM records, which means that there’s no way for the end user to know that it isn’t legitimate.

What’s next?

The best way to stop these kinds of attacks is to stay on top of your domains to avoid domain hijacking, which can be catastrophic for your brand and its users. Domain hijacking is the process in which people change the configuration or registration of a domain name without its owner’s permission. This is often done by exploiting configuration errors or by abusing privileges at domain registrars or hosting providers.

Know what your domains point to before the bad guys do. You can use vulnerability scanning and penetration testing tools to probe endpoints inside your namespace after discovering subdomains through Domain Discovery.

You need to know which domains are in your namespace before you can review them and decide if they’re still in use. You’ll also want to ensure that domain name managers in your namespace responsibly decommission your domains when they’re no longer needed. Poor domain lifecycle management is often the root cause of critical domain vulnerabilities.

Need help identifying which domains are in your namespace? Contact domaindiscovery.uk today to find out more about how we can help you.