Dscape
Dscape (ie Digital Landscape) is a search engine analysis tool that I created for a marketing agency to help clients understand where their website lies within the greater digital landscape, particularly the Google search engine.
Background
The agency provided some search engine tools before the development of Dscape, particularly one-off SEO audits, keyword research and PPC ad spend. The SEO audits were performed with some spider tools like Screaming Frog, SEMrush and DeepCrawl (now Lumar), but besides that it was just a matter of reading through the results and collating them.
Then we found that there was a gap in the market when it came to providing insights about a company’s target keywords and how they compare to their competitors when it come to taking up space on the search engine results page (SERP).
Concept
The idea is that the client would give us a set of keywords and their georaphical location, and we would do the rest. The workflow is as follows:
- Create new client account
- Add in their keywords
- Run these keywords through the SERP and get:
- Organic: Top 100
- Paid: Up to 12
- Related questions: Up to 12
- Related queries: Up to 7
- Video results: Up to 4
- News results: Up to 3
This is enough for a basic analysis, but this data can be enriched even further:
- Domain:
- Topics: (tags) and sentiment analysis
- Ad services: Which do they use (display / on-site ads)
- Companies
- Basic info: Name, email, market cap, employee headcount etc
- Categories: Location, industry, category
- Advanced: Tags & tech stack
How Dscape works
Let’s go through in some more detail about how each of the steps works.
Client account
The simplest step is to create a client account. This is just something that is used to connect all the keywords that a client provided.
Keyword checker
I had already built a suite of automation tools, including a keyword checker. This tool connects to the Google Ads API to retrieve keyword data such as search volume, competition level, and cost per click (CPC). We originally used it in our PPC campaigns to uncover low-cost, underutilized keywords that could maximize clicks within a client’s budget.
The tool also allows us to segment keywords by location and language, making it possible to deliver insights at both global and local levels.
While its ability to generate keyword ides wasn’t useful for this project, it still provides an authoritative source of metrics that we can use to quantify the effectiveness of a set of keywords. The only adjustment needed for SEO use was to make the tool callable from other programs, which is something I accomplished easily by switching its output from dynamic HTML to JSON.
Run the search engine crawler
We used a search engine crawler called SERPAPI to then process the keywords that were provided by the client. This gave us a set of organic results, paid results, related queries, questions, video results and news results. The result would look like this for each keyword (using “n64 homebrew” as an example):
This result gives us organic, news, video, and related searches. There are no paid, news or questions results
{
"search_metadata":
{
"id": "68d6a5a3bf04d540a5b150d9",
"status": "Success",
"json_endpoint": "https://serpapi.com/searches/74d3dcf5eaf36b5f/68d6a5a3bf04d540a5b150d9.json",
"created_at": "2025-09-26 14:39:31 UTC",
"processed_at": "2025-09-26 14:39:31 UTC",
"google_url": "https://www.google.com/search?q=n64+homebrew&oq=n64+homebrew&hl=en&gl=us&sourceid=chrome&ie=UTF-8",
"raw_html_file": "https://serpapi.com/searches/74d3dcf5eaf36b5f/68d6a5a3bf04d540a5b150d9.html",
"total_time_taken": 1.14
}
,
"search_parameters":
{
"engine": "google",
"q": "n64 homebrew",
"google_domain": "google.com",
"hl": "en",
"gl": "us",
"device": "desktop"
}
,
"search_information":
{
"query_displayed": "n64 homebrew",
"total_results": 1270000,
"time_taken_displayed": 0.23,
"organic_results_state": "Results for exact spelling"
}
,
"inline_videos":
[
{
"position": 1,
"title": "A NEW N64 HOMEBREW MINIGAME COLLECTION!?!?",
"link": "https://www.youtube.com/watch?v=Ad2DIzO4z6U",
"thumbnail": "https://serpapi.com/searches/68d6a5a3bf04d540a5b150d9/images/97d420f0f4d1aaaebdb0e706d8625a7ed04cd655f06e54fdf39928c55a8762d1.jpeg",
"channel": "Captain Jack",
"duration": "7:23",
"platform": "YouTube"
}
,
{
"position": 2,
"title": "21 NEW N64 games *you can play now*",
"link": "https://www.youtube.com/watch?v=A_XSY3w8-Pg",
"thumbnail": "https://serpapi.com/searches/68d6a5a3bf04d540a5b150d9/images/97d420f0f4d1aaaeb11cec82a5c6990adea88f6975b6221284ef09ad0f266dce.jpeg",
"channel": "Hard4Games",
"duration": "29:58",
"platform": "YouTube"
}
,
{
"position": 3,
"title": "THESE NEW N64 HOMEBREWS ARE INSANE!",
"link": "https://www.youtube.com/watch?v=FoRg_E1V4yk",
"thumbnail": "https://serpapi.com/searches/68d6a5a3bf04d540a5b150d9/images/97d420f0f4d1aaae417afaaa456f7c2f64227a07521168e34db8a47a4fd8fe63.jpeg",
"channel": "Captain Jack",
"duration": "5:21",
"platform": "YouTube"
}
]
,
"organic_results":
[
{
"position": 1,
"title": "Nintendo 64 Homebrew",
"link": "https://n64squid.com/homebrew/",
"redirect_link": "https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://n64squid.com/homebrew/&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQFnoECBgQAQ",
"displayed_link": "https://n64squid.com › homebrew",
"favicon": "https://serpapi.com/searches/68d6a5a3bf04d540a5b150d9/images/c9365df4eded7fe24defe0456a9c1dfcfa0ca865b705b655715580cc4ccd2041.png",
"snippet": "There have been many Nintendo 64 homebrew games and demos made throughout the years. This is a project to document these games and how to make them too.",
"snippet_highlighted_words":
[
"Nintendo 64 homebrew"
]
,
"sitelinks":
{
"inline":
[
{
"title": "N64 SDK",
"link": "https://n64squid.com/homebrew/n64-sdk/"
}
,
{
"title": "Brick 64",
"link": "https://n64squid.com/homebrew/roms/brick-64/"
}
,
{
"title": "Super Mario Bros 64",
"link": "https://n64squid.com/homebrew/roms/super-mario-bros-64/"
}
,
{
"title": "Smash Remix (v2.0)",
"link": "https://n64squid.com/homebrew/roms/smash-remix/"
}
]
}
,
"source": "N64 Squid"
}
,
{
"position": 2,
"title": "N64brew Wiki",
"link": "https://n64brew.dev/wiki/Main_Page",
"redirect_link": "https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://n64brew.dev/wiki/Main_Page&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQFnoECBsQAQ",
"displayed_link": "https://n64brew.dev › wiki › Main_Page",
"favicon": "https://serpapi.com/searches/68d6a5a3bf04d540a5b150d9/images/c9365df4eded7fe24defe0456a9c1dfca06dc20f793fecc696c19362233be605.png",
"date": "Mar 2, 2025",
"snippet": "Welcome to the N64brew Wiki! This wiki is a collaboration among the homebrew community, proving accurate documentation of the Nintendo 64, its ...",
"snippet_highlighted_words":
[
"homebrew",
"Nintendo 64"
]
,
"source": "N64brew Wiki"
}
,
{
"position": 3,
"title": "r/N64Homebrew",
"link": "https://www.reddit.com/r/N64Homebrew/",
"redirect_link": "https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.reddit.com/r/N64Homebrew/&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQFnoECBoQAQ",
"displayed_link": "3.5K+ followers",
"favicon": "https://serpapi.com/searches/68d6a5a3bf04d540a5b150d9/images/c9365df4eded7fe24defe0456a9c1dfc2542277e2af6ac1e6d1494102cba3617.png",
"snippet": "r/N64Homebrew: A sub-reddit for N64 Homebrew Discussion, & Creation.",
"snippet_highlighted_words":
[
"N64Homebrew",
"N64 Homebrew"
]
,
"source": "Reddit · r/N64Homebrew"
}
,
{
"position": 4,
"title": "Homebrewn Nintendo 64 Games And Demos (N64 Roms) ...",
"link": "https://www.nesworld.com/article.php?system=n64&data=n64homebrew",
"redirect_link": "https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.nesworld.com/article.php%3Fsystem%3Dn64%26data%3Dn64homebrew&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQFnoECBwQAQ",
"displayed_link": "https://www.nesworld.com › article › data=n64homebrew",
"thumbnail": "https://serpapi.com/searches/68d6a5a3bf04d540a5b150d9/images/c9365df4eded7fe24defe0456a9c1dfcca013a3004b125187f568e6e2ca6336e.jpeg",
"favicon": "https://serpapi.com/searches/68d6a5a3bf04d540a5b150d9/images/c9365df4eded7fe24defe0456a9c1dfcbe1134cd3a1058b517b84f75e68d5509.png",
"snippet": "What you find here is a large section of Nintendo 64 homebrew developed over the years, by homebrew I mean games/apps and other binaries for use on the N64 ...",
"snippet_highlighted_words":
[
"Nintendo 64 homebrew"
]
,
"source": "NESWorld"
}
,
{
"position": 5,
"title": "List of Aftermarket N64 releases (Please help contribute!)",
"link": "https://www.videogamesage.com/forums/topic/14099-list-of-aftermarket-n64-releases-please-help-contribute/",
"redirect_link": "https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.videogamesage.com/forums/topic/14099-list-of-aftermarket-n64-releases-please-help-contribute/&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQFnoECCkQAQ",
"displayed_link": "https://www.videogamesage.com › forums › topic › 140...",
"thumbnail": "https://serpapi.com/searches/68d6a5a3bf04d540a5b150d9/images/c9365df4eded7fe24defe0456a9c1dfc85ce917e537794128891d82d3b5a5b1d.jpeg",
"favicon": "https://serpapi.com/searches/68d6a5a3bf04d540a5b150d9/images/c9365df4eded7fe24defe0456a9c1dfc9e1c09e94e68423d998b485d3e9db069.png",
"date": "Dec 11, 2023",
"snippet": "Unlike the NES there currently isn't a HUGE aftermarket/homebrew scene on the N64, but there is certainly more and more activity on this front",
"snippet_highlighted_words":
[
"homebrew",
"N64"
]
,
"source": "Video Game Sage"
}
,
{
"position": 6,
"title": "N64 Homebrew",
"link": "https://www.etsy.com/market/n64_homebrew",
"redirect_link": "https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.etsy.com/market/n64_homebrew&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQFnoECDsQAQ",
"displayed_link": "https://www.etsy.com › market › n64_homebrew",
"thumbnail": "https://serpapi.com/searches/68d6a5a3bf04d540a5b150d9/images/c9365df4eded7fe24defe0456a9c1dfc2c36932956dc03b1aafdd4458b9a7a59.jpeg",
"favicon": "https://serpapi.com/searches/68d6a5a3bf04d540a5b150d9/images/c9365df4eded7fe24defe0456a9c1dfc8a0fd717d09532e1cfde6ccf8156cc3b.png",
"snippet": "Check out our n64 homebrew selection for the very best in unique or custom, handmade pieces from our video games shops.",
"snippet_highlighted_words":
[
"n64 homebrew"
]
,
"rich_snippet":
{
"bottom":
{
"detected_extensions":
{
"rating": 4.7,
"reviews": 1600
}
,
"extensions":
[
"4.7 store rating (1.6K)"
]
}
}
,
"source": "Etsy"
}
,
{
"position": 7,
"title": "What are the best ROM Hacks and Homebrews I can ...",
"link": "https://www.reddit.com/r/n64/comments/xl4x55/what_are_the_best_rom_hacks_and_homebrews_i_can/",
"redirect_link": "https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.reddit.com/r/n64/comments/xl4x55/what_are_the_best_rom_hacks_and_homebrews_i_can/&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQFnoECDcQAQ",
"displayed_link": "20+ comments · 3 years ago",
"favicon": "https://serpapi.com/searches/68d6a5a3bf04d540a5b150d9/images/c9365df4eded7fe24defe0456a9c1dfc37ad5784474b49c9b38798e1b96b893c.png",
"snippet": "What are the best ROM Hacks and Homebrews I can download for this baby, and where do I find them? I've only used Archive.org for ROMs.",
"snippet_highlighted_words":
[
"Homebrews"
]
,
"source": "Reddit · r/n64"
}
,
{
"position": 8,
"title": "N64 Homebrew",
"link": "https://www.ebay.com/shop/n64-homebrew?_nkw=n64+homebrew",
"redirect_link": "https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.ebay.com/shop/n64-homebrew%3F_nkw%3Dn64%2Bhomebrew&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQFnoECD0QAQ",
"displayed_link": "https://www.ebay.com › ... › Video Games",
"thumbnail": "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcR-QIAmI0Moh_DC_oqEWIiu8S7ExQWV-B4A_KtHS26OpWOiofLAz9Qq&usqp=CAE&s",
"favicon": "https://serpapi.com/searches/68d6a5a3bf04d540a5b150d9/images/c9365df4eded7fe24defe0456a9c1dfcc57de0981d94ea0648e75859da6eca13.png",
"snippet": "N64 homebrew refers to unofficial games or software created by hobbyists and developers for the Nintendo 64 platform. These are not commercially released ...",
"snippet_highlighted_words":
[
"N64 homebrew"
]
,
"rich_snippet":
{
"bottom":
{
"detected_extensions":
{
"price_from": 22,
"price_to": 153,
"currency": "$",
"rating": 4.6,
"reviews": 3100
}
,
"extensions":
[
"4.6 store rating (3.1K)",
"$22 to $153"
]
}
}
,
"source": "eBay"
}
,
{
"position": 9,
"title": "N64 Homebrew - Collection by torte00",
"link": "https://itch.io/c/1920454/n64-homebrew",
"redirect_link": "https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://itch.io/c/1920454/n64-homebrew&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQFnoECDkQAQ",
"displayed_link": "https://itch.io › n64-homebrew",
"thumbnail": "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSVWQvizhGmgBrbK_1VB9o94eG_drf7yk3QzGjB5Ow&usqp=CAE&s",
"favicon": "https://serpapi.com/searches/68d6a5a3bf04d540a5b150d9/images/c9365df4eded7fe24defe0456a9c1dfcfb5a3ffa646d048710e09b43c965a208.png",
"snippet": "Whip around and have a rhythmical whack with friends! Zhamul. Action. [N64] Telocation: Gemini. $15. Winner of the N64 Homebrew Jam #1. teamultrarare. Puzzle.",
"snippet_highlighted_words":
[
"N64 Homebrew"
]
,
"source": "itch.io"
}
]
,
"related_searches":
[
{
"block_position": 1,
"query": "N64 homebrew switch",
"link": "https://www.google.com/search?sca_esv=0a869f774643a72d&gl=us&hl=en&q=N64+homebrew+switch&sa=X&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ1QJ6BAgwEAE",
"serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=N64+homebrew+switch"
}
,
{
"block_position": 1,
"query": "N64 homebrew download",
"link": "https://www.google.com/search?sca_esv=0a869f774643a72d&gl=us&hl=en&q=N64+homebrew+download&sa=X&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ1QJ6BAg2EAE",
"serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=N64+homebrew+download"
}
,
{
"block_position": 1,
"query": "N64 homebrew reddit",
"link": "https://www.google.com/search?sca_esv=0a869f774643a72d&gl=us&hl=en&q=N64+homebrew+reddit&sa=X&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ1QJ6BAg1EAE",
"serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=N64+homebrew+reddit"
}
,
{
"block_position": 1,
"query": "N64 homebrew physical",
"link": "https://www.google.com/search?sca_esv=0a869f774643a72d&gl=us&hl=en&q=N64+homebrew+physical&sa=X&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ1QJ6BAg0EAE",
"serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=N64+homebrew+physical"
}
,
{
"block_position": 1,
"query": "N64 homebrew cartridge",
"link": "https://www.google.com/search?sca_esv=0a869f774643a72d&gl=us&hl=en&q=N64+homebrew+cartridge&sa=X&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ1QJ6BAgzEAE",
"serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=N64+homebrew+cartridge"
}
,
{
"block_position": 1,
"query": "N64 emulator",
"link": "https://www.google.com/search?sca_esv=0a869f774643a72d&gl=us&hl=en&q=N64+emulator&sa=X&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ1QJ6BAgxEAE",
"serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=N64+emulator"
}
,
{
"block_position": 1,
"query": "N64 ROMs",
"link": "https://www.google.com/search?sca_esv=0a869f774643a72d&gl=us&hl=en&q=N64+ROMs&sa=X&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ1QJ6BAgyEAE",
"serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=N64+ROMs"
}
,
{
"block_position": 1,
"query": "Best N64 homebrew games",
"link": "https://www.google.com/search?sca_esv=0a869f774643a72d&gl=us&hl=en&q=Best+N64+homebrew+games&sa=X&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ1QJ6BAgvEAE",
"serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=Best+N64+homebrew+games"
}
]
,
"pagination":
{
"current": 1,
"next": "https://www.google.com/search?q=n64+homebrew&sca_esv=0a869f774643a72d&gl=us&hl=en&ei=pKXWaJW2KIqrptQPnJ6twQE&start=10&sa=N&sstk=Af77f_f1wpnJ4C7j8ab_Zhq6PUzy3nTlrkTWP9sViu7xh4ghfuHopsswSmKPCRiUI9EmehpGZkECRGzrkEjbwScXS-2ffty2bOW-rQ&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ8NMDegQICxAW",
"other_pages":
{
"2": "https://www.google.com/search?q=n64+homebrew&sca_esv=0a869f774643a72d&gl=us&hl=en&ei=pKXWaJW2KIqrptQPnJ6twQE&start=10&sa=N&sstk=Af77f_f1wpnJ4C7j8ab_Zhq6PUzy3nTlrkTWP9sViu7xh4ghfuHopsswSmKPCRiUI9EmehpGZkECRGzrkEjbwScXS-2ffty2bOW-rQ&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ8tMDegQICxAE",
"3": "https://www.google.com/search?q=n64+homebrew&sca_esv=0a869f774643a72d&gl=us&hl=en&ei=pKXWaJW2KIqrptQPnJ6twQE&start=20&sa=N&sstk=Af77f_f1wpnJ4C7j8ab_Zhq6PUzy3nTlrkTWP9sViu7xh4ghfuHopsswSmKPCRiUI9EmehpGZkECRGzrkEjbwScXS-2ffty2bOW-rQ&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ8tMDegQICxAG",
"4": "https://www.google.com/search?q=n64+homebrew&sca_esv=0a869f774643a72d&gl=us&hl=en&ei=pKXWaJW2KIqrptQPnJ6twQE&start=30&sa=N&sstk=Af77f_f1wpnJ4C7j8ab_Zhq6PUzy3nTlrkTWP9sViu7xh4ghfuHopsswSmKPCRiUI9EmehpGZkECRGzrkEjbwScXS-2ffty2bOW-rQ&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ8tMDegQICxAI",
"5": "https://www.google.com/search?q=n64+homebrew&sca_esv=0a869f774643a72d&gl=us&hl=en&ei=pKXWaJW2KIqrptQPnJ6twQE&start=40&sa=N&sstk=Af77f_f1wpnJ4C7j8ab_Zhq6PUzy3nTlrkTWP9sViu7xh4ghfuHopsswSmKPCRiUI9EmehpGZkECRGzrkEjbwScXS-2ffty2bOW-rQ&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ8tMDegQICxAK",
"6": "https://www.google.com/search?q=n64+homebrew&sca_esv=0a869f774643a72d&gl=us&hl=en&ei=pKXWaJW2KIqrptQPnJ6twQE&start=50&sa=N&sstk=Af77f_f1wpnJ4C7j8ab_Zhq6PUzy3nTlrkTWP9sViu7xh4ghfuHopsswSmKPCRiUI9EmehpGZkECRGzrkEjbwScXS-2ffty2bOW-rQ&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ8tMDegQICxAM",
"7": "https://www.google.com/search?q=n64+homebrew&sca_esv=0a869f774643a72d&gl=us&hl=en&ei=pKXWaJW2KIqrptQPnJ6twQE&start=60&sa=N&sstk=Af77f_f1wpnJ4C7j8ab_Zhq6PUzy3nTlrkTWP9sViu7xh4ghfuHopsswSmKPCRiUI9EmehpGZkECRGzrkEjbwScXS-2ffty2bOW-rQ&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ8tMDegQICxAO",
"8": "https://www.google.com/search?q=n64+homebrew&sca_esv=0a869f774643a72d&gl=us&hl=en&ei=pKXWaJW2KIqrptQPnJ6twQE&start=70&sa=N&sstk=Af77f_f1wpnJ4C7j8ab_Zhq6PUzy3nTlrkTWP9sViu7xh4ghfuHopsswSmKPCRiUI9EmehpGZkECRGzrkEjbwScXS-2ffty2bOW-rQ&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ8tMDegQICxAQ",
"9": "https://www.google.com/search?q=n64+homebrew&sca_esv=0a869f774643a72d&gl=us&hl=en&ei=pKXWaJW2KIqrptQPnJ6twQE&start=80&sa=N&sstk=Af77f_f1wpnJ4C7j8ab_Zhq6PUzy3nTlrkTWP9sViu7xh4ghfuHopsswSmKPCRiUI9EmehpGZkECRGzrkEjbwScXS-2ffty2bOW-rQ&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ8tMDegQICxAS",
"10": "https://www.google.com/search?q=n64+homebrew&sca_esv=0a869f774643a72d&gl=us&hl=en&ei=pKXWaJW2KIqrptQPnJ6twQE&start=90&sa=N&sstk=Af77f_f1wpnJ4C7j8ab_Zhq6PUzy3nTlrkTWP9sViu7xh4ghfuHopsswSmKPCRiUI9EmehpGZkECRGzrkEjbwScXS-2ffty2bOW-rQ&ved=2ahUKEwiVmI-I1PaPAxWKlYkEHRxPKxgQ8tMDegQICxAU"
}
}
,
"serpapi_pagination":
{
"current": 1,
"next_link": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=n64+homebrew&start=10",
"next": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=n64+homebrew&start=10",
"other_pages":
{
"2": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=n64+homebrew&start=10",
"3": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=n64+homebrew&start=20",
"4": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=n64+homebrew&start=30",
"5": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=n64+homebrew&start=40",
"6": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=n64+homebrew&start=50",
"7": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=n64+homebrew&start=60",
"8": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=n64+homebrew&start=70",
"9": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=n64+homebrew&start=80",
"10": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&q=n64+homebrew&start=90"
}
}
}Now, collating this data is more than enough for a basic Dscape analysis. If a client gives us 100 keywords in 5 locations, that’ll give us 500 SERP queries and 50,000 organic results and a few thousand non-organic results.
Analysing it
With this, we can provide an analysis on who are the major players in this area. Using the volume data from Adwords, we can estimate the amount of traffic that each ‘rank’ gets per month.
For example, if a keyword has a volume of 1,000 and we assume that 33.58% of people click on the first link, 16.42 click on the second one, then we can deduce that this keyword drives 335 people to the first website and 164 to the second one.
We’d then aggregate all this data and find which websites are the most likely to appear among all the competitors and how the client’s website compares. This gives us a good idea of where there are some easy wins, and where we can further focus their SEO efforts.
Enriching the results
Just getting a list of website with their ranking is good enough for most clients, but for those that want to get a little bit more out of their dscape, we can further enrich the data by either crawling the pages themselves or by analysing the companies that appear on the results.
Crawling the results
Since the SERP results have given us a very long list of URLs, those can be crawled and analysed.
The simplest way to analyse them is to search for specific keywords. This could be a mention of a brand name, product category or whatever the situation calls for.
The more complicated way would be to use a LLM to read the page and give us some sentiment analysis. In this case we used IBM Watson to do the work for us. This is a little bit more complicated, but the results gave us a deeper understanding of what the sentiment was around particular keywords and how to approach writing pages about them.
Company analysis
We can also improve the quality of our results by enhancing them with company firmographics. These are obtained by taking a domain and sending it over to a tool called Clearbit, which gives us company info if it exists.
Some of the info is fairly simple like a company name, HQ location or headcount, but there are others that are lists like categories, technology or tags.
Exporting
All this data would then be collated into a giant xlsx file with different tabs for each result type, and as many columns as the user requires. This is then sent over to the analytics team which analyses it and extracts any useful insights for the client.
Dscape under the hood
Dscape is made using a combination of various different technologies and programming languages. Let’s tackle one at a time.
Database
The heart of the whole project is the database. It lets the program save data for use later for various purposes.
It saves the state of a process, such as how many keywords have been processed recently to prevent re-crawls. It also is where we store the results of all of our operations to prevent anything from being stored in files on the server.
Here is the structure of the database. It is all in fifth normal form (5NF) except maybe the country tables which might have some redundancy.
The client process is built from left to right, with the most important tables being keywords, the various results tables and domains. While results are enough for a Dscape, the enrichment provided by the other tasks are what really give it its power.
The only disadvantage of having it set up this way is that all the joins cause it to be a bit slow, so temporary sub-tables are useful when running processes to speed things up.
Source code
The source code is built using various languages. Technically it is still owned by the company so I can’t share the source code, but I can still tell you about it.
Dscape was meant to be a web app that you could access through the browser. Ideally it was going to be sold as a SaaS product so we’d have user authentication and everything but that was out of scope so we kept it as an internal tool.
The backend is written in PHP to keep with the theme of a web-based application, so that is what is used to generate all the HTML on the pages. The pages themselves have a basic CSS that I wrote to make it look like a terminal (even though it is clearly structured like a website). Javascript with jQuery is used to provide interactivity, mostly for selecting tabs and performing AJAX commands to the PHP scripts.
Since most of the commands are quite long (processing 100 keywords or domains takes a while) the scripts were made to be modular and iterable to prevent them from timing out and having to start again. So in a certain way, the JS interface controls the flow commands while the backend does the hard work, one at a time.
An example
When running the SERP crawl, the user can specify a previously-made client (or none) and then press the start button. That’s all the input needed.
The JS will then ask the PHP script to check the DB for the client’s keywords, sorting them by oldest first (uncrawled keywords are set to 01-01-0000). It will then contact the SERP API to get the results one by one until all keywords are exhausted. ‘Exhausted’ meaning “keywords that have not been crawled in 30 days” to allow keywords to be re-crawled after a period of time, since SERPs dont really change that often.
Once the PHP script get a response from the SERP API, it will check which results it got (organic, paid, etc) and place them in the appropriate tables in the DB.
At this time, it will also check to see if there are any new domains from these results, in which case this domain will be added to the domains list.
Finally it will record the time taken to process this keyword and feed this information back to the user interface to let them know whether the crawl was successful. It will then proceed to the next keyword.
All of the different functions work mostly the same. UI -> JS AJAX request -> PHP/DB -> AJAX response -> HTML feedback
APIS
APIs are at the heart of Dscape. They serve to get access to large swathes of data from around the internet easily. Here’s a list of 3rd party APIs that the project relied on:
- Google Adwords (now Google Ads) – keyword stats
- SERP API – search engine results
- Clearbit – company firmographic enrichment
- Google Geolocation – getting city/country info from latitude/longitude
- Restcountries – country info
- IBM Watson – Natural Language Understanding
- Linkedin – More firmographics
Most of these were free, but in total it is about $200-300 per month
Infrastructure
The project was originally run on the same webserver we used for hosting the website as we didn’t have any other Linux machine at the time that could host it.
As we got clients on board, we could justify the costs of hosting it on its own infrastructure so it was moved to AWS. The setup was fairly simple:
- Containerising on Docker
- Hosting on Elastic Beanstalk (EBS)
- Database on Relational Database Service (RDS)
- SSL management on Load Balancer & Certificate Manager
The configuration for all of this was set up on scripts within the git repository to ensure consistency between deployments.
CI/CD
Continuous integration and continuous delivery (CI/CD) is the process of automating development deployment steps to make it easier to move from writing code to seeing it running in a dev/staging/prod environment.
I set up a CI/CD pipeline using Gitlab to automate deployment to staging and to prod (Dev was run locally for speed). It would take in all the environment variables (API keys), paste them into the target platform and put all the files in there as well. It would also set up all the configurations for the load balancer, server timeouts and more.
This made it easy to test and redeploy the system any time there was an update.
Things that could have been done better
While the system I built worked well, it did have a few things that would cause hiccups.
First, it is dependent on AWS and isn’t transferrable to another platform. A lot of how it works relies on the way that AWS is set up and the configuration was very finicky; typically when something failed it was because AWS changed the way something had to be set up. If the staging environment was down at the time when the SSL certificate had to be renewed (a cost-saving measure), a whole kerfuffle has to be reconfigured to get it up and running again.
It is also very reliant on external APIs. While most of them worked without an issue using CURL, the Google Ads one was constantly expiring every few months. I’d put in a service test to alert us when it was about to expire so we have time to upgrade. Some unit testing would be good too.
Finally the biggest change I’d make: since the end product was never made available directly to the clients, I’d have redone the whole thing in Python using CLI. Using the C-enhanced libraries plus not having to rely on HTTP request for running commands, it would be much easier to automate without requiring the user to run a browser tab constantly. Later down the line we could hire a front-end developer to make a pretty visualiser that a normal user could interface with.
Conclusion
Dscape is one of my best professional achievements, for what it’s worth. I worked on it over the course of a year or so alongside other projects, but it is something that I built by myself from start to finish. There were suggestions made by other people in the team and our clients which helped guide the way.
I think that the thing that I did the best was to make the database resilient to redundant data by employing fifth normal form. At no point was there ever an issue with ambiguity.
The part that caused the most headache was the hosting and configuration. If I kept things simpler by running a single python script, it would have been much faster to run and easier to work on. I could even get more people to help build it..
In the end, Dscape was a practical solution that gave clients real visibility into their digital position and opportunities for growth. While there are areas I would approach differently today the system successfully bridged the gap between raw SEO data and actionable insights. More importantly, it demonstrated how a well-designed tool can evolve from an internal experiment into a scalable framework for delivering lasting value to clients.
