google-serp-scraper

This script currently parses the first page of the Google Search Engine Results Page.

# google-serp-scraper

This script currently parses the first page of the Google Search Engine Results Page.

Example usage:

python3 main.py [keyword to search for] [your fqdn to search for]

<h3>$ python3 main.py "apline" runestone.academy</h3>

1: www.instagram.com/a.p.line/%3Fhl%3Den

2: www.ap-line.com/products/apline-set  

3: secure-media.collegeboard.org/apc/ap10_computer_science_a_q2.pdf  

!!! 4: runestone.academy/ns/books/published/csawesome/Unit5-Writing-Classes/APLine.html

!!! You rank 4 for "runestone.academy/ns/books/published/csawesome/Unit5-Writing-Classes/APLine.html" with keyword: "apline"

5: www.alpine-usa.com/products/receivers  

6: ww.cs.ucf.edu/~dmarino/hs/apcsa/aptestsols/2010/APLine.java  

7: www.etsy.com/market/apline  

8: apcomputersciencetutoring.com/exam-review/apline-free-response-answer/  

9: soundcloud.com/apline  

10: www.ancestry.com/name-origin%3Fsurname%3Dapline  

11: www.youtube.com/watch%3Fv%3DL6-jGSL-V9g  

12: www.carpet-wholesale.com/colors-291586/carpet/aladdin-carpet/stylish-story-ii-12'.php  

13: www.amazon.com/apline-Trunk-Nissan-Qashqai-2014-2021/dp/B09HKZ9L2X  

14: www.amazon.com/apline-Cross-2009-2022-Luggage-Carrier/dp/B09RWRGDZD  

15: beside-u.com/apline-crossbody/  

16: www.needen.com/liberty-bags-lb8722-apline-fleece-mink-touch-luxury-baby-blanket-406517/c15803-soft-yellow/s760291-one%3Fformat%3Damp  

...

95: www.pacsun.com/guess-originals/apline-vintage-logo-t-shirt-0097513380105.html  

96: www.spellcheck.net/misspelled-words/apline  

97: www.tiktok.com/tag/apline%3Flang%3Den  

98: ww.lakesaudio.com/Audio_Upgrade  

99: www.reedssports.com/saltwater-apline-duck-boot-size-10

Copy to Clipboard

import argparse
import bs4
import requests
import re

def parse_links(your_domain, keyword):
    '''Finds and parses a list of the first Google SERP.'''
    full_link = []
    response = requests.get(f"https://www.google.com/search?num=100&q={keyword}")
    soup = bs4.BeautifulSoup(response.text, 'html.parser')
    links = soup.find_all('a')

for link in links:
        if link:
            parsed_link = re.findall('\A<a href="/url\?.*?"><', str(link))
            for link in parsed_link:
                sliced_link = link[24:-3]
                split_link = sliced_link.split('&')
                full_link.append(split_link[0])

return full_link

def parse_sitelinks(fqdn_item_split):
    'Filter out Google Sitelinks.'
    prev_item = ""
    parsed_fqdn = []
    for item in fqdn_item_split:
        if item != prev_item:
            parsed_fqdn.append(item)
        prev_item = item

return parsed_fqdn

def get_rank(parsed_fqdn, your_domain, keyword):
    '''Gets current pagerank of domain on Google Search Engine Results Page.'''
    i = 1
    for item in parsed_fqdn:
        link = item.split('/')
        if link[0] == your_domain:
            print(f"!!! {i}: {item}\n!!! You rank {i} for \"{item}\" with keyword: \"{keyword}\"")
        else:
            print(f"{i}: {item}  ")

i = i + 1
    return

if __name__=="__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("keyword", help="Provide a keyword to search for (e.g. apline).")
    parser.add_argument("your_domain", help="The FQDN of your website (e.g. soundcloud.com).")
    args = parser.parse_args()
    get_rank(parse_sitelinks(parse_links(args.your_domain, args.keyword)), args.your_domain, args.keyword)

Check the latest work on this project at: Scrape Google SERPs

Author
Recent Posts

Jesse Shelley

Webmaster at CyberTutorials.org

I have many years of experience as a criminal investigator, evidence custodian, IT administrator, and as an information security specialist.

I've performed dozens of digital forensic examinations on cases involving criminal matters such as murder, domestic terrorism, sexual assault, child exploitation, CSAM, CSEC, illegal wiretapping, bomb threats, kidnapping, military, corporate, and casework from alphabet agencies such as the FBI and DEA.

Scrape Google SERPs with Python

Scrape Google SERPs with Python

google-serp-scraper

fuzzbuster

Code Snippet (C#): Enumerate Active Directory Domain Accounts (sAMAccountName)

Symmetric Encryption – Bidirectional Python Socket

Python Socket Programming – Simple Bidirectional Communication

Spin the Content of a Message in Python

Leave A Comment Cancel reply

Scrape Google SERPs with Python

Scrape Google SERPs with Python

google-serp-scraper

Related Posts

fuzzbuster

Code Snippet (C#): Enumerate Active Directory Domain Accounts (sAMAccountName)

Symmetric Encryption – Bidirectional Python Socket

Python Socket Programming – Simple Bidirectional Communication

Spin the Content of a Message in Python

Leave A Comment Cancel reply