MIRA GST lookup scraper

Scrapes https://www.mira.gov.mv/Pages/View/gstlookup by incrementing the first 7 digits of the GST number (1000001GST501, 1000002GST501, …) and saves the results as CSV, to Google Drive if you want.

How it works

• Uses a headless Chromium (Playwright) to type each number into the lookup form, so it doesn't depend on the site's internal API. JSON (XHR) responses and HTML result tables are both flattened into CSV columns.
• Rate limiting: waits --delay seconds plus a random 0..--jitter seconds between lookups (default 3–5 s). On HTTP 429/5xx or errors it backs off exponentially (up to --max-backoff, default 300 s) and retries up to --retries times.
• Resumable: the CSV and a <out>.progress file are saved every --save-every numbers. Re-running the same command continues from where it stopped.

Option A: Google Colab (easiest way to save to Drive)

Open gst_scraper_colab.ipynb in Colab (File → Upload notebook, or open it from GitHub) and run the cells. It mounts your Drive and writes to MyDrive/mira_gst/gst_results.csv.

Option B: Run locally

pip install -r requirements.txt
playwright install chromium

# See what the lookup page sends and returns for one number
python gst_scraper.py --discover --start 1000001

# Scrape a range
python gst_scraper.py --start 1000001 --end 1010000 --delay 3 --jitter 2 --out gst_results.csv

To upload to Google Drive from a local run, create a Google Cloud service account with the Drive API enabled, share a Drive folder with the service account's email, and add:

--drive-folder-id <FOLDER_ID> --drive-credentials service_account.json

If you use Google Drive for desktop, you can instead just point --out into your synced Drive folder.

Useful options

Option	Default	Meaning
--start / --end	1000001 / 9999999	7-digit range to scan
--suffix	GST501	text after the 7 digits
--delay / --jitter	3 / 2	seconds between requests
--keep-empty	off	also write rows for numbers with no result
--stop-after-misses N	0 (off)	stop after N consecutive not-found numbers
--input-selector / --button-selector	auto	CSS selectors if auto-detection picks the wrong field
--headed	off	show the browser window

Please respect MIRA's terms of use, and keep the delay reasonable.