One of the challenges when trying to test any security product is to try and generate some traffic that will trigger the security features to actually work.

This is where my generate-url-requests.py python script comes in handy!

At a high-level this python script will download Cisco Umbrella’s Top 1 million hostnames from the previous day and attempt to access them.

Some useful things about this script:

• It will by default output the results to stdout about what it’s doing.
• Using the --outputfile argument you can write the output to a file as well.
• You can define how many hosts you want to grab from the sample csv file (defaults to 10000).
• You can specify the number of workers that will attempt to go out and connect to hosts (defaults to 10), which means…
• It’s multi-threaded! 🙂

Now if you have a security device that’s performing some sort of SSL/TLS decryption, you’ll need to make sure python knows where to find the Corporate Root CA otherwise you’ll get a lot of Decryption errors from the script.

To determine where the extra Corporate certificates need to go, get into the python interpreter.

>>> import certifi
>>> certifi.where()
'/etc/ssl/certs/ca-certificates.crt'

Some example locations:

Debian	/etc/ssl/certs/ca-certificates.crt
macOS	/opt/homebrew/lib/python3.11/site-packages/certifi/cacert.pem
Windows	C:\\Users\\admin\\AppData\\Local\\Programs\\Python311\\Lib\\site-packages\\certifi\\cacert.pem

Take note of the double-back slashes that python is using to escape the single backslash.

Now that we have the certificate location, we’re going to put any Corporate CA into those locations based on your operating system of choice. The Corporate CA is a Base64 encoded file that if you were to open it up, it would look something like this:
-----BEGIN CERTIFICATE-----
<random series of characters>
-----END CERTIFICATE-----

Copy the contents and paste it at the end of the files identified by the certifi.where() python function.

Let’s march on to run the script. Generating the help screen.

$ python3 generate-requests.py -h
usage: generate-requests.py [-h] [--cleanup] [--outputfile OUTPUTFILE] [num_connections] [num_workers]

Connect to random hostnames with multiple threads.

positional arguments:
  num_connections       Number of connections to establish. Default 100.
  num_workers           Number of worker threads. Default 10.

optional arguments:
  -h, --help            show this help message and exit
  --cleanup             Clean up downloaded files and exit.
  --outputfile OUTPUTFILE
                        Save output to a file.

To run without any arguments will attempt to connect to 100 hosts with 10 worker threads.

This is achieved by running python3 generate-requests.py.

Here is some sample output (I’ve summarized some of the output).

$ python3 generate-requests.py 
Attempting to download source file http://s3-us-west-1.amazonaws.com/umbrella-static/top-1m-2023-10-23.csv.zip
Download successful.
Saving contents to file.
Save successful.
Attempting to extract top-1m.csv.zip
Extraction successful.
Thread ID: 140289217894144, Error connecting to https://a958.casalemedia.com: HTTPSConnectionPool(host='a958.casalemedia.com', port=443): ...
Thread ID: 140289217894144, Error connecting to https://intl.cloud.tencent.com: HTTPSConnectionPool(host='intl.cloud.tencent.com', port=443): ...
Thread ID: 140289354323712, Error connecting to https://imumugosion-prezuniment-interabirolen.org: HTTPSConnectionPool(host='imumugosion-prezuniment-interabirolen.org', port=443): ...
Thread ID: 140289251464960, Error connecting to https://a1873.dscb.akamai.net: HTTPSConnectionPool(host='a1873.dscb.akamai.net', port=443): ...
Thread ID: 140289234679552, Hostname: www-psychologytoday-com.cdn.ampproject.org, Status Code: 404 (Not Found)
Thread ID: 140289345931008, Error connecting to https://frownlinesandcrowsfeet.com: HTTPSConnectionPool(host='frownlinesandcrowsfeet.com', port=443): ...
^CExiting due to Ctrl+C
Thread ID: 140289354323712, Hostname: b7zuvcmijd.execute-api.us-east-1.amazonaws.com, Status Code: 403 (Forbidden)
Thread ID: 140289354323712 is exiting. Remaining threads: 10 of 10
Thread ID: 140289345931008, Hostname: ustvstaticcdn1-a.akamaihd.net, Status Code: 200 (OK)
Thread ID: 140289345931008 is exiting. Remaining threads: 9 of 10
Thread ID: 140289226286848, Hostname: l.costacoffee.club, Status Code: 200 (OK)
Thread ID: 140289226286848 is exiting. Remaining threads: 8 of 10
Thread ID: 140289243072256, Hostname: dcintl2.push.oppomobile.com, Status Code: 404 (Not Found)
Thread ID: 140289243072256 is exiting. Remaining threads: 7 of 10
Thread ID: 140288664270592, Error connecting to https://idc.seller.shopee.co.th: HTTPSConnectionPool(host='idc.seller.shopee.co.th', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError("...")))
Thread ID: 140288664270592 is exiting. Remaining threads: 6 of 10
Thread ID: 140289268250368, Hostname: hottrenddance.xyz, Status Code: 404 (Not Found)
Thread ID: 140289268250368 is exiting. Remaining threads: 5 of 10
Thread ID: 140289251464960, Hostname: vanandelinstitute.sharepoint.com, Status Code: 403 (Forbidden)
Thread ID: 140289251464960 is exiting. Remaining threads: 4 of 10
Thread ID: 140289259857664, Error connecting to https://display.api.mina.mi.com: HTTPSConnectionPool(host='display.api.mina.mi.com', port=443): ...
Thread ID: 140289259857664 is exiting. Remaining threads: 3 of 10
Thread ID: 140289234679552, Error connecting to https://hadi.eitaa.com: HTTPSConnectionPool(host='hadi.eitaa.com', port=443): ...
Thread ID: 140289234679552 is exiting. Remaining threads: 2 of 10
Thread ID: 140289217894144, Error connecting to https://mx2.hc169-40.ca.iphmx.com: HTTPSConnectionPool(host='mx2.hc169-40.ca.iphmx.com', port=443): ...
Thread ID: 140289217894144 is exiting. Remaining threads: 1 of 10
All worker threads have completed.

Thanks for reading! I’m always open to improve the things I create. Feel free to look at my Github Repository.