--- title: Using a Squid Container to Quickly Capture Requested URLs description: Learn how to quickly capture requested URLs using a Squid container and WSL to streamline your proxy whitelist process. image: https://blog.arkahna.io/hubfs/blog-images/Squid%20cache.jpeg --- [Skip to the main content.](https://blog.arkahna.io/using-a-squid-container-to-quickly-capture-requested-urls#main-content) [![Arkahna_Brand+Word-Horizontal-WHITE](https://blog.arkahna.io/hs-fs/hubfs/logos/Arkahna_Brand+Word-Horizontal-WHITE.png?width=4000&height=1445&name=Arkahna_Brand+Word-Horizontal-WHITE.png "Arkahna_Brand+Word-Horizontal-WHITE")](https://arkahna.io) [![Arkahna_Brand+Word-Horizontal](https://blog.arkahna.io/hs-fs/hubfs/logos/Arkahna_Brand+Word-Horizontal.png?width=2286&height=826&name=Arkahna_Brand+Word-Horizontal.png "Arkahna_Brand+Word-Horizontal")](https://arkahna.io) - [Plan](https://arkahna.io/plan) - [Build](https://arkahna.io/build) - [Accelerate](https://arkahna.io/accelerate) - [FeatureBoard](https://arkahna.io/featureboard) - [Marketplace Elements](https://arkahna.io/accelerate/marketplace-elements) - [Run](https://arkahna.io/run) - [Sigil](https://arkahna.io/sigil) - [Blog](https://blog.arkahna.io) Toggle Menu Toggle Menu - [Plan](https://arkahna.io/plan) - [Build](https://arkahna.io/build) - [Accelerate](https://arkahna.io/accelerate) - [FeatureBoard](https://arkahna.io/featureboard) - [Marketplace Elements](https://arkahna.io/accelerate/marketplace-elements) - [Run](https://arkahna.io/run) - [Sigil](https://arkahna.io/sigil) - [Blog](https://blog.arkahna.io) - [Home](https://arkahna.io) - [FeatureBoard](https://arkahna.io/featureboard) - [Sigil](https://arkahna.io/sigil) - [Blog](https://blog.arkahna.io) [Facebook](https://www.facebook.com/arkahna) [Linkedin](https://www.linkedin.com/company/arkahna) [X](https://twitter.com/ArkahnaAU) 3 min read # Using a Squid Container to Quickly Capture Requested URLs [![Picture of Ryan Royals](https://blog.arkahna.io/hubfs/ryan%20royals.png) Ryan Royals](https://blog.arkahna.io/author/ryan-royals) : Feb 27, 2025 8:58:03 AM [linux](https://blog.arkahna.io/tag/linux) [data](https://blog.arkahna.io/tag/data) ![Using a Squid Container to Quickly Capture Requested URLs](https://blog.arkahna.io/hubfs/blog-images/Squid%20cache.jpeg) Odds are whether you work in any Application or Infrastructure team, a someone will someday ask you "What are the URLs you need to get added to our whitelist proxy". Your first response is probably going to be to Google away your app stack, trying to figure out who sings out to what, but hopefully before wasting hours reading heaps of doco sources, you will think to just capture the traffic you need and go from there. There are plenty of ways to capture network traffic, a immediate way that springs to mind is Wireshark. This tool works great, but is better suited to capturing Layer 3/4 traffic (IP's and Ports), than it is capturing the Layer 7 side of the world (URLs). Wireshark is also very noisy in the logs as you will most likely be capturing your primary Network Adapter, and every app wants to sing home every 5 minutes to check if someone has sent you a cat gif. ![light saber fighting GIF](https://blog.arkahna.io/hs-fs/hubfs/blog-images/light%20saber%20fighting%20GIF.gif?width=250&height=188&name=light%20saber%20fighting%20GIF.gif) For my use case, I wanted to capture traffic from Terraform, so we can run a `terraform init`, `terraform plan` and `terraform apply` with all the Providers and Modules downloading as we go. I chose to use [Squid](https://www.squid-cache.org/) as a proxy server as I have used it before, and I found a lightweight [Container Image](https://github.com/sameersbn/docker-squid) I could use to spin it up on my machine without much hassle. I also have opted to use [Windows Subsytem for Linux (WSL)](https://learn.microsoft.com/en-us/windows/wsl/about) as a clean environment to run Terraform from, as that way I don't have a heap of noisy applications ruining my logs. From here I have a few steps to get the end result that I want: **Download and Configure the container**, **Configure WSL to use the Proxy** and ultimately **Log the traffic that I need**. The below is a simple guide on how I got this done. There are some optimizations to the workflow that could be improved I'm sure, but I just wanted to get in and out without spending 10 hours to make 10 seconds work easier (Just to try something different this time), and I want the confidence on the journey to not make other issues to diagnose. ## 1. Download and Configure the Container *Assuming you have* [*Docker*](https://docs.docker.com/engine/install/) *already installed.* *This guide is an abbreviation from the* [*readme in the containers' repo*](https://github.com/sameersbn/docker-squid)*.* 1. Pull the container using `docker pull sameersbn/squid:3.5.27-2docker pull sameersbn/squid:3.5.27-2` 2. Start the container using `docker run --name squid -d --restart=always --publish 3128:3128 --volume /srv/docker/squid/cache:/var/spool/squid sameersbn/squid:3.5.27-2` 3. Log into Bash on the container using `docker exec -it squid bash` 4. Now we are in the container, we will need to install a tool to edit a text file. For me I use Vim, but you can substitute whatever tool you want: 1. `apt-get update` 2. `apt-get install vim` 3. `vim /etc/squid/squid.conf` 5. The `/etc/squid/squid.conf` is a very well documented configuration file that can get you configured relatively easily. In the file every line is a comment, so we can delete everything and instead add the line `http_access allow all`. This does what it says on the tin, and and allows all traffic. 6. `exit` from bash and out of the container, and restart the container using `docker restart squid` ## 2. Configure WSL to Use the Proxy Super easy step, but one of those commands that are easy to forget if you don't use everyday: export ftp\_proxy= export http\_proxy= export https\_proxy= ## 3. Run Your Workload to Capture Traffic Now inside of WSL, do what you need to do to capture all the traffic you need. For instance, you might capture detailed records of network traffic—such as packet sizes, source and destination addresses, ports, and protocol types—to reveal communication patterns and any irregularities. It’s also useful to log system events, including process activities, resource usage, and error messages, to help identify performance issues or potential security breaches. ## 4. Check the Logs In your primary shell, log back into the container to grab your logs docker exec -it squid bash cat /var/log/squid/access.log ## 5. Tidy up Hopefully now you have a good log of the traffic that you generated. An easy tip is copy it out of the terminal, and paste it into Excel, using the Text to Column function delimited by Spaces. When you are done done, remember to tidy up the container with `docker stop squid`, and exit from your WSL shell to clear the `export` so you can resume web traffic as per normal. - [Tweet](https://twitter.com/share) [![GitHub Self Hosted Runner on Ubuntu](https://blog.arkahna.io/hubfs/Github%20selfhosted%20runner%20thumbnail.png)](https://blog.arkahna.io/github-self-hosted-runner-on-ubuntu) #### [GitHub Self Hosted Runner on Ubuntu](https://blog.arkahna.io/github-self-hosted-runner-on-ubuntu) ![Picture of Lee Borlace](https://blog.arkahna.io/hs-fs/hubfs/lee%20borlace.png?width=30&name=lee%20borlace.png) [Lee Borlace](https://blog.arkahna.io/author/lee-borlace) : Feb 5, 2024 8:00:00 AM At Arkahna we build solutions for customers with tight security requirements such as government organisations. Often this means that resources like... [github](https://blog.arkahna.io/tag/github) [linux](https://blog.arkahna.io/tag/linux) [Read More](https://blog.arkahna.io/github-self-hosted-runner-on-ubuntu) [![Data Science via VS Code. Part 3: DataFrame with some basic exploratory tasks](https://blog.arkahna.io/hubfs/blog-images/Designer%20(7).png)](https://blog.arkahna.io/data-science-via-vs-code.-part-3-data-frame-with-some-basic-exploratory-tasks) #### [Data Science via VS Code. Part 3: DataFrame with some basic exploratory tasks](https://blog.arkahna.io/data-science-via-vs-code.-part-3-data-frame-with-some-basic-exploratory-tasks) ![Picture of Samuel Parsons](https://blog.arkahna.io/hs-fs/hubfs/profile-pictures/sam%20parsons.jpeg?width=30&name=sam%20parsons.jpeg) [Samuel Parsons](https://blog.arkahna.io/author/samuel-parsons) : Aug 13, 2024 12:31:58 PM Part 1: install, extensions, virtual env. Part 2: Initial Libraries and Data Import Whew! Data is in, virtual environment is up, and we have executed... [AI](https://blog.arkahna.io/tag/ai) [data](https://blog.arkahna.io/tag/data) [Read More](https://blog.arkahna.io/data-science-via-vs-code.-part-3-data-frame-with-some-basic-exploratory-tasks) [![Building a report or solving a problem: the importance of communication at the beginning of a data project](https://blog.arkahna.io/hubfs/blog-images/banner%20question.webp)](https://blog.arkahna.io/building-a-report-or-solving-a-problem-the-importance-of-communication-at-the-beginning-of-a-data-project) #### [Building a report or solving a problem: the importance of communication at the beginning of a data project](https://blog.arkahna.io/building-a-report-or-solving-a-problem-the-importance-of-communication-at-the-beginning-of-a-data-project) ![Picture of Vicki Turner](https://blog.arkahna.io/hs-fs/hubfs/PXL_20210808_061339127.MP%201.jpg?width=30&name=PXL_20210808_061339127.MP%201.jpg) [Vicki Turner](https://blog.arkahna.io/author/vicki-turner) : Feb 17, 2025 12:46:52 PM As a Data Analyst, when given the initial brief for a report that has been requested, the first instinct can be to open Power BI, hook up to the data... [power platform](https://blog.arkahna.io/tag/power-platform) [data](https://blog.arkahna.io/tag/data) [Read More](https://blog.arkahna.io/building-a-report-or-solving-a-problem-the-importance-of-communication-at-the-beginning-of-a-data-project) [![arkahna-brand-word-white](https://blog.arkahna.io/hubfs/arkahna-brand-word-white.png)](https://www.arkahna.io) - [Privacy Policy](https://arkahna.io/privacy) - [Terms of Sale](https://arkahna.io/terms) © 2026 Arkahna Pty Ltd [Facebook](https://www.facebook.com/arkahna)[X](https://twitter.com/ArkahnaAU)[Linkedin](https://www.linkedin.com/company/arkahna) [GitHub](https://github.com/arkahna) We acknowledge the Traditional Custodians of the lands we live and work on, the Aboriginal and Torres Strait Islander peoples of Australia, and the Māori peoples of Aotearoa New Zealand.  We pay our respects to Elders past and present. Return to top ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Ryan Royals", "url" : "https://blog.arkahna.io/author/ryan-royals" }, "dateModified" : "2025-02-27T00:58:03.397Z", "datePublished" : "2025-02-27T00:58:03.000Z", "headline" : "Using a Squid Container to Quickly Capture Requested URLs", "image" : [ "https://blog.arkahna.io/hubfs/blog-images/Squid%20cache.jpeg" ], "mainEntityOfPage" : { "@id" : "https://blog.arkahna.io/using-a-squid-container-to-quickly-capture-requested-urls", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://blog.arkahna.io/hubfs/Arkahna_Brandmark.png" }, "name" : "Arkahna Pty Ltd" } } ```