Advanced Charles Proxy Tutorial for Ad Ops

I’ve already written a number of posts on how to use Charles Proxy for ad operations work, including how to setup Charles for mobile debugging, both over Wi-Fi and cellular networks.  This post is an advanced Charles Proxy tutorial, meant to highlight best practices and advanced tips once you know the basics.  I’ve suggested use cases for each that apply in the ad ops world, or that I use the tool for specifically.

Save / Open Charles Proxy Sessions

If you’ve ever been frustrated when trying to reproduce a technical error, being able to save your Charles Proxy session, send it to someone else, and have them open it on their machine to review the same issue is a godsend. I find this is especially helpful if you need to loop in the engineering team’s help, or send evidence of a particular issue to a vendor’s team.  Charles can even open sessions saved in browser based developer tools as well, so you don’t necessarily need to pay for a Charles license for everyone on your team to benefit from this feature.

Multi-Session Windows

As a best practice, I also try to record each page load I render as a separate session, so I don’t get confused as to what happened on what refresh cycle with my browser.  In this instance Charles Proxy easily outpaces a browser developer tool, which typically requires you to open a new tab if you want to load a new session and seems to consume a huge amount of system resources per session.  For whatever reason Charles is much more efficient in the way it leverages RAM.

Another option on this front is to leverage the Auto-Save feature in Charles, which can automatically save your session every X minutes or so.  Once Charles saves your history to a file, it clears that data from the active session, freeing up system resources.  The multi-session approach makes sense if only need to test for a short period of time, while the Auto-Save feature is a better option if you’ll be testing lots of things over an hour or longer.  For example, if you were validating an ad server migration across a number of websites.

Repeat Advanced

This is a great one for header bidding QA, and catching mobile app store redirects.  Let’s say you wanted needed a sample of responses from the same partner 10, 20, 100 times to catch a bothersome issue that only happened occasionally. You could for example, make 10 requests to the same header bidder and see their average latency or a sample range of bids for your own cookie by using the Repeat Advanced feature in Charles Proxy.  This feature duplicates prior requests and executes them however many times you like.

Step 1: Right click on a request you want to repeat and select Repeat Advanced from the menu:

Step 2: Specify the number of times you want to repeat the request – the default is 10.  You can also specify concurrency, which is how many requests you want to run at the same time, and if you want to delay requests by any particular amount of time.  If you leave it at 1, then each request will run in sequence, one after the other.

Step 3: You’ll see Charles generate your new, identical requests in a separate tab, where you can review them individually.

I find it useful to understand if bidding is consistent or erratic for my own user ID, or if I’m looking for the advertiser making high value bids, I can iterate multiple times and search through the results without having to reload the entire page and sort through all the other activity.

Block Cookies

Cookies make a big difference on user experience in general online, but especially when it comes to ads since virtually all programmatic advertising is user targeted these days.  I’ve found that blocking cookies but allowing ads is often a great way to catch low quality advertisers and adjust ad quality rules in your exchange. The reason for this is cookie-less users are seen as low quality traffic to most advertisers because they aren’t trackable.  This means there is less competition for this inventory, making it cheap to buy, and therefore attractive to bottom-feeder advertisers who often run spammy / viral ad content.  Not to mention ad operations people are real users and tend to have cookies enabled, so targeting cookie-less users is a great way to hide from the ad quality hunters out there.

Another great use case for this is if you have any frequency based products.  I once worked for a publisher that offered a ‘first-frequency’ roadblock product, in which the same ad would serve to any user on their first entry page to the site, irrespective of where that was.  To test that it worked correctly, I used this block cookies feature so that I was always on my first pageview from the ad server’s perspective.

It’s a good idea to only block cookies from selected locations when using this feature – specifically your ad tech partners – as blocking cookies from your content server might change the way your site displays. More general web developers typically use this tool when trying to view their site as a web spider might, for instance.

Black List Domains

The Black List tool in Charles Proxy allows you to block requests to specific domains, and thus see how your site or application performs without those resources involved.  You can use this tool to essentially create your own custom ad blocker on the fly, or perhaps more useful, test how your site might perform from a page load perspective if you removed a specific header bidder.  Charles also has a White List feature if you wanted to approach from the other end of the spectrum, but I find ad tech tends to have too many dependencies to get much use from it.

Charles Proxy Breakpoints

I nearly wrote a separate post on this feature.  It’s one of the more brilliant options in Charles, and one not available in browser-based developer tools (at least in the same way).  Breakpoints allow you to essentially pause a webpage as it renders when the browser hits a certain kind of request.  You can then change the attributes or key values passed on that request, allowing you to test out different scenarios without changing the code on your page. This one will change your life as an ad ops person!

In the instructions below, I’ll show you how to setup a breakpoint to stop the request to DFP so you can change various parameters of the request before the browser executes it.  You could use this to put your own browser into various targeting segments for which you wouldn’t otherwise qualify.

Step 1: Add / enable a breakpoint by navigating to Proxy > Breakpoint Settings.  You can add however many breakpoint triggers you want here, remove those you’ve setup in the past, or even import / export breakpoint lists if you want to be really fancy.  Maybe you have a breakpoint list for all header bidders, a breakpoint list for your ad server + DMP, and a breakpoint list for just your ad server for various use cases.  You can make these common across your entire Ad Ops team to reduce the technical know-how required by any single user, which is great for junior staff still learning how to use the tools.

Step 2: Once you click ‘Add’ in the above step, you’ll be prompted with the below window where you can configure the breakpoint settings by specifying when Charles should intercept the request and prompt you for action.  For example, there are usually lots of calls back and forth with the ad server, but you might only care about the initial GET request to the /gampad/ads endpoint because that’s the one that passes the decision parameters you actually want to edit.  Similarly, you might only care about the request and not the response.

Step 3: Test your setup by refreshing a page you know should trigger your breakpoint.  If the breakpoint is triggered, Charles Proxy will automatically open a new tab called ‘Breakpoints’ and list all the requests that qualified.  In this case, we just have a single request, as see in the left pane.  You’ll see the current details of the request in the Overview button, and you can edit anything you want by selecting the Edit Request button.  Simply double-click into the value cell and change what you like.

In this case I’m just highlighting the URL parameters, but you can see from the buttons on the very bottom of the image, Charles will let you change the headers and cookies as well.  It’s incredible really; changing the headers means you can change your user agent; changing cookies means you can impersonate another user (say another team member that can uniquely replicate a particular problem on your site).

Step 4: Click the Execute button once you’ve changed what you need about the request and the browser will proceed with the call.  You can validate the request was change per your instructions by going back to your Session tab, finding the request, and navigating to its relevant details in the Overview tab.

Regex Filtering

This one is all about sophisticated searching, and while it has a bit of a learning curve, it’s worth the effort if you find yourself pouring through a long list of requests in each session.  For example, if you find you frequently want to filter to a specific list of domains – say, five header bidder domains and your ad server – you need a ‘contains any’ search filter that a Regex filter can provide.

Here’s an example that filters on common header bidder’s endpoints (AOL / AppNexus / Index / PubMatic / Amazon / OpenX, respectively):

.(adtech|jpt|cygnus|gads|aax|acj).  

To activate regex searching, click the Settings button to the right of the Filter box and click the checkbox that says Filter uses regex shown below:

You can learn all about Regex on this guide, which is one of the simplest explanations I’ve seen, and by using the excellent RegExr tool to test your expressions.

Did I miss anything?  Tell me in the comments and check out my other posts on Charles Proxy.