Google Analytics - Multi-armed Bandit Experiments

Multi-armed Bandit Experiments - no wonder my A-B stats look like they do...

Steven L. Scott, PhD, Sr. Economic Analyst, Google
January 23, 2013 | 9:40 AM

---

This article describes the statistical engine behind Google Analytics Content Experiments. Google Analytics uses a multi-armed bandit approach to managing online experiments. A multi-armed bandit is a type of experiment where:

• The goal is to find the best or most profitable action
• The randomization distribution can be updated as the experiment progresses

The name "multi-armed bandit" describes a hypothetical experiment where you face several slot machines ("one-armed bandits") with potentially different expected payouts. You want to find the slot machine with the best payout rate, but you also want to maximize your winnings. The fundamental tension is between "exploiting" arms that have performed well in the past and "exploring" new or seemingly inferior arms in case they might perform even better. There are highly developed mathematical models for managing the bandit problem, which we use in Google Analytics content experiments.

Read More 

Adding an additional layer of authentication to websites with HTTP Reverse Proxies

Adding an additional layer of authentication to websites with HTTP Reverse Proxies

Securism Blog
Walter Goulet, Jan 31, 2012

---

For my last client engagement, I was tasked with adding an additional layer of authentication to a publicly accessible website without actually modifying the website’s application code or web server configuration. After doing a little research, I came upon the idea of using a HTTP reverse proxy to intercept requests to the website and perform additional authentication before letting end users access the target web application.


If you are not familiar with the concept of a HTTP reverse proxy, think of it as a specialized web server that inspects incoming HTTP requests, forwards them to another (usually internal) web server after any local processing is completed, and makes sure that HTTP responses sent back to the client contain proper host names. The following diagram illustrates the typical processing flow when using a SSL enabled reverse proxy server that is acting as a front end to an internal web server.

Read More 

Finding the Next Great Opportunity

Head of Amazon Web Services on Finding the Next Great Opportunity

First Round Capital - The Review
April 17, 2013

---

With the frequency that you hear “cloud computing” mentioned at conferences and in the tech press, you might believe it’s just a tired buzzword that salesmen use to induce boring people to get excited about buying enterprise services.


James Hamilton wants to convince you otherwise. He approaches technical innovation from a skeptical perspective, with a flair for hands-on empiricism. As a Vice President and Distinguished Engineer on the Amazon Web Services (AWS) team, he’s been hard at work building the basic technology of cloud computing before the first impeccably coiffed consultant first uttered the phrase.

Read More 

Lessons from Pixar: Why Software Developers should be Story Tellers

Lessons from Pixar: Why Software Developers should be Story Tellers

First Round Capital - The Review
Martin Reddy, May 9, 2013

---

Film Concepts in Software

“ Talk to users in their own language. Use concepts and processes that your users are familiar with in order to communicate software designs as naturally as possible. ”


It's often said at Pixar that “story is king” and that this is the primary reason for Pixar's success. Appropriately then, all films at Pixar begin with the Story Department, where the initial script is developed and expressed on storyboards to refine the look and feel of the film.

In many ways, designing a great software product is like creating a great story. It’s not surprising that modern agile processes refer to “user stories” as a way to capture requirements from the perspective of the user. So very early on we took this concept to the next level and created a Story department in our software group.

Read More 

The one cost engineers and product managers don't consider

The one cost engineers and product managers don't consider

First Round Capital - The Review
Kris Gale, VP Engineering at Yammer, June 20, 2013

---

I talk a lot about costs. I believe good engineering is about finding the most cost-effective solution to a problem, whether that cost is measured in dollars, hours, morale or lost opportunities. Some costs are paid immediately and some are assumed as debt. Everyone in business knows this intuitively, but some costs are less obvious than others, so it's important to point them out to your team.

Among the most dangerously unconsidered costs is what I've been calling complexity cost. Complexity cost is the debt you accrue by complicating features or technology in order to solve problems. An application that does twenty things is more difficult to refactor than an application that does one thing, so changes to its code will take longer. Sometimes complexity is a necessary cost, but only organizations that fully internalize the concept can hope to prevent runaway spending in this area.

Read More