The Fall of Trada and the Facebook Marketplace that Never Was
Recently I’ve been listening to The Dropout - a podcast about Elizabeth Holmes and Theranos. There’s something about this story of a failed startup, the hubris, and the deception of dozens of powerful people which really captures the imagination. Far more than a story of a successful startup would. I wonder why that is?
Perhaps, to borrow a phrase, all successful startups are alike. All dysfunctional startups are dysfunctional in their own way.
As I was pondering this, it got me to reflect on my own history with startups, and one in particular: Trada.
Trada
Launched in 2010 by CEO Niel Robertson, the company was a search advertising marketplace targeting small to medium sized businesses using a pool of crowdsourced optimizers.
Crowdsourced… that’s not a phrase you hear much these days. But it was the blockchain of its day. Crowdsourcing perhaps ultimately finds its source in Surowiecki’s The Wisdom of Crowds and this anecdote about Francis Galton:
Galton was at a country fair where a live ox was placed on display. Fairgoers were invited to guess the weight of the ox after it had been slaughtered and dressed. Eight hundred ordinary people from all walks of life tried their luck. They included experts such as butchers and farmers, as well as non-experts. Out of interest, when the contest was over, Galton collected the used tickets and averaged the punters’ individual guesses. This figure represented the “wisdom of the crowd”, and in this case the crowd had guessed the ox would weigh 1197 pounds. After it had been slaughtered and dressed the ox weighed 1198 pounds. The crowd’s judgement was essentially perfect.
A similar experiment can be performed with a jar of gumballs. Ask everyone to count them, average the guesses, and it will probably be better than any single guess. Indeed this idea was so much a part of Trada’s story, that it was even used as part of the logo.
When I joined Trada in 2011 it was a rapidly growing startup that had just received major funding from Google Ventures and Foundry Group. Situated in downtown Boulder, our office was the former Daily Camera building:
A strange mixture of traditional cubicle farm, empty offices with a seemingly endless supply of abandoned furniture, a giant open common room with deafening chipotle-like ceilings, a freight elevator to a basement straight out of a horror film, and, this being Boulder, a half dozen dogs on any given day, Trada had a unique office.
I'm not sure which is the more apt: the dramatic foreshadowing of a startup inhabiting the dessicated remains of a dying industry only to itself die a few years later, or the exciting atmosphere of a growing startup in a playful open office only to step in the occasional pile of dog shit on your way out.
Although office life was fun it was also at times surreal. Consider the Palio di Trada:
Equally at home in an episode of Silicon Valley, this razor scooter race complete with hay bales was a reference to the Palio di Siene - a horse race involving Italian contrade (city wards) - with Palio being the former name of the company.
Beginnings
I had been actively looking for a new job for a year or more when my former manager reached out. An afternoon of interviews later I was handed an offer on the spot by the CSO.
Given an immediate 20% raise, and a chance to escape my duties as a pure frontend developer in a Microsoft shop, taking that offer seemed like a no-brainer. And so I resigned and started two weeks later.
And like starting almost any job, those first few days you have that dreadful feeling of buyer's remorse. For me the moment was seeing the codebase for the first time. It was shockingly low quality. The frontend was hobbled together by backend engineers who didn't really know Javascript (or for that matter want to know Javascript), and the UI that was in dire need of a designer. (which they had just hired) The backend was Ruby on Rails, the popular framework-of-the-day in 2011, and it's Active Record ORM which was both a stunning feat of engineering and uniquely unintelligible when you already knew SQL.
My first few months at the company were devoted to optimizing SQL queries to make pages on the site faster. I didn't really know what I was doing, having never written SQL or Ruby professionally, but one thing I did know: any time you take on an optimization project, it's really important to make sure you have some way of measuring performance. After deep-diving into our Google Analytics one day, I discovered that basically no advertisers used the site and I recommended that we spend our time on the optimizer side instead. Had I been more observant I would've realized this was a major warning sign that the company was headed for disaster.
Now to be clear about the state of this codebase. I found it shocking, but I know now that it's entirely typical. Almost any successful startup will go through a rebuilding phase, where every legacy system is replaced by one that fits the needs of a now much-larger user base. The twitteresque Replace Rails with Scala phase of the startup if you will.
Thankfully a couple months after I started, myself and two of the founding engineers were given a new project: Facebook Advertising. Up till this point Trada had only ever supported search networks and Facebook showed real promise as a new marketplace.
To give you a sense of what this entailed, let's consider optimization.
Optimization in Google is about choosing the best keywords and copy for your ads. Keywords are auctioned on-demand based on the search queries users enter. A good keyword might dramatically increase your conversion rate, but usually good keywords are highly competitive.
For example, the most expensive keywords are for things like insurance, loans and attorneys, with the staggering cost of more than 50$ a click. (And if you ever wondered how Google makes so much money… well here’s your answer)
Trada’s advertising customers had small budgets. Maybe a few thousand dollars a month. So bidding on a keyword like that would eat up their whole budget in only a few clicks. Optimizers were tasked with being creative about their keyword choices - finding the keywords which would still lead to a conversion, but which were dramatically less expensive than the most competitive keywords.
Facebook advertising was significantly more sophisticated. In Google you pick keywords and topics and you can target by language or location, but in Facebook you can target by almost anything you can imagine. For example you might be selling novelty baby clothes, so you target women who recently had a child, are in the top 10% financially of US zip codes, and who "like", say, Babar the Elephant.
Incidentally after decades of endless soothsaying by corporations attempting to divine precisely what it was consumers liked, who could've imagined we would have simply handed over this vital information to Facebook for nothing.
Anyway, now suppose Babar is in high demand. It's suddenly the new Paw Patrol and prices have skyrocketed, and our optimizer, in a flash of creative brilliance, decides to target Republicans instead of people who like Babar. For whatever reason this oblique connection holds a surprising appeal to the denizens of Facebook. Clicks abound for pennies on the dollar.
And it was precisely this sort of creativity which was bedrock to the principles of Trada, the kind of creativity an algorithm was unlikely to ever come up with, but given enough optimizers you just might see it happen. In theory crowdsourcing an ad campaign could beat even the best advertising agency.
To borrow another phrase, given enough optimizers, all ad campaigns are affordable.
The Project
So our task was to build a marketplace on top of Facebook's advertising API. Advertisers would setup campaigns with a budget and optimizers would create ads. Optimizers, being paid on the difference between the goal and the actual cost, were incentivized to reduce cost, but also to drive clicks, since good optimizers would receive more of the budget. Over time poor optimizers would be removed from the campaign and eventually this unplanned, ad-hoc, emergent pool of optimizers would deliver more value to the advertisers then they could ever hope to do so on their own.
We took the existing database and slightly modified it for our needs, but largely kept the schema the same. (Which itself was almost entirely defined in Ruby on Rails) An entirely new UI was designed and I built it using a lot of custom Javascript and jQuery. In addition we leveraged Resque - the Redis job queuing system from Github - to handle all of our automated jobs (payouts, Facebook API updates, etc).
And although we worked closely with a sales associate and a couple account managers, the project proceeded with surprisingly little oversight.
I suppose this was largely due to the fact that the CSO (the guy who hired me) left the company shortly after we started the project. Never did find out what happened to him.
And come to think of it, shortly after that the senior engineer on the project disappeared too. One day he just wasn't at his desk anymore.
So there we were, expected to finish this project with just two engineers, and, since the other developer on the project decided to focus on devops, the whole crushing weight of the product suddenly found itself on my shoulders.
Now commonsense would dictate that a project in this position - a project with minimal management, a lack of direction and the loss of its principal engineer - would certainly fail. But commonsense was wrong.
Given the right skillset, unfettered access to make changes with little to no oversight, complete knowledge of a from-scratch brand-new system, and the passion to work on the project - a single developer can be incredibly productive. And not just some mythical ten-xer developer, but even the lowly journeyman developer.
What would take the entire rest of the team months to complete could be finished in a day or two. For example when optimizers complained about the payout algorithm we used, we rolled out a new one similar to professional golf in an afternoon. The new site was faster, looked a whole lot better and was much, much easier to maintain.
And I’ll tell you, as the developer in that situation, it's invigorating. The stars align and there’s an exuberant vitality. Like suddenly you’ve discovered what you were always meant to do, all is right with the universe and you can accomplish anything you put your mind too. You might spend your entire career chasing after that feeling. (I wonder if this isn’t an echo of the imageo dei - a small taste of what the ancient of days felt like programming a meticulously crafted universe)
Of course this feeling never lasts. God’s program may be perfect and bug-free, but “to err is human”, and the code you produce in this furious frenzy hardly ever withstands the light of day. Inevitably your beautiful creation collapses under the weight of its own contradictions.
Imagine this scenario: You're tasked with fixing a bug, open a file and run across something truly atrocious and you’ll cry out in anger: “the developer who wrote this deserves to die”
And git-blame, like Nathan before him, will say to you: “You are the Developer!”
It's happened to me more than a few times...
Remember the Balmer curve:
computer programmers obtain quasi-magical, superhuman coding ability when they have a blood alcohol concentration percentage between 0.129% and 0.138%
I wonder if the Balmer Curve is not just a special case of the more general principle; that all programming is a form of intoxication. Ideas which seemed so brilliant at the time fail to sustain the scrutiny of cold hard rationality later - if you even remember them at all.
The Fall of the Facebook Marketplace
And so we somehow managed to launch the product. We basically finished on time and had customers using the product, albeit not entirely successfully. Optimizers were quite good at churning through an advertisers budget, and even at getting clicks, but not so much at driving sales. (pro tip: bikini-clad women have a remarkably high click-through rate)
But we never really learned whether or not the product would ultimately succeed because it was unceremoniously canceled a few weeks after we launched it and I was moved back to the general engineering population.
Now take this for what it is, the musings of a lowly engineer who didn't have a particularly good understanding of what was going on at the time, but Trada wasn't doing well.
I guess the problems we were seeing with Facebook were happening with Google too. Maybe optimizers had figured out how to game the system sufficiently and they could make a lot of money without really delivering value to the advertisers, or maybe they were simply bad at their job and the system couldn't react quickly enough to their incompetence, but either way advertisers were not happy and they were jumping ship.
Retention Island
So what was Trada's response?
Behold, Retention Island:
Complete with torches, a real wood deck, Tiki Bar and real allergy-inducing sand a few inches thick, customer service was moved to this common room in a desperate attempt to stop the bleeding and retain as many advertisers as possible.
I suppose the beach theme was intended to remind employees that there was light at the end of the tunnel?
Trada's Demise
After the Facebook product died it wasn't long before I quit, chasing after a consulting gig, but where did Trada end up?
Well in 2013 the company "restructured" laying off most of their employees, in 2014 Matt Harada took over as CEO and in 2015 the company was shut down by its creditors.
So what to make of this sad story? Was Trada's collapse inevitable? Like Theranos were there giant warning signs all over the place and only gross negligence would miss them?
I don't think so.
For one, I wonder if the idea undergirding Trada couldn't have worked out in the end. Maybe with a few different decisions along the way things could've turned out differently.
And for another, the surreal environment of a startup like this sure seems strange, but successful startups do crazy things too. And sometimes it's those crazy things which can knock the startup out of its tailspin.
I mean, just look across the street at Gnip - another Boulder startup, which nearly collapsed, laid off half its team and eventually managed to get acquired by Twitter a few years later.
No I think Trada probably could've made it in the end. But what do I know.