Lately I've been disappointed with the lack of progress in the way we do web development. It's still a difficult, time consuming and error prone process, and it's only gotten more complex with time. Although we've probably finally moved passed the buggy IE6 era, we've also added a whole host of new challenges to the mix: HTML5, mobile, etc...
And yet the fundamental processes and technologies we use to create these applications are the same. It seems to me that web developers are entirely too focused on the lower-level details of web development which ought to have been solved years ago.
This post is part of a new series where I will be jotting down some ideas I have about how one might do web development differently. I'm not sure any of these ideas are new - some are rather mundane - but my hope is that perhaps in another 5 years web development might look very different than it looks today.
But before I continue, a brief discussion on:
At some level, hubris forms one of the basic primary motivators for all software development. We always assume that we can build it better, faster, simpler and cheaper than whatever else already exists. We imagine that we can see something that those who came before use failed to see. It's a can-do attitude (see Programming Mother***er or Move Fast and Break Things, etc...) and contrary to all the naysayers we set out to do the impossible.
But I'm also reminded of Chesterton's famous fence:
In the matter of reforming things, as distinct from deforming them, there is one plain and simple principle; a principle which will probably be called a paradox. There exists in such a case a certain institution or law; let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.
Pride is a funny beast. When you're caught up in the midst of it it can drive you to accomplish seemingly incredible feats. You feel on top of the world - like you could do anything you put your mind too. But it always, ultimately, results in failure. The projects we work on will never be as good as we had hoped: they're bug-ridden and complex. We rediscover all the things that made the prior solutions what they were.
It's fascinating to look at a project like Node.js and see that it is slowly but surely becoming a system just as convoluted as Java or Ruby on Rails (run npm install on any moderately complex site and tell me how that's much different from Bundler or Maven)
And sadly there's a deeper tragedy to our hubris - one we're rather terrified to admit to. Everything we work on is terribly transient. How much of the code that we write today will even be in use in 10 years? Will anyone remember that startup you were a part of?
What do people gain from all their labors at which they toil under the sun? [...] No one remembers the former generations, and even those yet to come will not be remembered by those who follow them. Ecclesiastes 1:3, 11
So with that aside, we'll press on:
Let's ditch the web server.
A pretty standard web stack is made up of the following components:
Designing an architecture which can handle failure reliably is difficult, but one simple tactic is to remove parts of the system which aren't necessary. For example if we don't need a database then that's one less component which can fail.
In this case we can get rid of the load balancer and the web server and replace them with a statically hosted web site in Amazon S3 and an API service.
S3 is reliable, fast, mature, extremely durable and has generally good uptime (probably better than I could hope to achieve). It's also cheap. Usage is straightforward:
s3cmd has a sync command which can help here.And that's it for the static component of our architecture. Ideally with a setup like this we want to move as much functionality as we can to the client. That may mean moving a lot of code to Javascript (or languages which can compile to Javascript like CoffeeScript or Dart) and perhaps using client-side frameworks like AngularJS. Also useful are command line tools which can help build this portion of the site. For example my blog is built from a "database" of local files which are compiled into HTML using Go templates.
Now unless our site is extremely simple there will also be pieces that need to be done on the server. Rather than use a full-fledged web framework like Rails we can build a much more rudimentary API. Simple APIs are easier to make, easier to understand and a whole lot easier to test than websites. And more than that a simple API gives you the flexibility to pursue other platforms besides the browser. (For example Android and the iPhone)
To demonstrate how this is done consider this tool: badgerodon.com/tools/rbsa. It takes a symbol, runs some analysis and renders a chart and table showing the results. The analysis is done by a server written in Go at this endpoint: http://162.243.222.36:9000/rbsa?symbol=vfinx
func init() {
handle("/rbsa", func(w http.ResponseWriter, r *http.Request) {
sym := r.FormValue("symbol")
sym = strings.ToUpper(sym)
var obj struct {
Data map[string]float64 `json:"data"`
Indices map[string]string `json:"indices"`
}
if sym != "" {
data, err := rbsa.Analyze(sym)
if err != nil {
http.Error(w, err.Error(), 500)
return
}
indices := make(map[string]string)
for k, _ := range data {
indices[k] = rbsa.DEFAULT_INDICES[k]
}
obj.Data = data
obj.Indices = indices
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(obj)
})
}
Crucially the domain names don't match so we need to setup CORS. (or use a separate subdomain under the same root domain and set them to the root on both sides)
func handle(pattern string, f http.HandlerFunc) {
http.HandleFunc(pattern, func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Access-Control-Allow-Origin", "*")
f(w, r)
})
}
For this case that's all we need to do. A more robust solution would also allow for load balancing, failover and retry. These are actually fairly straightforward to do purely in Javascript.