2014-04-21 22:10 How to Build Things With Go: Tries

I'm working on a new series of video tutorials for how to build things with Go. The first one is on Tries:

Source Code: github.com/calebdoxsey/tutorials

2014-02-10 21:28 Rethinking Web Development: Canvas UI

So here’s a sadly common story:

A startup with an existing web application is seeking to get into the mobile space by producing both an Android and iOS app. Seeing as they both already have the necessary skills to produce a web application and they’d rather not have to learn, build and maintain another two versions of their application, they attempt to create a mobile application using web technologies via PhoneGap.

Almost without fail that first version of their application is a disaster. Slow, clunky, unstable and riddled with bugs, they’ll spend the next 6 months trying to get it to work in some sensible way. Eventually they’ll admit defeat, scrap the old app, and build a new, native one from scratch.

And that’s pretty depressing. One of the supposed advantages of the web was that we could finally do away with so many operating system idiosyncrasies. No longer would we have to make 3 versions of every piece of software (windows, mac, unix) and instead we could make one application which could run everywhere. And then after perhaps half a decade, we threw all that out and recreated the multiplicity of platforms we thought we’d gotten rid of.

Actually it’s worse than that. A mere 3 versions of the application is a pipe dream. In reality you also have to consider all the various devices, in all their various resolutions and capabilities, and all the various versions of the operating systems. This list of Android devices is the stuff developer nightmares are made out of. Serious Android shops really do just have 100s of mobile phones to test on. (because it’s the only way to be sure it actually works for your users)

I can’t believe I’m about to say this, but it makes the almost 10 years we spent working around IE6 bugs seem easy by comparison.

Looking back at the one Android app I built I did come to appreciate one thing about it though. I think perhaps our original inclination about building one app for multiple platforms wasn’t entirely hopeless - it was just in the wrong direction.

Why do we think that HTML, CSS and Javascript represent the ideal set of technologies to build user interfaces? When you actually consider their origins and the original purpose for which they were designed, they are a remarkably poor fit for what we ended up doing with them. HTML is a language to represent hypertext - that is to say a language that gives some structure to basically textual content, and to provide links between different sets of textual content. It’s designed for an encyclopedia. But we aren’t building encyclopedias, at least not most of us, and how much of a modern web application even resembles textual content?

Now I’m not sure I can actually make this argument - it’s the kind of thing that’s really hard to recognize until you see an alternative - but I’ll give it a shot anyway. Here are a few of the problems I see:

  • Fatally flawed by it’s original ties to the hideously complex SGML, it’s far too forgiving nature for malformed content and its dearth of elements to represent concepts we actually work with, HTML is a hack we put up with because we have to. As a technology it failed to deliver on its ultimate goal: the Semantic Web. (I think we can thank Google for managing to give structure to a system which was supposed to have it, but never actually did)
  • When it comes to web applications the separation of the semantic meaning of content (HTML) from it’s presentation (CSS) and interactive function (Javascript) is not actually all that useful. Most of what makes a user interface has no real meaning outside of it’s intended function inside that interface. Structuring buttons as buttons, lists as lists, tables as tables and text as text may make the life of a data-crawling robot easier, but the end-user doesn’t ultimately care about such things.

    Proof of this can be seen in this basic point: nobody stores their structured, semantic data as HTML. If I query Twitter for a Tweet I get back a JSON object that describes everything from it’s title to it’s author and date of publication. I take this object, transform it into HTML and CSS, and include it in my page. If HTML delivered on what it promised APIs like Twitters would return content as HTML which I could just drop into my page. But it doesn’t deliver and HTML is a cumbersome and inadequate tool to represent structured data.

    And yet web developers, suckled as they were at the teat of Designing with Web Standards, are obsessed with the pointless pursuit of semantic value.

  • WYSIWYG editors, though popular for a time (with products like Dreamweaver), ultimately fell by the wayside. Probably most web designers today know HTML & CSS, and perhaps even work directly in them. Maybe that’s a good thing, but something about it just seems wrong. If a designer is comfortable with Photoshop why shouldn’t he be able to export what he makes in a usable format?

    I spent many years translating photoshop mocks into HTML & CSS. I don’t it anymore, but that job still exists. Why? Shouldn’t a computer be able to do this?

  • CSS is not easy. It’s deceptive, because some things are trivial. But the layout model, coupled with it’s cascading nature, means that full-fledged UIs can be surprisingly difficult to implement. Everyday examples include centering content, implementing float & overflow properly, trying to use the incredibly complex z-index algorithm, and - of course - working around subtle browser inconsistencies.

    Now if you don’t agree with those examples, consider the fact that it took years for browsers to finally mostly pass the Acid3 test. If the people who write browsers can’t implement these things properly, how much hope do you have of using them properly?

  • CSS is, in general, not very reusable. Have you ever tried to take a component you wrote in one site and put it in another? Cascading rules lead to complex, surprising and unpredictable interactions among components, and there doesn’t seem to be any general approach to writing CSS that solves these problems. (though certainly many have tried) The best solution I’ve seen is a style guide, but the truth is most web sites are a hodge-podge of 10,000ish line CSS files that no one can really grasp.

  • Newish web development tools are remarkably conservative. LESS, SASS and all their cousins are languages designed to produce CSS. But they’re still basically CSS in how they work. No one is re-inventing the layout model and about the riskiest thing you see are automatic polyfills or image substitutions with data-uris. The same can be said for HTML generation languages (with most looking like Markdown and friends - glorified variable substitution languages) and even most Javascript competitors like Coffeescript.

    It seems to me that a healthier ecosystem would see more radical offerings along the lines of the Google Web Toolkit. Not that there haven’t been attempts to pull this off, rather the attempts are generally of very low quality. Maybe a better base set of technologies would make things like this easier to pull off.

I could probably keep going, but at this point I want to introduce a radically different way of writing web applications which I called Canvas UI.

Canvas UI

The idea is this: rather than using HTML & CSS as the bedrock of frontend web development we use a more rudimentary graphical API (Canvas, WebGL or SVG), build frameworks on top of that base and develop our actual application using an entirely different set of technologies.

Over the last couple weeks I threw together an example which implements the same functionality as my previous WebRTC example. Source code is available here. Though crude, buggy and clearly not a viable solution, I think there’s enough here to show what it could look like.

The application is drawn on a large, full-page canvas, with the only HTML & CSS being used to create that canvas. All drawing is done via Javascript in an OO class-based component model like this:

var incomingTitle = new Text({
	text: "Incoming",
	fontFamily: FONT,
	fontSize: "20px",
	color: "#333",
	left: function() {
		return this.parent().left() + (this.parent().width() / 2 - this.width() / 2);
	},
	top: function() {
		return this.parent().top() + (this.parent().height() / 2 - this.height() / 2) - 40;
	}
});

Not for the faint of heart, this is a no-frills approach to web development. To list just a few of things you’d need to pull this off:

  • Canvas has no DOM. Objects are painted to the screen and have no other existence outside of what you give them. Therefore if you want to implement events you will have to do them entirely manually. (That is to say, hook them to the body and calculate the position of your GUI elements to determine whether or not a click fell on them)
  • Some graphical elements are quite complex in their implementation: for example a large block of text. You won’t get scrollbars, multiline layout, floating images (or other content) or the kind of CSS positioning trickery you may be accustomed too: line-height, text-overflow, indentation, padding, margins, … (unless of course you choose to implement them)
  • Forms will have to be completely rethought. None of those controls will exist anymore.
  • Many of the everyday tools web developers use won’t be all the useful anymore. Inspecting a page won’t tell you much, and you can’t go adjusting CSS rules on the fly.
  • For that matter accessibility is pretty much shot too… (though maybe you could work around this by dumping your textual content to the page in a format screen readers could figure out)

So you might wonder why anyone in their right mind would suggest such a thing. It’s because it offers maximum flexibility. Sure throwing out everything we have given to us with a modern browser is hard, but think of the other side: no longer beholden to whims of the w3c and browser makers you can develop entirely different approaches to rendering GUIs. To give a few examples:

  • You could make an engine which could translate Android or iOS applications into web applications (and thus reverse what everyone is currently trying to do). My example implements drawable shapes similar to how Android does it:

    new Shape({
      solid: {
        color: "#F5F5F5"
      },
      stroke: {
        color: "#DDDDDD"
      },
      corners: {
        bottomLeftRadius: 10,
        bottomRightRadius: 10
      }
    })

    That wasn't too hard to put together, but the sky's the limit here on how it could work.

  • You can change browsers in ways that wouldn’t be possible otherwise. For example you could implement the LaTeX text rendering algorithms which would give you superior text-wrapping and hyphenation, justification and kerning.

  • A lot of what we do now could be generated: you can make your own layout languages, generate javascript plumbing code for ajax and events and abstract away having to think about resources. (We tend to think this is impossible but Android manages to do it just fine)

  • This freedom opens up rendering options we never really had. For example imagine a comic speech bubble:

    In traditional web development you would implement this use a whole bunch of images thrown together or really crazy CSS. (like this) In Canvas it’s a formula:

    ctx.moveTo(l, t+a);
    ctx.arcTo(l, t, l+a, t, a);
    ctx.lineTo(r-a, t);
    ctx.arcTo(r, t, r, t+a, a);
    ctx.lineTo(r, b-a);
    ctx.arcTo(r, b, r-a, b, a);
    ctx.lineTo(l+40, b);
    ctx.lineTo(l+15, b+15);
    ctx.lineTo(l+20, b);
    ctx.lineTo(l+a, b);
    ctx.arcTo(l, b, l, b-a, a);
    ctx.lineTo(l, t + a);

    That may look complicated, but it’s a lot more flexible. There are only a few low-level, primitive operations you need to learn.

  • Graphics like this are re-usable. That speech bubble is easy to restyle (changing its fill style) and transform (adjusting its dimensions) and I can copy and paste it into another project. With an SVG backend you could potentially export things directly from an editor.

And I suspect given a full-blown framework with lots of features we’d see ways of doing things we couldn’t imagine before.

So where do we go from here? Well the truth is I don’t really have much drive to finish this project. I already know all the standard web technologies, and UIs are mostly a means to an end for me. I’d rather focus on real coding. But maybe some day we’ll see some radically different approaches to building UIs for the web.

This is the final post in this series. The previous 4 were: WebRTC, Non-RESTful APIs, Cloud IDEs and Static Hosting.

2014-01-30 21:05 Rethinking Web Development: WebRTC

Fundamentally web applications are client-server applications. Web developers write code that runs on a server and end users (clients) connect to that server (via a browser using HTTP) to perform tasks. In recent years this rather standard definition of the web application has come under fire. Increasingly code is no longer run on a server (but rather via javascript on the client) and HTTP is no longer, necessarily, the protocol used to communicate between the two machines (SPDY being the rather obvious alternative, but also things like WebSockets which don't really fit in the HTTP bucket).

However, even more dramatic than those two shifts has been the relatively recent introduction of WebRTC. If you're not familiar with WebRTC (the RTC meaning Real Time Communication), it's a technology that allows for peer-to-peer communication. That is to say, end users can communicate with one another directly, without the need for an intermediate server.

It seems to me, at least at this moment, that this is a technology that is generally not well understood and it's potential has not been fully realized. WebRTC ought to be a seismic shift in the way we build web applications. It's not yet, but I suspect it will be in a few years.

In this post, the penultimate post in the series, I will give a brief overview of how to use WebRTC and then discuss some of the possible implications for web development.

How to Use WebRTC

Getting started with WebRTC is not easy. Most of the documentation is very confusing (good luck understanding the spec), there aren't a ton of examples out there yet, and, up to this point at least, the technology has been in a constant state of flux. Nevertheless WebRTC is not merely experimental, it's a fully functional technology available in both Firefox and Chrome today.

To underlie that point, if you've not seen it, you should see the AppRTC project. This is a video chat app, similar to Skype, implemented almost entirely in the browser (with a small bit of server code) and using peer-to-peer transfer of data. For a mere demonstration, it's surprisingly useful and calls into question all of the applications out there that attempt to implement this functionality using custom Java, Flash or similar installed applications.

But back to the question at hand: how does one use WebRTC?

For the purposes of this tutorial I built a small chat application which consists of two components: a server-side Go application which facilitates the initial signaling process between the two peers and a client-side Javascript application which implements the actual WebRTC workflow. First let's take a look at a high level description of the process.

For this example, suppose we have two end users who want to talk to each other: Joe and Anna.

Joe arrives first, connects to a server, subscribes to a topic and waits for someone else to show up.

Anna arrives next, connects to the same server, subscribes to the same topic and at this point the server tells Joe that someone else connected.

Joe sends an "offer" to Anna (via the server) indicating that he wants to establish a peer to peer connection.

Anna receives the "offer" and sends an "answer" to Joe.

Both Joe and Anna trade ICE candidates. ICE stands for interactive connectivety establishment (described here) and basically represents the various ways the two parties can reach each other.

Finally the connection is made, one of the ICE candidates is agreed upon and Joe & Anna can communicate directly with another.

Code

The (mostly) complete source code for this example can be found in this gist. The server is implemented in Go, the client in Javascript and communication between the server and the client occurs over a WebSocket. WebSockets are actually fairly straightforward to implement using Go and they make it so I don't have to worry too much about storing things on the server. (As a long-polling comet server would require)

When a user first connects, the application will ask them to enter a topic. This topic is how the two peers are connected on the server (they have to enter the same thing). You can think of it like an agreed-upon virtual meeting location. That topic is sent to the WebSocket like so:

function startWebSocket() {
  ws = new WebSocket("ws://api.badgerodon.com:9000/channel");
  ws.onopen = function() {
    ws.send(JSON.stringify({
      topic: topic,
      type: "SUBSCRIBE"
    }));
  };
}

The server receives this connection, sends back the user's ID and subscribes them to the topic: (IDs are generated using a full cycle PRNG)

id := <-idGenerator
out <- Message{
  Type: "ID",
  To: id,
}
for {
  var msg Message
  websocket.JSON.Receive(ws, &msg)
  if msg.Type == "SUBSCRIBE" {
    msg.Data = out
  }
  in <- msg
}
// in subscribe
topics[topicId][userId] = out
for id, c := range topic {
  if id != userId {
    send(c, Message{
      Topic: topicId,
      From: userId,
      To: id,
      Type: "SUBSCRIBED",
    })
  }
}

And now the first user waits for someone else to show up. When that happens the same process is repeated, except this time the first user is informed that the second user subscribed, so he proceeds to begin the process of establishing a peer-to-peer connection and sends the second user an offer (via the server):

case "SUBSCRIBED":
  to = msg.from;
  startPeerConnection();
  sendOffer();
  break;
pc.createOffer(function(description) {
  pc.setLocalDescription(description);
    ws.send(JSON.stringify({
    topic: topic,
    type: "OFFER",
    to: to,
    data: description
  }));
}, ...);

The second user sees this offer and sends an answer in reply:

case "OFFER":
  to = msg.from;
  startPeerConnection();
  setRemoteDescription(msg.data);
  sendAnswer();
pc.createAnswer(function(description) {
  pc.setLocalDescription(description);
  ws.send(JSON.stringify({
    topic: topic,
    type: "ANSWER",
    to: to,
    data: description
  }));
});

In addition to the offer and the answer both users have also been forwarding along ICE candidates:

pc.onicecandidate = function(evt) {
  ws.send(JSON.stringify({
    topic: topic,
    type: "CANDIDATE",
    to: to,
    data: evt.candidate
  }));
};

Finally once the first user receives an answer, and an ICE candidate is agreed upon the two peers are connected. RTC has the ability to stream audio and video, but for this example I used an RTCDataChannel:

pc = new RTCPeerConnection({
  iceServers: [{
    // stun allows NAT traversal
    url: "stun:stun.l.google.com:19302"
  }]
}, {
  // we are going to communicate over a data channel
  optional: [{
    RtpDataChannels: true
  }]
});
dc = pc.createDataChannel("RTCDataChannel", {
  reliable: true
});

Actually sending messages is trivial:

function onSubmit(evt) {
  evt.preventDefault();
  var text = chatInput.value;
  chatInput.value = "";
  var msg = {
    type: "MESSAGE",
    data: text,
    from: from,
    to: to
  };
  onMessage(msg);
  dc.send(JSON.stringify(msg));
}

Implications

There have been two large, and completely divergent trends when it comes to networked applications: the cloud and decentralized, distributed networks. On the one hand companies like Google, Apple and Facebook have been moving all of their user's information and applications to their data centers. Your email, pictures, games, etc... are no longer on your computer, but rather they are accessed over the internet, and stored in the cloud. This comes with a whole host of advantages, but it also comes with a steep cost: Google reads all your emails, Facebook knows everything about your personal life, Apple can track all your movements and although these companies certainly provide a great deal of value with their products, in the end their goal is not to make you a happy customer (because most of the products are free), but to use the information you freely provide them for their own ends. (and mostly just to serve you ads)

At the same this has been happening, we've also the seen the rise of decentralized, distributed networks. From file-sharing applications (like BitTorrent) and streaming content providers (like Spotify) to instant messaging applications (like Skype) and even digital currencies (like BitCoin). Decentralized networks have no single, governing authority. Data is spread across the network and communication is peer-to-peer.

And though the architecture of the web has always, fundamentally, been client-server, in some ways peer-to-peer architecture represents a more accurate representation of one of the original goals of the web:

The project started with the philosophy that much academic information should be freely available to anyone. It aims to allow information sharing within internationally dispersed teams, and the dissemination of information by support groups.

Now perhaps at one time, the Googles of this world had intended to organize the world's information. But increasingly, the fundamental goal of companies like Google is not merely to organize the world's information, but to own it. You can see this when they killed Google Reader. Google doesn't want you to read blog posts on other servers, they want everyone to use Google Plus as their blog. With peer-to-peer networks it may be possible to undermine this trend. People can own their own content again.

This example demonstrates one of the most obvious use cases for this technology: video, audio or textual chat among peers. But there are other possibilites:

  • The chat application has a far broader usage than most people realize. Of course there are the typical Google & Facebook-chat like applications, but there are also feedback libraries (like Olark), support applications (like you might see on Comcast's website) and a whole host of multiplayer games.

    Furthermore, WebRTC is secure by-default. In this day-and-age of NSA skepticism where Google reads all your email, and Facebook knows your entire life history, WebRTC is a breath of fresh air. You can finally have your privacy back and still get the robust accessibility and flexibilty of a modern web application.

  • Bit Torrent has demonstrated the power of a distributed file sharing network. With HTML5 technologies it is possible to build such a network directly in the browser. For example: ShareFest. Could someone implement an in-browser DropBox? (perhaps, similar to SpaceMonkey?) Or Mega? One of the downsides of WebRTC is that users must be connected at all times: once their browser closes they are unreachable: but that issue doesn't seem insurmountable... and maybe with a few helper nodes such a system could be sustained.

  • A distributed social network (ala Diaspora) may have a better chance of succeeding if it's just as easy to use as Facebook or Twitter. Storing all that data is challening (particularly with large media like pictures and video), but the relationships and textual updates are much more realistic storage-wise. Store that data among your (actual) peers and perhaps it could even be fairly reliable.

  • With the introduction of things like WebGL, WebAudio, PNACL, asm.js and a myriad of other HTML5 technologies, it's possible to build real games that exist solely in the browser. WebRTC makes it so those games can be multiplayer. This isn't a new idea - consider Artillery - but, as far as I know, it's not something which has really been realized yet.

  • One of the reasons internet companies are so successful is that it's very difficult to build something like Facebook. It requires a tremendous amount of capital and large, robust, highly reliable and fast systems can't be put together by just anyone. How many projects have been sunk by their inability to scale?

    And yet peer-to-peer networks have the potential to scale in a way the cloud never could. A signaling server can handle an enormous amount of load with no issues. If the vast majority of your application can be re-written to run purely on the client, most of your server-side concerns evaporate.

    Of course, most web applications can't be moved entirely to the front-end. Nevertheless this isn't an all or nothing game. The more work you can push to your end users, the less work you have to do on your own machines. Is it possible to do some of that background processing in a browser? It might be more possible than you imagine: modern browsers have threads, type arrays, full blown databases, offline capabilities, file-system access, etc...

And those are just a few of the things that came to mind in the last week. It's an exciting time to be a web developer and it'll be interesting to see just what turns up in the next few years.

If you managed to make it this far: thanks for sticking with it. Stay tuned for my final post in this series, where I will propose an even more radical change to how we build web applications.

2014-01-27 08:30 Rethinking Web Development: Non-RESTful APIs

Software development is a strange industry. Applications are hard to build: they take months of work, have lots of moving parts and they're extremely risky - often being built under tight deadlines in competitive markets. And they're usually built by surprisingly small teams of developers; developers who probably learned a very large chunk of the knowledge needed to build the application as they built it. It's kind of amazing that anything we build works at all.

And so I'm constantly surprised that we obsess over things which don't really matter. Developers will have vigorous arguments over everything from arbitrary stylistic choices (tabs vs spaces, where to braces, ...) to tool choice (like which editor to use) and naming conventions. Sometimes it seems like the amount of energy we expend discussing these things is precisely correlated with its level of arbitrariness.

Actually it's worse than that. Have you ever met an Architecture Astronaut? They take your crude, simple but working project and transform it into a more "correct" version. It's not actually functionally any different (one hopes), but it now has umpteen levels of indirection, factories, interfaces, powerful abstractions, design patterns and a myriad of other features you never really knew you needed. It also has the quality of being basically unmaintainable because you don't actually understand it anymore.

A remarkable demonstration of this can be seen in the Fizz Buzz Enterprise Edition. It's funny because it's not all that different from reality.

Actually this tendency to transform arbitrary decisions into moral categories and then hang them as a millstone around the neck of others is a much broader phenomena. At its root is perhaps the need for markers to indicate who is in the group and who is out of it, and then once we have those markers established the need to assert our mastery (as a game of one-upmanship). This tendency is only exacerbated by the challenges we're presented with: we aren't actually confident that we know what we're doing and the systems we build are often incredibly difficult to get working. Rather than address the hard problems, we isolate the easy things and focus on them. (Maybe I can't build a distributed database, but I can tell you that your function should start with a capital letter)

So I thought I might tackle one of these software shibboleths: the Restful API.

REST

REST is a complicated architectural style described here. I'm not particulary interested in tackling the actual academic meaning of the term, rather what it has become as a popular buzzword implementation. I have in mind the focus on proper HTTP verbiage (GET, POST, PUT, DELETE), resource URIs, and thinking almost entirely in terms of the representation and transfer of resources.

For example suppose we are building a RESTful API for email. We might have a URL structure like this:

{GET|POST|DELETE} /emails/{EMAIL_ID}
{GET|PUT} /emails

GET /emails lists the most recent emails, PUT /emails creates a new one, GET /emails/{EMAIL_ID} gets a particular email, POST /emails/{EMAIL_ID} updates an email, DELETE /emails/{EMAIL_ID} deletes an email.

So here's why I don't like this approach:

State

REST focuses on state, but state is not the primary building block of an application. What does it mean to "create" an email? Does that mean it sends it? How can you "update" or "delete" an email? Suppose you have 1,000,000 emails... listing them all doesn't really work anymore does it? Consider this approach to sending emails:

PUT /emails/{EMAIL_ID}/send

A URL like this doesn't make sense. It would be read "Create a 'send' record for email {EMAIL_ID}". What exactly would you send to this endpoint? An empty JSON object? ({}) It's an uneasy fit.

Suppose instead you add a "state" field to your email:

{"id":1234, "subject": "whatever", ... , "state": "unsent"}

And you would update that record and POST the update. This is a better approach from a URL perspective, but it's much messier from an implementation perspective. On the server I have to detect changes to this object and act accordingly (ie State was changed, therefore I will send the email). Do I merely update the record in my database, schedule the email to be sent later, and respond that everything is fine? Or do I wait for the action to be completed in its entirety? Maybe I add additional types of state:

{ ... "state": "sending" }, { ... "state": "sent" }, { ... "state": "bounced" }, ...

But "state" in this sense is not really a property of an email itself, rather it's more a property of our system: I'm currently sending your email, I sent your email, I tried to send your email but it was blocked, ...

Problems like this aren't unusual - they're typical. Modern web applications aren't glorified Wikipedias, they're the desktop applications of yesteryear: described in terms of user workflow and actions not in terms of the mere transfer of resources.

Caching is Broken

One of the supposed advantages of a RESTful architecture is that it lends itself to caching. Unfortunately those caching mechanisms are notoriously difficult to use properly.

Consider a web application with a lot of Javascript (which is basically all of them). Somewhere in the HTML for that site the Javascript has to be included:

<script src="../assets/js/site.js"></script>

That's web dev 101, and it's wrong. For 2 reasons:

  1. Most web applications change frequently. When you change site.js there's no guarantee that your end user will get the latest version the next they visit your site unless you explicitely make it so your web server adds headers to invalidate the cache.
  2. If you add headers to invalidate the cache everytime a user comes to your site that means they're downloading 100s of KB of script everytime they reload (which can be devestating to performance)
The solution is to use a hash as part of the name of the script and add aggressive caching headers:

<script src="../assets/js/site-d131dd02c5e6eec4.js"></script>

Let's just call a spade a spade here: that's a hack. The modern web developer spends and inordinate amount of time optimizing the performance of their applications to work around issues like this. This is because the architecture to which they're beholden is fundamentally flawed. A well designed system makes the typical case easy, not hard.

For more guidance on caching read this article by Google. Speaking of Google, they got so fed up with slowness of HTTP, they silently replaced it with SPDY on their servers.

Clean URLs

Perhaps you've read this blog post: URLs are for People. Well, I disagree. URLs are not for people. Nobody enters them manually, and rarely do they even bother to look at them. Your domain matters, but outside of that if you're spending more than a few minutes thinking about how you want to layout your URLs, you are wasting your time focusing on a part of your system that doesn't actually matter. (And one thing I love about that article is his two primary examples of bad URLs are two of the most popular sites on the internet: Google and Amazon...)

HTTP Verbs

If you look at a framework like Ruby on Rails it places a great deal of emphasis on using the correct HTTP verbs. What's bizarre about this is even in the original construction of web servers with simple CGI-based forms, the set of HTTP verbs was not widely supported. GET and POST were widespread, but their lesser-known cousins DELETE and PUT were unreliable. This leads Rails to add code like this:

<input name="_method" type="hidden" value="delete" />

So why advocate for a system which doesn't actually work out of the box?

Bi-Directional Communication

RESTful architectures are client-server architectures. All requests originate from the client, and all responses originate from the server. Therefore it is impossible for the server to initiate communication with the client. Sadly almost every application needs server-initiated communication.

For example it'd be great if our email application could tell the client when a new email came through. The RESTful solution to this problem is to poll periodically - a clumsy, inefficient and unreliable process.

Alternatives

Perhaps the main alternative to REST when it comes to APIs is RPC, which could be pulled off with any number of mechanisms (AJAX, WebSockets, ...). But RPC is not a magic bullet, it has its own set of issues (which probably led to the creation of REST in the first place). My point is not to offer an architecture which is superior to REST in every way, rather I think that the application ought to drive the discussion about architecture. If REST works for you application, then by all means use it, but if it doesn't don't be afraid to use something else.

Too often we measure the quality of an application by it's conformity to a set of pre-defined rules - the Thou Shalt Not's of web development - when we should really be treating those rules as suggestions - conventions that someone once found useful in building their own application.

2014-01-21 06:00 Rethinking Web Development: Cloud IDEs

Machine configuration is a hard problem. Having worked at a few different companies on a variety of platforms, one seemingly universal truth is that getting the application up and running he first time is never easy. From the dozens of configuration files that need to be written and the databases that need to be installed, loaded and migrated, to the keys, dependent libraries and source code - you'd be lucky to get everything up and running in your first week, and it will probably take months to really feel confident editing the code.

There's an entire ecosystem of tools related to this problem: machine imaging tools (Docker, Packer, Vagrant, ...), configuration management tools (Chef, Puppet, Ansible, Salt, Capistrano, ...), dependency management tools (Maven, Bundler & Gems, Pip & VirtualEnv, NPM, Cabal, NuGet, ...), build systems (make, cmake, SCons, Waf, Maven, Ant, SBT, MSBuild, ...), continuous integration servers (Jenkins, Team Foundation Server, Cruise Control, ...), and that's not even mentioning the myriad of cloud based solutions often built on top of these tools. It's a dizzying array of technologies: each with their own set of idiosyncracies and deficiencies.

Sadly the truth is I probably spend more time configuring machines than I do writing code: and that's probably just the nature of the beast when it comes to large, complex software applications.

But maybe not.

One way we end up making this whole process a lot harder than it ought to be is by multiplying the platforms we end up supporting. To take a common practice: most startups use some sort of open source stack running on top of (probably cloud-based) linux machines. They also usually give their developers fancy OSX-based laptops. These are radically different platforms (though thankfully not nearly as radically different as would be a Windows laptop). Everything from the way you install software, to the underlying available system libraries is in some way slightly different between the two.

Now building an application which can run on multiple platforms is certainly possible - most startups manage to pull it off. But it's one of those subtle differences which can add up to a configuration nightmare.

For example when I upgraded to Mavericks on my laptop it made it so that I couldn't install the version of therubyracer we use on some of our projects. This is because therubyracer depends on libv8, a compiled library which used C++ constructs which were incompatible with the clang compiler which Apple chose to link GCC to. (Why Rails would require an entire javascript interpreter is a different problem)

Of course newer versions of this library fix this problem, but they also broke our build, so I just figured out some hackish workaround to the problem. (StackOverflow is your friend) I figure out hackish workarounds like this all the time. It's just the reality of working on a complex system with many moving parts. But it sure seems a lot harder than it ought to be.

So what if instead of trying to move our application to our desired development environment, we moved our development environment to our application.

Cloud-Based IDEs

In recent years many tools have arisen which might just pull this off. Here are a few: Cloud9, Nitrous.IO, Koding, CodeEnvy, ShiftEdit, SourceLair, etc... A browser-based text editor is no longer a curiosity or future pipe-dream. It's very much a reality in the here and now.

Consider a setup I currently use:

  • I have a stripped-down, Chromebook laptop. It's small, cheap, does most of what I want and also has excellent battery life. I'm also highly unlikely to break it. (I have a knack for breaking computers)
  • I spun up a 5$/mo Digital Ocean, Arch Linux machine and put all my source code on it. It's fast, always available (from anywhere), easily replaceable and I can have a far greater degree of confidence that the code I write on it will work on its final destination.
  • I'm using the Cloud9 IDE. It has the ability to edit code directly through SSH. I can fire up an SSH session on another tab, run an auto-rebuilder (like glitch), and code away in pretty much the same fashion I always did before.

There are a few cons:

  • You have to always be connected to the internet. (But who am I kidding... I once spent 3 months trying to write code without constant internet access and it was a complete disaster)
  • The tools tend to be immature and buggy and everything feels just a little unfinished.
  • Cloud machines can be unreliable. If a Nitrous.io machine or a Koding machine is unavailable, you can't really get anything done anymore. (the Cloud9 ssh option mitigates this to a great degree)
  • I suppose it could be expensive

And the pros:

  • Collaborative text editing. Try it out in Nitrous.io sometime. It's kind of amazing.
  • Can code from anywhere.
  • No local machine config and a reliable consistent environment.
  • A lot more confidence that my code will work when I deploy it
  • No more having to worry about hosing my local system by upgrading sometime (or God-forbid it crashes for some reason)
  • Onboarding is potentially trivial. Image your dev environment and fire up a new instance. Man do I wish I had this when I taught a bunch of teenagers web development.

I suspect that in the next few years the tools will improve dramatically. But even in their current state this is a legitimately viable option for development, and I kind of wish I had made the switch sooner.