Dropbox Wrangles With the Server State Dilemma in its API
Dropbox developers have a problem, and it’s this: The contents of a shared folder in Dropbox’s storage should have a function that allows the user to request only the most recent changes. But the combined length of the parameters for this function exceed what an old World Wide Web Consortium (W3C) document said — though not very pointedly — to watch out for when implementing a GET method. Should Dropbox implement the POST method instead, in a way that the Web doesn’t expect?
Let’s face it: If Dropbox goes against the grain, but its implementation works, whom does it hurt?
You GET What You Pay For
It is no secret among Web developers that Hypertext Transfer Protocol was not created with the intent of managing remote procedure calls, nor did its architects envision it for that purpose. In one of the founding documents of HTTP, written by Tim Berners-Lee in 1991 — a document that today looks more like something jotted down in Evernote — he abruptly closes his list of ideas for version 0.9 with the following: “Requests are idempotent. The server need not store any information about the request after disconnection.”
Without realizing it at the time, Berners-Lee planted the seed for the principle that made remote interactions between services possible using HTTP: Requests don’t change server state. If they did, the asynchronicity of Web transactions would render any multi-user database scheme impossible.
That version 0.9 document did not even include the word POST.
When it became evident HTTP would be an exchange for transactions, the POST method was created to let browsers (later, “user agents”) provide servers with data intended for a database, the transfer of which could imply a change of state. POST was not idempotent; one method executed a second time by a server could change the state to something unexpected. Oftentimes the browser doesn’t allow POST to be reloaded without warning the user first.
There would seem to be simple distinctions between the two methods. GET seems passive enough, while POST is more of a demand. GET doesn’t change the state of what it’s getting; POST does affect the state of the resource receiving its data. They appear to be in clear opposition to each other. So you should know which one to use, when, and why.
But they are not exactly mirrored opposites, and that’s the problem. Although Berners-Lee did not imagine a maximum length for the GET method, years later, when the W3C specified HTTP 1.1, it cautioned that such a maximum may exist virtually. “Servers ought to be cautious about depending on URI lengths above 255 bytes,” it stated, “because some older client or proxy implementations might not properly support those lengths.”
The Unexpected Virtue of Idempotence
“HTTP was developed for a specific hierarchical document storage and retrieval use case,” the Dropbox development team stated on its company blog Monday, “so it’s no surprise that it doesn’t fit every API perfectly. Maybe we shouldn’t let HTTP’s restrictions influence our API design too much.”
It was part of a document explaining to other developers why Dropbox chose to implement a particular API call, named /delta, using the POST method of HTTP rather than GET. It’s not like Dropbox broke any hard and fast rules. For years, it’s been a commonly repeated suggestion to use POST instead of GET in order to leverage HTTP’s message data field as a location for long parameters.
POST is typically more reliable than GET, says this tutorial on Java Server Pages, and should be used instead whenever sensitive data is being passed to the server — on account of GET’s parameters being encoded in plain ASCII as an appendage to the query string. And in this tutorial from “Compare Anything” site Diffen.com, a commenter suggests POST is superior to GET for its looser size restriction alone. So if Dropbox stepped on a landmine, it’s been stepped on before.
But perhaps because it’s Dropbox, the move has met with some criticism. On his personal blog Monday, Evert Pot, a lead developer with the Sabre project and a member of the PHP Framework Interop Group, calls Dropbox’s decision “poor API design.”
Yet Pot doesn’t put the blame on Dropbox’s shoulders. “We’re clearly running into artificial limitations here that are poorly solved by existing technology,” he writes. He suggests an alternative that was first specified in 2002 as part of an extension to Web Distributed Authoring and Versioning (WebDAV), a system that’s still officially under development for enabling remote editing and collaboration of files in Web servers.
REPORT can have a request body, argues Pot, is safe, and is idempotent. It probably got forgotten because WebDAV is largely forgotten, he says. “However, its semantics are well-defined and it works everywhere. I would love to see more people start picking this up and adding it to their HTTP API tool belt.”
Pot links to this six-year-old question he posted to Stack Overflow, which essentially boils down to this: If any HTTP method can have a request body — the part of the POST method that makes it attractive to Dropbox — then GET can have a request body as well. So why not use it?
The highest rated answer to that question, over this long stretch of time, came from HTTP principal author Roy Fielding. Indeed, GET can theoretically be parsed as if its message body were useful, Fielding advised. But since it’s not useful, it should not be parsed that way.
“The requirements on parsing are separate from the requirements on method semantics,” Fielding wrote. “So, yes, you can send a body with GET, and no, it is never useful to do so.”
Still, one developer responded to Dropbox’s blog post Monday with a comment suggesting that using a request body with GET has never been expressly forbidden. But in a comment posted Wednesday, insurance industry developer Sean Chambers posed a kind of “stateless” solution, using the PUT method instead: “You can still enforce HTTP semantics by performing a PUT to the ‘delta’ resource with the updated state,” Chambers wrote, “and the new state of that resource would be the delta between your two other resources.”
Thus it would appear a solution is feasible to Dropbox’s dilemma without mangling the protocol, and also without following someone else’s principles purely for aesthetic reasons. A long-term solution to the problem for all developers may be for W3C to amend the specifications for HTTP/1 and HTTP/2 in such a way as to explain the proper way to apply the old methods in a new context.
Feature image via Flickr Creative Commons.
