As a matter of fact, there is already a railroad track connecting our two cities. A few times each week, this track is being used for freight transports, chemicals and other stuff you probably do not want to know about. The local train is not allowed to use that track for security reasons. For a change, I will not blame my self-proclaimed security-paranoid friend and colleague Arne Blankerts for complicating matters, but I have to admit that I feel rather comfortable knowing that the likelihood of a passenger train running into a freight train loaded with dangerous chemicals in the middle of my hometown is somewhere close to zero.
Next to the train station, there is a railway gate, because the existing railroad crosses a major east/west road. Since only a few freight trains per week pass by, this is not an issue. In fact, I have never seen the railway gate closed in more than five years.
The plans to extend the railroad, which is undoubtedly a good idea in itself, have sparked a heated discussion: is a railway gate suitable or should an underpass be built? The frequency of passing trains will increase from a few trains per week to one train going forth and one train going back every twenty minutes.
As you can imagine, local politicians, employees of the railroad company, and, of course, citizens, have been taking part in this discussion. I have to admit that due to time constraints, I have never attended those discussions, but I have read through the meeting minutes. The discussion boils down to one party claiming that level crossing will cause disastrous traffic jams, while the other party claims that given the short closing cycles, the impact on car traffic will be barely noticeable.
From a legal point of view, the discussion is most interesting: according to law, no new railroad gates must be built. Existing railroad gates, however, can remain. My personal point of view is that a railroad gate will be a disaster, and drown our city in a permanent traffic jam. This view is based on my personal experience. In the city where I grew up, there was a level crossing with a railway gate, which was replaced by a underpass at some point, because the waiting time for road traffic was unbearable. In nearby cities, more railroad gates existed. Since they caused similar problems, they were all gradually replaced by underpasses. As a small anecdote, after the underpass had been built in my hometown, a new firehouse had to be built, because the underpass turned out to be too shallow for the fire trucks.
So why, about twenty years later, go with level crossings again? Did nobody learn from the past? Or are twenty years enough to forget all the lessons learned, and start over by making the same mistakes again? After all, we have gradually replaced existing railway gates in the last century (it sounds much more dramatic this way).
The discussions about whether it is acceptable to cut a city of 18.000 in half by a railroad gate that is closed twice every twenty minutes somewhat reminded me of the discussions that frequently take place in software projects. There are different positions, and usually a few people on the table are absolutely positive about a fact or decision, but they usually cannot back up their claims by hard facts and figures, at least not ad-hoc. Still, a decision has to be made. So who wins? As Eric Ries so aptly put it in his book "The Lean Startup" (a very recommended read, by the way), the biggest ego usually wins.
Does this lead to the best decisions? How often have you encountered a manager, senior, or a paying customer who, with absolute certainty, could explain to you how the world turns? And how often did this explanation turn out wrong? These days, more and more companies question the traditional way of making decisions, and run experiments instead.
A website or a web application with a significant amount of traffic puts you into a great position for running experiments. All you need is a way of running multiple software configurations simultaneously, which is rather easy to do if you are using some sort of dependency injection container or factory that can be configured to "substitute" implementations (classes) by alternative implementations.
The selection of the software configuration can be based on whatever criteria you deem useful. To provide the user with a consistent site or application, the decision whether to run an experiment is made just once for each session. Being able to cleanly run experiments ("cleanly" meaning without polluting the code with lots of if
statements) comes almost for free with a modern and clean software architecture. If it does not, you might want to rethink your software architecture, or complain to your framework vendor.
It is actually pretty difficult to determine whether the result of an experiment is statistically significant. When you make the "Buy" button in your web shop bigger, for example, sales might go up. But are you sure that this increase can really be attributed to the bigger button? Maybe you ran a special offer at the same time that gave your shop a cross-selling boost?
Just because all the statistics involved are a difficult beast, you should not give up too early. Many successful online and offline businesses have been experiment-driven for quite some time. Fast food restaurant chains try out something new in a few restaurants, and if it works, they will roll out the experiment to all restaurants. IKEA does the same with furniture. Amazon, to name an online business, has been continually running A/B tests for a long time. Booking.com, an online hotel booking agent, is very successful with continually trying out new rapidly prototyped features that get cleaned up once they have demonstrated that they deliver business value.
As for the extended railroad, why not close the railway gate as if the two local trains every twenty minutes were already running, and keep doing this, say, for two weeks? It would be easy enough to find out how this would affect traffic by measuring the queue length or average waiting time. Should it turn out to be a real disaster, the experiment can be ended quickly, and without any significant cost. This seems like a much better choice than having to build an underpass to fix immediate problems, with ongoing train and car traffic.