cyberhoboing with dominic tarr

Why Software is Important

A Hypothesis:

The demand for software increases exponentially, each piece of software enables more software. As we get more software, we want more and more.

Before I learned to program, all I wanted to create was a few games. Now that I am much more experienced, and have written a lot of code, I now have so many ideas that I’ll probably never be able to implement them all. Even if I never have another idea again… As I implement new things I learn new skills and thus get even more ideas.

Here is a short list of some things I’d like to exist:

  • A role playing game where you cast magic spells my typing “magic words”. Except the “magic words” are actually a programming language, but you won’t realize you have learned to program until you have gotten good at the game.

  • video chat that uses facial recognition to animate a virtual puppet with your expressions. If you had eye tracking, you could also animate eye contact. When another participant looks directly at your avatar, you should see them make eye contact, but another participant should see them looking to the side (just like in physical space)

  • An operating system where every application is downloaded over the network and runs inside a secure sandbox. Like tabs in a web browser, except native, and with fully configurable permissions. Also, like an app-store, but without a central authority that rejects applications they dislike.

  • An open-source webmail that is as good as gmail.

  • A system of polyfils so that it’s possible to pipe data from the command line directly into widgets in a web browser. I want to be able to pipe a csv into a graphing application in a browser, even if it’s on another computer. I want to be able to write an HTML file, and push deploy it to a particular mobile device. I want to be able to tail -f the status of a check box, and see each update as lines of true and false in my terminal.

  • A modular database where you plug together primitive parts that are optimized for a particular type of data, (say, mutable data, append only data, graph data, relational data) we have begun to explore this idea with level-db. What if you could have a database that profiled itself and switched out different underlying data stores depending on how you use it, like how v8 optimizes javascript dynamically.

  • A framework for creating arbitrary applications that are highly secure and are p2p, without a central server.

  • An HTML5 map application that works offline (that can replicate areas of the map)

  • A 2d drafting application that where you select shapes and can use those shapes to describe new shapes, say, two points can describe a line, or a circle. and a point and a circle can be used to describe a line that passes through the point and is tangential to the circle.

  • a secure p2p package manager, like npm (used in the os mentioned above)

  • A text editor that runs in the terminal, but has all the same keyboard shortcuts as notepad (so user interfaces are consistent, and you don’t end up like vim users, accidentally hitting :wq to send an email)

  • A system to describe the behavior of FSMs with a regular-expression-like-DSL (regular expressions describe pretty much the same layer of the Chomsky hierarchy as finite state machines, but are much more familiar to application developers) This could be used to verify the correctness of systems that are both async and stateful (something I’ve found very difficult to test)

  • A pure js implementation of node.js’s crypto module.

Some of these I have started on already. some of these are less ambitious than the others, but all of these would comprise of a considerable number of sub problems, and would also lead to new possibilities, and give me new ideas for new programs to write.

Software does not have a talent drought. The demands for software is increasing faster than new humans are learning to code. Even in just a few decades, most parts of the structure of society have become computer programs. Between the point that photons hit the camera’s detectors to when the jpeg is rendered in your browser, the news is all software. The banking system is all software. The music you listen to is recorded with software, if not made entirely with software.

Yet, despite all this, the quality of most software leads a lot to be desired. If you have had the misfortune to be exposed to the enterprise legacy systems you’ll know what I mean.

The biggest problems facing humanity are really software problems.

Even politics is a software problem. Politicians use spin and scare tactics, instead of making decisions based on objectively measurable data - because there is no software to facilitate the right kind of data gathering and decision making.

A lack of education is a problem, because there we the right kind of educational software.

Currently, humanity stands at a crossroads like never before. Great power and great changes are looming on the horizon - genetic engineering, nano technology. Yet also great dangers - Climate Change, peak oil, and the fact that western governments are getting stupider and more corrupt. We need better software to solve these problems.

The good news is that we have already figured out how to reliable create high quality software. Open Source. Open source has already created all the best Operating Systems, Databases, Programming Languages, and Application Distribution (web browser & virtual machines). The problem is that open source is not yet wildly distributed. Apart from developer tools, most consumer facing or enterprise software is generally not open source.

I think the problem is that there has been no way for consumers to pay for open source. Open source has the illusion of being free - but it’s not. You pay for open source by contributing to open source and by writing new software. The open source community is what anthropologists call a “gift culture”. This happens often when there is an abundance of resources (because of the ever increasing demand for software). Instead of gaining power and respect for what resources you control (such as in a commercial culture), in a gift culture individuals gain respect and status by what they give away. Often, in a gift culture, the most respected individuals are by no means the wealthiest. This is certainly the case in open source!

What we need is a way to turn open source inside out, or at least expose a way for people who do not have technical skills to meaningfully contribute to it. Paying for the software in the traditional way is out of the question. To be able to sell something, you need to have the power to withhold it until you have received the payment. This is impractical with software because it’s so easy to copy. If you want to sell software in a traditional way, you have to add copy protection features, which actually make the software worse! On the app store you can purchase software in single units - but to be able to do that, apple controls the entire ecosystem all the way down to the physical hardware!

But there is one way you can reliably withhold software, and thus effectively demand payment: There is no easy someone non-technical to get software that has not been written yet.

You can’t sell software, but you can sell a promise to write software.

Before software is written is hangs there only as a lingering possibility. It must be mined out of the ether by highly trained monkeys in a process where they drink coffee, stare at screens, and hammer away at some very fancy typewriters. There is pretty much no way to get software that to feed these super-monkeys, or to be one. Being one is a curse as well as a blessing because the rate at which primates can create software is really rather low, not more than a few hundred lines of tested, working code a day on average. It’s totally feasible for one person to grow all the vegetables you need to live, but you could not in your lifetime create all the software one would like to implement.

Fortunately, it’s possible for these monkeys to work in teams. In fact, it’s really quite simple, because they actually work better if they are just let to themselves and given only the most gentle direction. If you try and dictate what they do they’ll only resent you. All that is needed is the right sort of guide to encourage them to communicate in the most effective way, and make it more rewarding for them to work together than to work on their own.

This is essentially what git, github and npm accomplish. Over the years, we’ve made pretty good progress in this area, but this has only increased the insatiable demand for software, and raised our expectations for software quality and functionality.

We have such need for software that we really need to look for a force multiplier, that increases our ability to create software. We don’t need to find another low hanging fruit, what we need is a ladder.

I have an idea for a ladder. Will be announcing it shortly. Watch this space!

(here it is: feedopensource)

Some Thoughts on the Economics of Software Development

The problem with software development, is that the ways used to fund it are either more suited to physical products, or are otherwise out of line with the interests of the users of the software. Here are my thoughts on the ways this fails, and also an idea for another approach.

A Summary of the Current Methods

There are currently only a couple of ways of funding software:

  • VC-funded free service
  • proprietary software licensed for a fee
  • open source, funded by charity, or by the creative drive of it’s author.

Each of these methods create a different kind of software. Essentially, different sources of funding mean different motivations, and these result in different decisions by the developers.

The “Free” Service

VC-funded software seems very good at first, but, they aren’t really doing it for free.

The investors are looking to make money, and there is often a “business model” question hanging over the software. This question is nearly always resolved by changing the software from something that enables users, to something that manipulates the user. If this is the sort of software you would like to use (and be used by), then you are already well catered to! Your Welcome!

Private Source

Someone builds the software, then you buy it from them.

What could be simpler? This works well with physical goods, because the exchange is easy to enforce. For example, I have made a bicycle, and you want to buy it. You give me some pieces of paper with pictures of George Washington, or Chairman Mao or whatever, and I give you a bicycle. This is naturally in line with the physics of physical items, but software is totally different. Instead of giving you the bike, I copy the bike to you. Now there are two copies. What’s more, you can also copy the bike to someone else. Now there are a lot of people riding my bike, without me being able to recover the value that I think I deserve for creating it.

So, to make the private source model work, you have to add extra copy protection “features” to make the software behave more like a physical good. Like turnstiles on a subway, these copy-protection features prevent you from accessing the natural physics of software. This takes extra work, and so you end up paying not just for the product itself, but also for the work to cripple the product so that it’s more difficult to copy.

If the value of the software is high enough (enterprise), then they can just use lawyers instead of copy protection. I’m not sure which is worse.

This is widely used for games, or enterprise applications. Apple’s App Store is a great example of an elaborate system for making software difficult to copy.

You purchase a crippled phone that can only install software that apple likes, and then you can pay for software, and also 30% to apple for the service of stopping you from being able to install anything else.

Open Source

Instead of going to the hassle of charging for software, open source just gives it away.

This eliminates the costs of erecting turnstiles. It also reduces the cost of evaluating software. You don’t have to pay for something to try it out. Open Source has created all the best operating systems, programming languages, developer tools, and databases.

But the scope of Open Source is pretty much limited to this. Open Source is created by enthusiasts, and it does get paid for, but intangibly.

A project usually starts because someone thinks it might be fun to implement. But then, if that project becomes useful, it becomes in the interest of other well funded parties. Any large project requires an extraordinary attention to detail and refinements.

It would not be possible to pursue such a project at hobby scale. though, still, there is a huge amount of valuable software essentially created by hobbyists. Tragically, these hobbyists may work a day job writing “free” services or private software.

Moving Forward

There have recently been some attempts to move beyond the hobby economics of open source. Each of these are interesting in their own way.

gittip

gittip.com enables you to anonymously donate to someone. This is the digital equivalent of busking.

You can easily give a small amount of money as encouragement to an artist who’s creating things you enjoy. But this doesn’t really influence them in any way. This is an interesting model, but not many people are earning enough to survive like this (and thus devote themselves fully to their art).

By the way, if you enjoy my creations, please feel free to offer me a token of your appreciation: gittip.com/dominictarr

Kickstarter, Indiegogo, etc.

Another approach is crowd-funding the development of a project.

The developer makes a bunch of promises of what they want to deliver, and how much they think it will cost. People either fund it or not. And then, development begins. If you have been following any software projects like this recently it won’t be hard to think of examples of one that hasn’t really gone according to plan.

All the crowd-funding platforms and geared towards a single, large, funding round. So, all promises and plans must be made up front, well in advance.

The problem with promises made well in advance is that they are hard to keep. Not necessarily because you didn’t mean them, but just because the situation changes, and when you start coding you learn things you didn’t know when you started.

Like representational democracy, you don’t know if the party you elect (fund) will keep their promises, or even are capable of achieving them.

Essentially, Kickstarter forces you to use the waterfall model. The waterfall model is widely understood to result in low quality software that is delivered late and over-budget. Quality Software must be built iteratively, changing the plan as your learn things, but the economics of crowd-funding platforms prevent this.

Bountysource

Bountysource takes the opposite approach of Kickstarter.

Instead of funding a whole project all at once, you can fund a specific issue, or feature that you want. It’s really too early to say what will happen with Bountysource, but my feeling is that Bountysource is too specific where Kickstarter is too vague. The features of a software project are not isolated. You can’t just buy a feature or bug fix. On the one hand, casual users may not understand which issues are really important, and on the other there may be a refactor that simplifies multiple features, or eliminates multiple bugs. At best, this may be suitable for the developer tools that open source is already good at.

If open source is to go truly mainstream, it needs be able to create software that is used by people who are not themselves developers, or even particularity competent with computers.

One More Way!

There is one more way software is funded.

If you have a software problem that is particular to you, or just want your own thing, you can hire a consulting company and get them to make you a bespoke system.

This can actually work pretty well, provided that communication between the developers and the clients is good, and an iterative process of building a prototype, testing it against client expectations, and refining/rebuilding the design. This process is often known as Agile Development.

The developers and the clients both have power in their relationship. The developers have the technical expertise to write the software, but the clients have the checkbook. If a project is going badly, they can potentially abandon it and find new consultants.

A lot of software is built this way, but it’s not scalable. The weakness is that is that it must provide sufficient value to a single party, who must be willing to foot the bill entirely. It doesn’t support the case where a small amount of value is provided to a large number of people.

But what if it could?

My idea

What we need is something that aligns the interests of the developers and with the interests of users. Something that works with the natural physics of computers and data, not against it. Something that scales, so that you can use to fund end-user software - not just developer tools.

Something halfway between Kickstarter, and Bountysource, but with the best parts of Agile development. Development and funding must be iterative. The developers should not have to make promises that will be hard to keep.

Instead, developers should build trust with the clients by making steady, incremental progress. If the progress is in the wrong direction, or the developers turn out to be incompetent, then clients can abandon the project without having over-committed.

If a project goes well, the clients can continue to fund the project. Since the uncertainty decreases as the developers build the clients trust, they could potentially fund much more than they would with a single round of funding, where they have to balance their desire for the promised features with their estimation of the projects eventual success.

I am currently working on a prototype of this idea, Watch this space!

Mad Science Architectures for Totally Distributed Applications.

What is scalability, anyway?

leveldb - followed by part2 http://www.youtube.com/watch?v=a8rTAIWjIpw

data, events, streams at webrebels 2013

watch live demos, and event the presentation software fail (based on https://github.com/jed/weenote) in the end I save the day by making a joke about java.

the database of the future

wizard talk.

distributed systems for realtime applications.