Tuesday, February 2, 2010

Lesson #2 - Consider Hosted Software Development Tools

This is my second post (the first is here) in my series on Lessons Learned from Zeemote.

In 2006 I was the first software engineer hired at Zeemote (a now defunct company) and at the time we still didn't even know what our product was going to be. We had a few hundred lines of demo code (written mostly by an outside contractor) and we had no bug database or version control in place. The closest thing to version control was zip files stored in email attachments.

Like many small companies we used an outside IT firm to manage our network and computers. Unfortunately as time progressed some in our company lost confidence with the firm and they were replaced. This became a recurring problem and during my time at Zeemote I witnessed 4 or 5 IT firms get hired and fired (a discussion I'll save for a future post).

One consequence of the above was that our developer tools (subversion and bugzilla) were being maintained by our software engineers rather than the IT "company of the month". Maintained is too generous a word though. As we grew and became increasingly busy and stressed out we had little time to really care for these tools. I don't think we ever made a single version upgrade and only occasionally checked that backups were functioning properly. I began to lose sleep over the situation.

Soon another problem presented itself. As we rushed to develop and test our second software product we contracted with two outside resources. One was a development house in London and the other was a testing house in Pennsylvania. We were suddenly becoming a global distributed software development company! But to use these guys we needed to provide remote access to our internal version control and bug database. This would require the IT "company of the month" to set up VPN and making sure that they only had access to what they needed. Given our experiences with IT this was going to take too long and the results were going to be error prone.

After a few intermediate steps we ended up with, what I regarded as, a great solution: hosted software development tools.

Over the past decade nearly all software products have been moving to the cloud. As it turns out, software development tools are following this trend too and several companies now offer hosted version control and bug tracking. These companies host and manage the software on their servers. They provide encrypted remote access to the tools and also a web based interface for simple administration tasks.

They handle the hardware and software maintenance issues (backups, upgrades, uptime) and you focus on your core work.

After some deep thought I came upon with the following minimum requirements that any service provider should meet in order to be considered:

  1. Must allow automated downloads of nightly backups to our servers. This protects us in case the provider goes out of business or has a catastrophic failure.
  2. Must use an open / standard data format. This prevents vendor lock in and allows us to easily switch providers if we become unhappy with them or they go out of business.
  3. Must provide a basic NDA or terms of use policy that clearly states the data is ours and reasonable steps will be taken to protect it. This is to minimally satisfy the IP folks and investors in the company.
  4. Must be within our (modest) software budget. Affordability is always a requirement.
  5. Must speak with us on the telephone and instill confidence. Anyone can set up a website these days. Talk to a real person before choosing a provider. A serious company will be happy to give you a demo and answer any questions or address any concerns you may have. After speaking with one company we ran for the hills as it became clear the "company" was just two college kids with precious little experience.
As we moved forward I faced very strong opposition against my plan to outsource these tools. I was surprised to find that my strongest opposition came from within my own software group! In retrospect I shouldn't have been surprised since they would be the ones using the tools so it would impact them the most. Some developers worried about the potential of poor performance or downtime while others simply struggled with the notion of giving another company "control" over our tools and data.

The concern about poor performance or downtime could be mitigated (at least partially) in two ways. The first was that if performance or downtime did become a problem we could always switch providers because of requirements #1 and #2. Also one of the tools we hoped to use, subversion, is explicitly designed to work "offline" anyway. In most circumstances when a subversion server is inaccessible, one can simply continue working and sync back later.

The second concern about losing "control" of our tools and data was, I think in our situation, more of a mental road bump. It was a bump that I initially struggled with this too but eventually got over.

Requirements #1, #2, and #3 show that we would be the ones who maintained ownership and real ultimate control. Sure we would be allowing another company (a partner) to have access to our code but that is a fact of life for most software companies. If you don't want anyone outside your company to have seen your code then:
  1. don't form any tight development partnerships
  2. don't hire any contractors
  3. don't fire anyone
  4. don't let anyone quit or retire
Another mental road block that software developers seem to have is in thinking that code is the company's "crown jewels". We need to get over ourselves! Of course code is really important but it is just one part of your business. Had our source code been accidentally (or purposely) leaked it is exceedingly unlikely that another "joystick for cell phones" company would have emerged or that any real damage would have occurred to our business. In fact releasing the source code probably would have had a net positive impact on our business rather than a negative one.

So sure it wasn't perfect (few solutions are) but we had to move very quickly and what we were doing wasn't sustainable. Having software engineers worry about maintaining servers and backups isn't productive or efficient. In balance outsourcing these tools was a no brainer.

While I faced internal opposition a new CEO joined the company and (luckily for me) he was open minded, willing to hear me out and gave me great support.

After searching we went with Codesion.com (formerly CVSDude.com). For an extremely cheap price they provided terrific service for us. While we did have a few outages and a few performance problems they were all quickly corrected. Overall it worked out great and I highly recommend them.

The lesson learned here is that hosted development tools can provide a great option for many software companies and should be strongly considered when building your development team's infrastructure.

I'm very proud to say that at Zeemote, Inc. we never lost of single line of code or piece of change history since the first week I joined the company.