Return to Conference Proceedings
The session will discuss some of the problems this increasingly common problem raises. These include:
The paper discusses the costs and benefits of doing business this way:
Possible mechanisms to address the problems and costs are discussed. It is obviously possible. There are lots of organisations doing it. Various mechanisms and some of the costs associated with them are presented.
All of these are valid and, in many cases, unavoidable reasons for doing development geographically dispersed. However, there are many costs (some hidden) associated with this style of development. For example, relocating staff is no doubt expensive, but then the cost of setting up the appropriate hardware, software and procedures for dealing remotely can become expensive as well.
Outsourcing is a fact of life for many organisations these days. Keeping up-to-date with what the outsourcing company has done is another matter. Many companies are taking advantage of the cheaper labour available in third world countries. Many large multinational organisations have development groups in many countries. There is frequently a need to split a project up as it is too large for one group to handle on their own within the time frame available. Sometimes this can result in sharing between many cities, states or countries.
Whenever there are multiple sites involved in a project, there is often a manager in charge of the overall project, but there is rarely someone charged with monitoring the project progress on a technical level. As a result, seemingly unrelated problems are reported separately at each site without anyone realising the real issue was the relationship between the sites.
At another level, each site is often aware of the costs it is bearing by being part of a dispersed development effort. They are not aware of how much the costs are escalating for the entire project.
In many cases, a problem is attributed to something like a 'bug' in the software that needs to be corrected without highlighting the cause of the bug. For example, the cause of the bug could be the merge of software from two sites being done incorrectly. Problem-tracking systems are only as good as the information they record. It is always difficult to document the real cause of a problem. Multiple site development just makes this problem worse.
Scenario 1: Large company spreads the load amongst different development groups
There are three pieces of a development project. Business units in Melbourne, Sydney and Brisbane are each assigned a component. The rest of the application has little impact on the work in Brisbane. As long as it is ready for integration testing, the Brisbane staff can effectively ignore the rest of the application. The Melbourne group is building the main body of the application. They need the sub-components being built by the Sydney group to even compile their code.
The main issue here is that the Melbourne group need to keep constantly up-to-date with the work in Sydney. Also, Melbourne needs to provide feedback to the Sydney group on problems and changes. Most organisations manage this problem manually. Typically there will be some sort of software configuration management at each site, but there is nothing to control the interactions between the sites. Almost inevitably some files will get out of synchronisation. Both sites will think they have the 'real' version of some file (or files) and development can proceed for sometime before the error noticed. The time required to recover from a situation as this can be considerable. It will take time and resources at both sites to recover.
There are other problems in this scenario. For example, typically, one or more people at each site spend a significant portion of their time just keeping things synchronised. This is a significant investment of staff that should be working on the main development effort. To make matters worse, they are frequently very senior people as it is vital to minimise problems in this area.
Scenario 2: Software development firm uses cheap labour
A major software house in the USA has decided it is more cost effective to do the bulk of the work in India. Head office integrates all the work done in India daily. The staff at head office are developing specialised pieces of the software and performing integration testing.
The main issue here is keeping up with the sheer volume of information. Typically there is a small group at the 'head office' end of the development, and a very large group at the offshore company. In effect, this scenario has the same problems as scenario 1. Different time zones can make it worse, depending on the actual countries involved.
Scenario 3: Distributor performs internationalisation on product
An American software vendor wants to sell its software into Japan, but does not have the skills to port the product to Japanese. The Japanese distributor offers to help do the work.
The distributor needs to see the code from the vendor to see where to make changes. They also need to be able to send information back to the original vendor about changes required in the base code to support internationalisation. There is a further complication in this scenario. It is the need to support double byte character sets, ports for additional languages, and by the need to possibly support multiple releases of the software.
Scenario 4: Customer can tailor a product, but the vendor still supports it.
A software vendor sells a package that the customer or vendor staff can tailor at the customer site. If there are problems with the implementation, the vendor still supports the product even though the customer has changed it.
The challenge here is to keep track of what your customers are doing. In many cases they could easily make a change and forget to tell you about it. Different sites will have different standards. It is very difficult to make all the sites follow a set procedure that will help you keep in synchronisation with all your customers and their individual tailoring.
"More is worse' scenarios:
Bigger may be better, but for software development, more is definitely worse. Supporting extra sites, time zones, operating system versions, programming languages and product releases just adds to the complexity and opportunity for error. There may even be a point at which adding another site may end up costing more in support that the site can add to the development effort.
In the example, we said that there was regular interaction between two sites and occasional interaction with the other. In this example it would be reasonable to say that two people are spending at least ten per cent of their time keeping the two main sites synchronised. One person is spending around five per cent (at the third site).
In addition, two people would be spending at least twenty per cent of their time merging the code between the two main sites. Even at this simplistic level, the costs start to mount:
2 x 10% x $50,000 + 1 x 5% x $50,000 + 2 x 20% x $50,000 = $32,500
This adds up to thirty-two thousand five hundred dollars, or sixty-five per cent of one man-year just keeping the sites synchronised. This does not take into account the amount of developer time to understand the changes that have occurred at each end as a result of the merges. It also does not account for rework needed due to misunderstandings and incorrect versions of files being sent between sites.
Again, it is difficult to give hard numbers in this environment, but it is possible to consume two or three man-years just because of the difficulties communicating between the sites. Every site added increases the effort at all sites to keep synchronised.
In nearly all sites, it is possible to improve the mechanisms in use in these areas. The ways we make the improvement will vary, but will nearly probably require automation, and integration into the development procedures and other processes in use.
The following is a list of requirements that a better mechanism for managing the interaction between development sites would have to meet.
This is not an exhaustive list. A tool that can meet all these requirements has the potential to save many man hours of development effort and rework.
The difficulties are mostly in the area of communication. Communication of the software under development and between the people performing that development are the biggest problem areas. These difficulties can add a serious overhead to your development effort.
As with many activities in software development and operations, it is possible to tackle the problem by automation. Any organisation that is currently involved in, or considering geographically dispersed development should pay close attention to the costs, methods and procedures associated with it.
Return to Conference Proceedings