Software Sweatshop

The Chicago Quality Assurance Association had an interesting lecture at the Willis Sears Tower on March 24th titled "Building and Managing Successful Offshore Teams Using Scrum" There is a big difference between offshoring internal IT projects versus offshoring commercial product development, but some of the same lessons still apply. In the case of commercial work, there needs to be a vested interest on both sides to adhere to the constant communication... doing this offshore is tricky. The main problem with traditional waterfall development (especially when working with offshore resources) is that problems are hidden and they only appear at the tail end of the project. It's no wonder that almost 65% of software projects fail! How many horror stories have you heard about offshore IT projects over budget, out of scope, or over due because no one knew what the offshore team was really doing?

Benefits of Distributed Agile Development

1). Lots visibility and transparency

2). Were able to deliver work fast enough with acceptable quality

3). Offshore team enjoyed agile/scrum, especially the intense collaboration with each other

4). Not one "onshore" resource lost their job

How To Do Distributed Scrum

1). The most important thing is that they took the attitude that outsourcing would succeed; no matter what

2). They did a lot of risk analysis and thought of everything that could go wrong: communication, source code violations, connectivity, etc.

3). Asked their friends and they said "don't do it"

4). Lots of cultural awareness training... learned about the Indian head bob

5). Refined the product backlog: for the first sprint they had 2 guys at a time in the US. Hires were all fresh off the street, not much experience

6). Created a sprint planning calendar

7). Trained Indian team in scrum

8). Lots of product training in the design, architecture, etc.

9). Held both teams to the same standard

10). Created team bios on MS Sharepoint

11). VSS was a problem so they had to merge code every month, it was a huge pain

12). Made sure user stories are well defined; good user story should be well developed

Challenges with Distributed Agile Development

1). Had to have a few conversations about underpromising and overdelivering with their offshore team because they took too many user stories and then ended up failing at the end of the sprint. this is a common issue with Indian developers. The desire to please is common in Indian culture, so it has to be tempered with a sense of creating realistic expectations.

2). Because of the problems with Visual Source Safe, they didn't see offshore code till after the monthly merge... code review revealed problems which they had to fix

For more on agile offshore software development listen to Andy Singleton's podcast on managing distributed agile software developmentand read Martin Fowler's article about offshore agile software development

Also check out Tackling Offshore Agile Development and Making Offshore Agile Work

Thanks for messing me up Rich... no really, as if I don't have enough problems already... I've got like 8 parking tickets to deal with (written by a vindictive little bicycle cop at that!) But I digress...

So agile development has always been interesting to me but extreme programming never fit my fancy. I knew nothing about it, but the word "extreme" made it sound impractical and unattainable, until I attended a talk by Rich Sheridan of Menlo Innovations (check out Agile Explained) here in Chicago a week ago. It was really good.

In fact it was so good that I sent him this email a few days ago:

Rich,

After your presentation last week I went to another agile (scrum)seminar and spoke to other people about agile (mostly scrum).

I have to say, none of it sounded as cool as the stuff you spoke about. User stories sounds much cooler than "backlogs", one week iterations sounds much better than 30 day sprints, and pair programming is probably one of the coolest programming concepts I've ever heard of. In all, XP sounds more fun, easier to use, and more intuitive.

Scrum seems more popular when working with distributed/offshore resources, so my team will probably end up using scrum, but by planting the seeds of XP, you really messed me up in the head!

Best,

Raza

to which he replied:

Its one of the free services I happily provide! If you really want to get messed up, though, I'll have the charge you for that!

Good luck!

All in a day's work I guess...

We're switching our offshore development team to an agile methodology and I'm now taking this thing really seriously. We'll probably do scrum and adopt principles of XP like pair programming (for more read here) Anyways, if you're starting to work with an offshore development company, you should insist that they follow an agile development method if you want working software.

Here are my notes from Rich's talk:

• Most development teams have critical info tied up in one guy. You have to keep him very happy, or else you're in big trouble. The lesson to be learned is that lone geniuses and primadonnas leave you very vulnerable.

• At his old company, they did an agile experiment. They decided to put all the programmers in one room and let them actually talk to each other, it worked great. They knew they were on to something...

• One day, their client (IBM) walked through and liked their setup... said it was the most amazing thing they'd ever seen

• Rich was influenced by Edison (hence the name Menlo Innovations) who said you oculd have minor inventions every 2 weeks and a major one every 6 months. Verrry inteeeresting... (if you're interested in innovation, check out Triz Innovation... he said there are 40 principles of innovation which can be applied to ANY situation to solve problems... amazing stuff)

• Collaboration needs to happen because when one guy does the same role, the conversation only happens in his own head and he starts making rogue and dangerous decisions himself

• Traditional project management is like a neutron bomb, it'll only kill the people... managing risk logs, work breakdown structures, burndown charts, etc. looks pretty, but it hides real problems

• So they discovered a proces where there were no meetings, but tons of communication, lots of communication but little email, a 40 hour work week with no weekends and defying Brooks law that throwing more developers at a problem slows it down. No cubes, offices, rooms, or doors, but developers were happier.

• Pairing is the most powerful tool they've ever had. Skills don't matter as much as collaboration and asking questions. Goal for new hires is to get their partner hired. Airlines pair pilots... Lorie Williams at the University of North Carolina did an experiment and found that the paired team produced 15% less lines of code with much better quality

• Because of agile development, they have no customer support emergencies.

• Experts don't work on what they're experts in, they just sit nearby and chime in. This is incredibly effective

• Estimating project work is extremely accurate. Each user story has an actual and estimated time. They keep track over the course of many projects to get an idea of how accurate their estimates are. Over time they can give pretty reliable estimates on software projects

• Clients have to take an "agile explained" class and agree to meet once a week. Committment is the main factor. This process won't work for projects where people are complacent and not invested in the process. (

• Testing is not a task, it's an ongoing activity and is build into an estimate

• Developers are NEVER be punished for missing an estimate... ever.

After the talk I asked Rich how to apply agile development principles to distributed or offshore development teams. A few years ago they had two interns from Ireland. After they went back to Ireland they continued to work for Menlo as consultants. Funny thing is that even though they were half a world away, they were paired with programmers at Menlo headquarters in Ann Arbor, Michigan. How cool is that?

A lot of this stuff is just common sense. Seems to me that no matter which flavor of agile development you choose, you'll have a lot better project outcomes.

Offshore software development is a real balancing act. With any IT project, you have to juggle constantly changing requirements, shrinking budgets, emerging technologies, employee turnover, and executive support. Add an 8 hour time difference, cultural barriers, and communication problems to the mix and you have a real recipe for disaster.

Whether you're working in PHP, .NET, or Java, if you're outsourcing software development, you'll quickly realize that offshore software development is about more than cheap prices. Good programmers are easy to find but finding a reliable business partner isn't. Make sure you're getting your money's worth. Look for reliability, experience, intelligence, expertise, committment, process, maturity, principles, honesty, responsiveness, trust, and vision.

How does your offshore partner rank?

Cutting costs is on everyone's mind these days. But before you ship software work over to a 3rd world country, think long and hard... Many companies aren't happy with their offshore team. This is the first in a string of posts about how to find, evaluate, and select a reliable offshore software development team. So if you're thinking about outsourcing software engineering work, listen up...

Outsourcing isn't for everyone. You can't send work overseas and expect to get a quality product without investing a lot of time into it. There are a few steps to outsourcing success: 1) selecting the right vendor 2) Getting executive support  and involvement 3) developing specs 4) having a long-term mentality 5) gettting them up to speed on your industry 6) establishing clear communication guidelines 7) having a good QA plan in place.

Choosing the Right Outsourcing Company

Finding a reliable vendor is probably the hardest part in the whole process of working with an offshore team. There is no shortage of outsourcing companies in India, Russia, Philippines, Vietnam, Ukraine, etc. So how do you actually go about finding an offshore vendor? Ryan Norbauer has a good primer finding outsourcing companies The bottom line is that you have to find someone that you can trust. You have to trust that they can get the job done. You have to trust that they will do what they say they'll do. You have to trust that they'll admit their mistakes. You have to trust that they're technically competent. You have to trust that they actually understand you. Nobody said it's easy...

Global sourcing is great (and profitable), if you approach it with the sophistication it deserves.   Here are a few ways to find the right team.

Thought Leadership - The best way to find a solid offshore team is to look for thought leaders in your field. Are experts in a specific technology or within a certain vertical? Do they publish articles for trade journals, speak at conferences, and blog about their experiences? Are they established experts in their fields? Are they leading the discussion around your industry, the problems you're facing, and solutions they've implemented? If they are, then you've found what you're looking for.

Referrals - Another good way to find your offshore team is to get a referral from a friend. If you're working with someone 10,000 miles away, wouldn't you want to know that they have a solid reputation? Some of our most solid clients came from referrals from current clients or friends. Needless to say, these relationships are usually built on mutual trust and respect.

Freelance Sites - Finding developers on sites like Elance, Odesk, Rentacoder, etc. look easy, but it's kind of a last resort. Ideally, you want to meet your team, or at least talk to them. In our early days, we looked for projects on these sites, but it was difficult to find the large, complex projects that we really wanted to get our hands on. I'm not 100% sure, but I have a hunch that if you have a complex software project and you post it on one of these sites, you'll get a lot of bids with no real way of knowing who to actually choose.

Meet Them - If you're reading this blog, you probably know that I own a Chicago outsourcing company. Most of our deals come from me actually meeting entrepreneurs and tech execs in person. They're based on a trust in our abilities and experience. I think it's safe to say that one of the reasons that we've grown is that we have a local presence, or a "throat to choke", if you will. 

Read - Read blogs, discussion forums, articles, white papers, presentations, etc. to get a feel for how the vendor thinks. Do they have experience in certain verticals? Are they problem solvers or just a body shop? What is their corporate culture? It helps to know how a company thinks long before you ever contact them. Needless to say, this blog is a way of promotion my own outsourcing company, but more than that, it's a way to provide insight. We've seen way too many offshore deals fail because  complicitly sit on the sidelines.

I recently met an MIS manager at a large Fortune 100 firm and he chuckled at the SoftwareSweatshop button I was wearing on my suit jacket. We started talk and he mentioned he's currently working with 2 offshore vendors. One is a large, established player, the other is a smaller, agile, sharper team. Guess which one he likes working with better? He finds himself giving more complex projects to the smaller team because of the expertise and attention they give him.

There's a lot of concern about the growing attrition at offshoring destinations like India. It's probably one of the biggest contributors to offshore project failures. With classic "lift and shift" of IT services to low cost countries, it's little wonder that offshore teams atrophy so quickly. Afterall, most talented developers don't want to do grunt work. Experienced developers either jump ship for companies where they can make the greatest impact, or start their own firms.

I spoke to another guy that owned a small 60 man operation in Chennai. They specialized in healthcare and insurance, but turnover was so bad that it was impossible to keep employees for longer than 6 months. They left as soon as they were trained. This tells me that there are 2 kinds of attrition:

Job Hoppers: These are usually younger employees who are looking for better opportunities. They're not always the most experienced or the most highly skilled, but given the talent crunch in places like India, they are still marketable. Most robust economies have this type of employee. They are the natural byproduct of a healthy, growing economy. The problem is their job hopping causes a major headache for employers under constant pressure to meet deadlines and produce high quality deliverables. It's important to know that most of the attrition in large companies takes place at the entry level, as employees seek to move up in their career.

Top Performers: Top performers are the guys who run the show. They get more done than 80% of their colleagues, and are the "go to" guys when there's a problem. When they leave, everyone gets worried. Top performers usually leave because they feel they aren't challenged or appreciated. They are quite seasoned and usually don't leave a job to gain more experience. If they leave it's because something is wrong. This is true for most professions, top athletes, doctors, salespeople, and programmers have matured past job hopping. They demonstrate responsibility and a willingness to do what others won't. If they don't go to a company, this kind of employee typically starts his own firm.

So if you're working with an offshore vendor, how do you minimize the sting of attrition? Here are some thoughts:

1). Hire small firms that are amazing at what they do. They will attract a certain type of developer, typically experienced professionals that have seen the problems you're dealing with. Smaller firms also look for leaders, but know that not everyone can be one. For example, my friend with the two offshore vendors feels much more confident in the abilities of his small, agile team than the bigger one. What's worse is that he sees the really good talent leaving the larger vendor in favor ofniche firms... or starting their own businesses.

2). Don't look for mere technical skill, look for commitment and a willingness to figure things out. Most vendors are the same in terms of technical expertise. You don't need rocket scientists, but you do need people that understand what you're trying to do and will bend over backwards to help you. Their corporate culture, hiring process, and business philosophy is a good indicator of the kind of vendor they will be.

3). Ask the same question in different ways. How much attrition have they had? What retention strategies are in place? What do they look for in candidates? What attracts employees to the firm? Why employees choose them over others? How do they engage employees?

So most of the companies I talk to get 800 calls a week from offshore companies. They're either working with a solid team already, or they aren't interested in building an offshore team at all. It's not for everyone, but outsourcing works very well if you find a sharp team and are willing to invest a lot of time and energy in developing a relationships and processes with your offshore team.

Afterall, some of the world's top software engineering teams work in remote, distributed environments. The key to working with an offshore team is finding someone that is not only technically competent, but that can add real value. A body shop of developers hacking away at your specs creates is worthless. You might save a few bucks, but it's commodity work for what will most likely be a commodity product.

A condensed version of an article about offshore outsourcing advice from Baseline Magazine. The point is that outsourcing seems easy, but anyone that's done it knows it's no cake walk.

========================================================================================

When outsourcing, organizations that focus on short-term cost reductions often rush through projects without adequate planning, due diligence or consideration of the long-term implications of the inevitable changes in business requirements or offshore market conditions. This not only causes them to overlook most of their savings opportunities, but often leads to project failure. However, an organization that knows what to look out for can avoid the mistakes that others have made in the past, mistakes that can be understood in the context of the seven deadly sins: pride, sloth, avarice, lust/extravagance, envy, gluttony and anger.

• Pride
  Many organizations succumb to the sin of false pride and plunge headlong into an offshoring initiative without performing due diligence. They assume that they (already) have the internal capabilities necessary to plan and manage an offshore operation, when this is often not the case. They also seriously underestimate the management resources required to successfully set-up and run such an operation.
• Laziness
  You can't just move an inefficient operation offshore and hope that lower salaries will result in cost savings. "Lift and Shift" doesn't work, and, when the process is inefficient it will, in fact, often increase the personnel resources required to do the job, wiping out the cost savings the organization expected to achieve.
• Greed
  Many organizations will not be concerned enough about the ultimate fate of the business, or, even worse, possess disdain towards the offshore operation. This makes it difficult for the offshore operation to retain knowledgeable and productive staff, leading to quality problems and cost overruns as greater numbers of inexperienced resources need to be thrown at the problem.
• Lust/Extravagence
  The desire to solve a problem by taking on more, cheaper personnel is extravagant and wasteful and has serious implications for service quality. It's important to remember that many offshore operations have lower productivity and excessively high turnover, reducing cost savings. Don't give in to the impulse to compensate for low productivity with more bodies: It's a false economy. Before you offshore the process, make sure it is efficient. If necessary, re-engineer and improve the process first. Furthermore, even after your processes are off-shored, it's important to continuously apply performance improvement initiatives.
• Envy
  Don't assume that offshoring automatically comes with big savings and be envious of your peers who are already doing it. Most of the claims you'll find by the promotors of the strategy don't take into account the lower productivity that comes with offshore personnel, higher communication costs with an offshore team, and the additional overhead required to govern an offshore process. Actual savings are often only half of what is claimed.
• Gluttony
  Don't offshore as much as possible as quickly as possible, like a glutton, with the ill-formed belief that this will maximize savings. Don't overlook the average organization's capacity to digest change, and even smaller capacity to digest offshore change -- because organizations that offshore too much too quick often spend the majority of their time firefighting. The key to success is selective offshore outsourcing, following a careful analysis of what processes are the most likely to lead to savings if outsourced.
• Anger
  Don't blame the outsourcer when the savings don't materialize, especially if you committed one or more of the sins above. Blame-wise, at least half will always rest with you, and if you committed multiple sins, all the blame rests with you.