Every company is a software company. I was recently speaking with some developers at an organization that was having a difficult time keeping its software aligned with its other processes.

I was surprised, as it was an engineering organization with a good reputation—very smart people all around, and the developers were very knowledgeable about modern languages and techniques. It had seemed like the type of place where, if anything, there might be a bit of a development nirvana: developers one day working on a piece of close-to-the-metal C code, crunching Big Data using distributed map/reduce the next, and making it visible to the world on the Web by the weekend. Everyone working hand-in-hand to deliver value; mutual exclamations of awe and appreciation filling the hallways; and unicorns dancing through rainbows in the courtyard.

One of the great truisms of software development is Gerald Weinberg’s “No matter how it looks at first, it’s always a people problem.” (Several of the other great truisms of software development are also by Weinberg. His “The Psychology of Computer Programming” and “Understanding the Professional Programmer” ought to be as universally read as Brooks’ “The Mythical Man-Month” and DeMarco and Lister’s “Peopleware.”) A corollary of this is that we tend to suspect that a group of really smart, well-meaning people will inevitably optimize their processes. Sadly, that’s not necessarily the case.

In this organization’s case, the problem stemmed from the viewpoint of an organization that very successfully produced large physical projects that often included components based on  speculative engineering. In other words, the organization as a whole was skeptical that “software is different,” and that software development could not be tamed by the same types of reasoned approaches that work with complex civil, mechanical or electrical engineering.

Here was a problem I’d seen before: the conflation of disciplined software development with “software engineering.” Without digressing into the question of whether or not software development may someday become engineering, it absolutely is not engineering today. Real engineering is as much about techniques for validating design decisions as it is about implementing them, while just about the only thing we know for sure about software development is that we fail when we try to make too many decisions before implementing the next step.

The management at the organization, so familiar with physical engineering, intuited that it needed to increase the upfront requirements and estimation effort, while the software developers felt that they needed to move away from waterfall approaches and adopt more incremental and agile techniques. When I began talking about tactics for introducing the latter, I was again surprised to hear a lament I did not anticipate: “That might work at a software startup, but we’re not a software company.”

This is another common misconception, but again I was surprised to hear it in this type of context. Usually, I hear that phrase from manufacturing and service companies that conceive of themselves as “really” being about the widget they produce; they develop software “just” for increasing their production, sales and distribution. Virtually every company develops software in service to some higher need. The only places where the company perspective is “the software will ship when it’s ready” are the game industry and self-funded startups. (By the way, they tell me that Duke Nukem Forever will ship in 2011 after a mere 14 years in development.)

In the case of this engineering organization, the user experience was almost entirely software-based, which is hardly unusual with today’s machines. It was all in service to the uniqueness and reliability of the underlying physical engineering, of course, but as long as things were not physically awry, the user experience was of control software, not of knobs and levers. (And even if things did go awry, the diagnosis began with software.) Given how visible and important software was, and given the obvious tension between the software and other engineering groups, I was taken aback that the software team was unable to make their case.

Although problems can arise even among a group of really smart, well-meaning people, the happy fact is that such a group has a great chance of working through its problems. In this particular case, I think the key would be for the software group to start with the principles of the Agile Manifesto and show how they align with an organization’s overall philosophy. Rather than starting from the premise of, “These techniques make sense to us as software developers,” they should start with the premise of, “We should use the techniques that best serve these principles.”

While not every principle in the Manifesto is necessarily going to be accepted by every organization, “satisfy[ing] the customer through early and continuous delivery of valuable software” is hard to beat as a principle. Once all parties agree to that, it should be easier to make the case for trying specific techniques on specific projects to see if they help achieve that goal.

And, in my experience, nothing succeeds like shipping working software.

Larry O’Brien is a technology consultant, analyst and writer. Read his blog at www.knowing.net.