A glittering performance of rare perception http://typicalprogrammer.com isOdd (and isEven) as a service

tl;dr Introducing new web service to determine if a number is odd or not.

Try it:

Is 43 odd? - yes!

Is 2 odd? - no

Is 3874.32 odd? - no, not an integer

Is foobar odd? - no. not a number

Rationale

A while back I found out about the is-odd npm module that give node.js developers a handy function isOdd to determine if a number is odd or not. The module has something like 500,000 downloads every day! Clearly this module fills a huge need in the node.js development world. But some people have expressed concern about excessive dependencies in the node/npm ecosystem. I thought, why not make this a web service? I can solve a few problems all at once:

• No new code dependency in your npm modules
• Nothing to download or install
• Can use with almost any programming language, not just Javascript under node.js
• No license
• Easy to test in a web browser

How to use

Just make an HTTP or HTTPS request to isodd.dev with any string or number: https://isodd.dev/43 or https://isodd.dev/848784. Then examine the HTTP result code and/or the message returned in the response body. isodd.dev is fully RESTful and idempotent, so your code can make as many asynchronous calls as it needs to.

The response gets communicated back with clear HTTP response code:

200 odd
409 even
405 not an integer
400 not a number

That’s it.

Technology

The back-end is Rust and Elixir using AWS lambda, and the Blockchain. I used vim on Ubuntu Linux, ChromeOS, git, and GitHub to develop isodd.dev. I plan to make the code available once I decide on the appropriate open-source license.

Is this free?

It is for now. I’m looking into how to monetize this service. For now enjoy the free and ad-free service.

Wait – there’s more!

During initial testing with early adopters I decided to make a companion service, iseven.dev. It works exactly like isodd.dev only for determining if a number is even. Of course you can just as easily do this with isodd.dev but why not use the best tool for the job?

Bugs and feature requests

I’ll have a support forum up soon. Please post bugs and feature requests in the comments until then.

Enjoy!

tl;dr I can do all of my programming and system admin work with an iPad Pro because I work on remote servers over ssh, and I prefer using the Unix/Linux command line.

A few months ago I bought an iPad Pro 10.5” with Apple Smart Keyboard to try to use it as my main work computer. I had been looking at the iPad Pro for a while, going back and forth between the two sizes and comparing to the 12” MacBook. I used a series of MacBooks until 2015 when I switched to a Chromebook Pixel LS, which I still use (best laptop I’ve ever used). I thought if I can use a Chromebook full-time maybe I can use an iPad.

The iPad Pro appealed to me because of size and weight, long battery life (better than a full day for my normal use), high-quality display (I have old eyes), security, and the many iOS applications available.

I’ve been able to use the iPad Pro for all of my work, and I’ve started to prefer it, but using an iPad meant changing my workflow and making some compromises.

Whether or not you can use an iPad Pro most or all of the time depends a lot on what kind of work you do. I won’t say it will work for everyone, but it works for me because I have fairly simple requirements. I live mainly in Thailand but travel quite a bit, but I work for companies in the US. I mainly do remote programming (working on legacy software with some new development) on web sites, with a lot of system administration work managing servers. The systems I work on either can’t run locally on a laptop or desktop, or getting them to run in that environment would take too much work. Since I usually have four or five active projects at a time trying to run them all locally takes up too much time, and I risk getting things working locally that don’t work when deployed to the live server. I used to run web sites locally on my last MacBook, and then did that for a while on the Pixel (using Crouton to run a full Linux distro), but too many systems I work on just don’t run on a laptop.

My clients normally have their code running on a cloud server (AWS, Rackspace, Digital Ocean, etc.) under some flavor of Linux. If they don’t have their code under source code control (git or subversion) when I take them on I set that up with a private GitHub repo. If they don’t have a development/staging environment I set that up too, usually on a separate server that duplicates their production environment with some features turned off (e.g. outgoing emails, credit card transactions).

How I work

I use ssh to access the servers and work from the command line. I have decades of experience with Unix/Linux and command line tools (grep/ack, screen/tmux, etc.) and editor (vim) so I feel right at home working in a terminal and shell. I have tried cloud-based IDEs like Cloud9 and CodeAnywhere. They work but can get very slow and hog up resources (because they run in JavaScript). Cloud9 works great on my Chromebook but not so well on iOS — I stopped using it shortly after I switched to the iPad. I don’t really like IDEs but I think cloud-based IDEs like Cloud9 (owned by Amazon) have a lot of potential and I expect they will continue to improve. I use repl.it for testing out bits of code, and that service has started to add IDE-like features and even web application hosting.

My work time breaks down like this, on average:

• 50% communicating with clients, translating their requirements into actionable, incremental tasks, researching solutions.
• 25% reading code, my own, legacy code I inherited, and code from programmers I’ve hired.
• 25% writing and testing code.

The main tools I use on the iPad Pro:

• Inbox (GMail client)
• Skype and Hangouts
• Slack
• Blink (ssh client that support mosh, a real timesaver)
• Working Copy (local copies of git repositories)
• CodeHub (GitHub client)
• Textastic (code editor with Working Copy and sftp)
• Dash (offline documentation)
• Google Drive, Sheets, and Docs
• Google Keep, for notes, to-do lists, saving web sites

I have written on my web site about the importance of communicating with clients and keeping work visible. I mainly use email for that, with occasional phone calls. For larger projects I try to get my clients to use GitHub issues, but my clients tend to prefer email, Slack or chat (Hangouts or Skype), and phone calls, and I don’t try to force them to accommodate me — I can copy/paste an email into GitHub in a few seconds.

Where the iPad Pro shines

• Fast. Very fast.
• Great display.
• Smart keyboard surprisingly good, and it resists spills.
• Single app at a time helps me focus on one task (I keep notifications turned off).
• Great battery life, charges fast.
• Light, can easily carry all day in a messenger bag.
• Apple Pencil cool for some things, like annotating PDFs (I don’t draw or do design work).

Where the iPad Pro gets clunky for programming

• I really want a trackpad (or mouse) sometimes. Touchscreen, even with the Pencil, doesn’t have the fine resolution I’m used to.
• Awkward text selection. If I didn’t mainly use vim over ssh I would have given up on iOS for this reason alone.
• Password management not as smooth as OSX or ChromeOS (I use LastPass).
• Multitasking not as good as laptop OS (but not as distracting either).
• Gesture overload — I can’t remember all of the gestures or get them wrong.
• Auto-correction and auto-completion set globally, sometimes enabled when you don’t want them.
• No escape key on the Smart Keyboard, a minor hassle for vim users (but easy to work around).

Programming workflow

Normally I work on code that lives on a remote Linux server, under git control, and I have ssh/sftp access to the server and to the GitHub repo. With that setup I have a couple of workflows.

First, I can use Blink to connect with mosh or ssh to the server and work from the command line. I normally work in this mode since I’m used to it — I worked mostly from a shell on MacBooks and my Chromebook too. You can find several ssh clients for iOS, they all work. I used to use Prompt from Panic but I like Blink because it supports mosh (more on that later).

Second, I can pull a whole git repo into the iPad using Working Copy and look at the code there, or use the Textastic editor to change code. Then I can commit and push my changes or sync changes over ssh (a new feature in Working Copy that actually hasn’t worked for me yet, but the underlying rsync protocol works so I have confidence). Assuming I have a fairly recent copy of a repo on the iPad I can work offline with Working Copy and Textastic, but obviously I can’t run or test my work. \[I have since found out that Working Copy doesn’t use rsync to sync a local repo with a remote.\]

A note on mosh vs ssh and remote sessions

I don’t want to waste time hunting for and hassling with wifi at hotels or coffee shops so I usually use my phone as a portable hotspot and pay for 4G data wherever I go. Plans with big data allocations don’t cost much in most places, and since I mainly work with text over ssh I don’t use a lot of data anyway. mosh enhances ssh connections so they work better over connections with latency or that cut in and out, keeping me online even if I ditch a coffee shop’s wifi and switch to my phone. I also use screen and tmux on the remote servers to keep multiple terminal sessions running when I disconnect. I can close the iPad and walk away and resume my session later exactly as I left it, even if I switch to my laptop.

I have a low-end GCE server at Google’s Taiwan data center I pay for (practically free), it runs mosh and ssh and I have all of my keys and server connections there. I’ve found that I get better and faster connections going to that server from my iPad (or laptop) with mosh/ssh, and then using ssh to my client’s servers, which run in the US. Kind of like a VPN. I can install whatever tools I want on my own server.

What I miss

Most of all I miss Chrome’s developer tools. I use those for debugging problems with HTTP headers, testing JavaScript/jQuery, and tweaking HTML and CSS. The MIHTool app gives me some of this functionality but it’s not as smooth or powerful as Chrome. I would love to get Safari’s developer tools in iOS.

Postman, for testing and experimenting with APIs. I can do much of the same thing with curl from the command line, but Postman’s collections make API testing almost fun.

Simple image editing. Something like Preview on OSX would work great for me. I don’t do graphics work but I do get images from clients in the wrong size or format. I can do this with the iOS photo editor but it seems like all of the graphics tools for iOS target photo editing, not resizing a logo or banner ad or converting from Illustrator format to PNG.

I use my Chromebook when I need to use the developer tools and test APIs.

iOS development on the iPad?

Despite Apple releasing the excellent Swift Playgrounds learning tool for the iPad, I don’t think it’s possible to develop iOS apps on iOS — you need a Mac. I have read rumors that Apple has iOS development tools underway. Not an issue for me since I don’t write iOS apps, but a dealbreaker for iOS (or Android) developers.

Why not the 12” MacBook?

The iPad Pro with Smart Keyboard weighs almost as much as the 12” MacBook, and they get similar battery life. But the iPad Pro has a faster processor and a better display, and costs hundreds less than a MacBook. I compared the MacBook to the iPad Pro several times and found the MacBook’s keyboard harder to use (surprised me too). Even after the last refresh the MacBook 12” still feels sluggish, whereas the iPad Pro seems very fast – the A10X processor beats the i5. And I already have a laptop, I wanted to try something different.

Acknowledgments

I didn’t think up this workflow on my own, though I was using ssh to remote servers from my Chromebook and MacBooks before I switched to the iPad. Other people have written good accounts of programming on an iPad:

The iPad Pro as main computer for programming — Jannis Hermanns

Coding on an iPad — Marc Hedlund

Can You Write Code on an iPad? — Andrew Brookins

Hey Apple, I Tried The iPad Pro As A Dev Machine. It’s Almost Awesome — Stuart Hall

Fri, 23 Mar 2018 00:00:00 +0000 http://typicalprogrammer.com//working-on-an-ipad-pro-as-my-main-computer http://typicalprogrammer.com//working-on-an-ipad-pro-as-my-main-computerWhat does code readability mean?

Programmers complain about readability and talk about bad code and unclean code, and the difficulties they run into trying to understand and maintain that code. What do we mean by readability? What makes code unreadable?

Russian translation of this article available thanks to Vlad Brown:
Что означает читаемость кода?

Reframing the problem

I hear programmers say that some code “is unreadable.” I read articles and books about readability and maintainability. What does that mean? Programmers usually attribute readability, or more often the lack of, to the code itself. But, like beauty, readability lies in the eyes of the beholder. When we say that some code “is unreadable” we actually mean one or more of:

• I can’t read the code because I don’t have sufficient experience or expertise (with the language or domain).
• I haven’t spent enough time trying to read and understand the code (“it’s not obvious” or “it’s not intuitive”).
• I don’t have much interest in understanding this code, I prefer to rewrite it in my own style.
• The code offends my sense of aesthetics; I would write it differently.
• The original programmer didn’t know how to write code.
• The code appears to violate some principles or patterns I believe in.

Simply reframing the statement “this code is unreadable” as “I can’t read this code” puts the problem into perspective.

By analogy, plenty of people find reading Homer, Shakespeare, or Nabokov difficult and challenging, but we don’t say “Macbeth is unreadable.” We understand that the problem lies with the reader. We may not have sufficient experience with the language and idioms. We may not have enough historical and cultural context (similar to lacking domain expertise when looking at software). We may not have the patience or desire to invest time learning how to read a challenging book. Wikipedia articles and Cliff’s Notes exist to give tl;dr versions of books to people who can’t or don’t want to read the original. When we observe this tendency in other (non-programming) contexts we may interpret it as laziness or short attention span. When we react this way to code we blame the code and the original programmer.

When I first read Knuth’s The Art Of Computer Programming as a teenaged amateur programmer I found Knuth’s math-heavy analysis of algorithms hard to understand. I didn’t think Knuth doesn’t know how to write, or that his examples and explanations needed refactoring and dumbing-down. I thought I needed to learn more about the math and analytic techniques so I could competently understand the books. As my comfort with the math improved Knuth’s previously unreadable books yielded a lot of great information.

I recently told a programmer friend that, in my experience doing a lot of maintenance work on legacy systems, all code yields to maintenance with enough study. I get more concerned about the time I require to understand the code, and the risks when making changes to it. Good code should yield to understanding, and present fewer risks when making changes. Those qualities come from the skill of the original programmer, and the constraints imposed at the time they wrote the code. The original programmer may have tried to predict future maintenance and write the code in a way they think of as clear and readable, but doing that requires making assumptions and predictions about unknown future requirements, and unknown future maintenance programmers. Programmers can easily get carried away making those assumptions, adding complexity and generality and getting farther away from the requirements, without actually making the code more readable or maintainable.

Programmer bias

Programmers usually think that they should focus on writing code. Reading code, especially someone else’s code, seems like grunt work, a necessary evil, often relegated to junior programmers in maintenance roles. Reading code and figuring it out does not feel like a creative or even productive activity.

I have personally witnessed (more than a few times) professional programmers dismiss working, production code as “unreadable” and “unmaintainable” after looking at it for just a few minutes. Their objections usually come down to aesthetics: “I hate PHP.” “The code uses tabs instead of spaces and looks bad in my editor.” “The code wasn’t written with classes and objects.” They haven’t spent enough time to understand the system, or learning enough about the business domain to read the code in context (like someone unaware of ancient Greek history dismissing The Iliad).

Programmers looking at code they don’t understand or don’t like can find examples of sacred principles and so-called best practices violated: “This breaks the Single-Responsibility Principle.” “This looks like a DRY violation.” “Globals are a code smell.” Then they propose rewriting the system, or maybe rewriting pieces of it in their preferred language, idiom, and style (often erroneously called “refactoring” because that sounds like a technical thing to the customer or boss).

Whenever I have to think to understand what the code is doing, I ask myself if I can refactor the code to make that understanding more immediately apparent. – Martin Fowler.

Any fool can write code that a computer can understand. Good programmers write code that humans can understand. – Martin Fowler

All respect to Martin Fowler, but those quotes illustrate my point. My experience leads me to expect I will “have to think” to understand unfamiliar code. How hard I have to think matters, but I don’t expect to immediately understand non-trivial code at a glance, even with my decades of experience. After I understand the code I may think I can write it more clearly and refactor it, or I may feel like I learned something new from the code and leave it alone. I don’t think code has to yield to understanding at a glance, especially considering the very wide range in skills and experience among programmers. And I don’t call other programmers fools if I don’t immediately understand their code. “Other humans,” or even other programmers, includes far too many people across a vast skill and experience spectrum to make “write code that humans can understand” a meaningful goal. If good authors wrote books that everyone could instantly understand we’d have nothing but The Hungry Caterpillar and Puppy Peek-a-boo on our bookshelves.

Just because people tell you it can’t be done, that doesn’t necessarily mean that it can’t be done. It just means that they can’t do it. – Anders Hejlsberg

That applies just as much when reading code as it does when writing code. Remember that the next time you or someone you work with declares some code “unreadable” and “impossible to maintain.”

The true test of intelligence is not how much we know how to do, but how to behave when we don’t know what to do. – John Holt

Simple vs. dumbing down

This came across my Twitter feed a while back:

Good code is simple. Code reviews are a great way to train teams to write simple code. Don’t be afraid to say “this is hard to understand.” – Eric Elliott

“Good code is simple” doesn’t actually say anything. My many years of programming experience and business domain expertise gives me a very different idea of “simple” than someone with less experience and no domain expertise looking at some code for a few minutes. What we call “simple” depends on our experience, skills, interest, patience, and curiosity. Programmers should say something when they don’t understand code, but rather than saying “this code sucks” they should say “I can’t understand this code – yet.” That puts the focus on the person who struggles to understand rather than on the code. I agree that code reviews improve code quality and team cohesion, but whether that translates to “simple” code depends on the programmers. Programming teams usually converge on common idioms and style, and that make programming together easier, but that convergence doesn’t mean the code will look readable to an outsider looking at it six months later.

Does that mean all programmers should dumb their code down so even beginners with no domain expertise can understand it at a glance? Should we strive to satisfy the Shakespeare for Dummies demographic? When faced with code I don’t understand I first question my own skills and patience, and if I have sufficient motivation (like a paying customer) I will spend time studying the code to improve my ability to understand it. I may have to look at language or framework documentation, or experiment with the code to figure out how it works. When I understand the code I may think that I know a simpler or more clear way to express it, or I may think that the code only presented a challenge to me because I didn’t have the skills or knowledge or right frame of mind. In my experience figuring code out takes significant time and effort, but when I get through that I don’t usually think the code has fatal readability flaws, or that the original programmer didn’t know how to write code.

Controlling complexity is the essence of computer programming. – Brian W. Kernighan

Complexity in programming happens at many levels, from the overall system architecture to the choice of variable names and flow control idioms. With experience and reflection programmers should develop a sense for unnecessary complexity, and learn how to control it in their own work. The presence of apparent complexity does not always mean the code sucks or resists reading; some problems prove complicated to implement and the resulting code will require some study to understand.

Readability performance art

We think of programming as a process of using and writing abstractions. When the abstractions map to the requirements in a reasonably obvious way, the code usually makes more sense. When the abstractions come from gratuitous application of “design patterns” or “best practices” for their own sake, the code gets harder to understand because you have to mentally follow disconnected chains of abstractions: the business requirements and the software implementation.

In the olden days programmers wrote code that few other programmers would see – the other members of their team or group, their mentors, maybe an analyst or project manager. Now we have open source and people posting their code in blogs and in Stack Overflow questions. Programmers, prone to investing their ego in their code, worry about criticism from a lot of people they don’t know. Slurs about competency abound in programmer forums. Code samples out of context get critiqued and subjected to stylistic nit-picking. That process can prove helpful for programmers with thick skins who can interpret criticism as either useful mentoring or useless insults. Less experienced programmers can get misled and brow-beaten. Public code reviews create a kind of programmer performance art, when programmers write code to impress other programmers, or to avoid withering criticism from self-appointed experts.

The key to efficient development is to make interesting new mistakes. – Tom Love

It helps to discover your own mistakes or have someone point them out to you in a helpful way. That’s how we learn to program, and learn how to write better code. I don’t think it helps to write code with the prospect of public humiliation in the back of your mind.

Programming is a creative art form based in logic. Every programmer is different and will code differently. It’s the output that matters. – John Romero

The most important property of a program is whether it accomplishes the intention of its user. – C.A.R. Hoare

Better programming comes through practice, study (from books and other code), and mentoring. It doesn’t come from trying to blindly adhere to rules and dogma and cargo cults you don’t understand or can’t relate to actual code.

How to write readable code

When I started programming back in the mid-1970s, one of my mentors gave me a copy of The Elements Of Programming Style (Kernighan and Plauger, 1974). From that book I learned that some techniques often worked better than others, I learned about language idioms, naming variables and functions, and other stylistic and aesthetic considerations that have stuck with me. From there I read quite a few other books about “good” programming, above the level of style and aesthetics. And I read a lot of code, some I understood and a lot that I didn’t (or didn’t understand right away). Like the English writing classic The Elements Of Style, Kernighan and Plauger focus mainly on coding techniques and style, not on the larger issues of system design, module decomposition, cohesion and coupling, and requirements gathering. The Elements Of Programming Style serves as a starting point, an introduction to developing a style that will make your code expressive, concise, and easier to read. Similarly The Elements Of Style advises “Omit needless words” and cautions writers away from passive voice and too many adjectives.

We build our computer (systems) the way we build our cities: over time, without a plan, on top of ruins. – Ellen Ullman

Well, yes, and we tend to like many of our cities. I agree with Ellen Ullman that software development often follows a more haphazard path than we like to tell ourselves. That just means we can’t reliably predict the future, and we don’t always have the resources or will to tear things down and start over. The genre of programming aphorisms brims with statements like this, which only make sense if we believe in a “right way” to develop software, a set of hard, proven principles that lead to “good” code, and that any code we don’t immediately understand must have inherent bad-ness, or indicates a fool at the keyboard.

Programmers seem to believe in a realm of beautiful, readable, easy-to-maintain code that they haven’t seen or worked with yet. They seem to think that other programmers get the time and support to write perfect, clean, tested code. That mythical realm doesn’t exist. All code baffles and frustrates and offends a significant subset of programmers. All software gets developed under time, budget, management, requirements, and skill constraints that prevent doing anything perfectly. We should keep those constraints and limits in mind when we look at code and immediately conclude the code resists understanding, or that only fools would have produced such software.

No one in the brief history of computing has ever written a piece of perfect software. It’s unlikely that you’ll be the first. – Andy Hunt

How do you learn to write readable code? Like learning to write readable English, you have to read a lot. Spend the time to try to understand code beyond superficial qualities that don’t match your biases and preferences. Emulate code you find especially readable. Read some of the popular well-reviewed books on programming style and code quality to get some benefit from more experienced programmers. Try to describe why code resists easy reading or maintenance in concrete terms, try your alternatives (refactorings), make sure they work, get other programmers to constructively review your code and listen to them.

Thanks to the Programming Wisdom @CodeWisdom Twitter feed for the quotes in this article.

Mon, 08 Jan 2018 00:00:00 +0000 http://typicalprogrammer.com//what-does-code-readability-mean http://typicalprogrammer.com//what-does-code-readability-meanHow to start freelancing and get clients

I get asked how to start freelancing and get clients more than any other question, so I’m going to give away all of the techniques and secrets and gotchas I’ve learned as a freelancer and as a manager/customer hiring freelancers.

I’m not selling anything, there’s no eBook or course to buy. This article is long because there’s no tl;dr version. If you are serious about freelancing you have to put in significant time and effort, there’s no shortcut. I’m writing about freelancing as a programmer and software developer (and sometimes system admin) because that’s what I do, but at least some of this information may be useful to freelancers in other fields.

If you don’t plan to freelance but want to find a full-time or part-time job, or a job that allows you to work remotely, some of this information may apply.

I want to make the distinction between building a freelance practice as a solo freelancer versus freelancing as a temporary employee. Lots of programmers fill slots in development projects and teams as temporary contractors (called 1099 employees in the USA). Not everything in this article applies to the temporary employee jobs. There’s nothing wrong with those jobs and in fact they are a good way to develop skills and work with people you might get work from later. If a company is looking to fill out a team they may only care that you know Ruby on Rails and React, because they have managers and team leads and senior developers who will assign and review your work. My very first programming job, at Nike in 1979, was that kind of project. They didn’t want to bring on 60 employees, most of whom would wash out or leave in a month or two anyway, so they hired a bunch of contractors, put them in a mentored sink or swim environment, and kept those who could produce. Some of this article may be useful for people looking for temporary contracting work, part-time employment, working for contract firms (aka “body shops”), or doing remote temporary jobs. I just don’t want to get a lot of comments like “I went to a bootcamp for six months with no prior programming experience and got a contract job based on that.” I know that happens, nothing wrong with it, but that’s not the kind of freelancing I’m writing about. What I’m describing is often called consulting.

What is freelancing?

Freelancing is a form of self-employment, also called contracting or consulting.

Freelancers are not employees, they contract with clients (also called customers) for specific tasks or projects. Freelancers can have more than one client at a time. In general freelancers are responsible for their own equipment and tools, education, work environment, taxes, health insurance, etc., though the details vary by nationality and country.

Freelance engagements are usually short-term and for some defined set of tasks. Some freelance engagements can go on for a long time, and some freelancers grow into trusted and important roles with their clients. I’ve had freelance jobs last a few days, and I’ve had jobs that have lasted ten years.

How do I get started freelancing?

You need to have marketable skills you can sell: programming, design, writing, system administration, etc. You can learn and develop skills after you’ve started freelancing, but if you are starting out with few or no marketable skills and little practical experience you’re probably going to have a hard time getting started. Most successful freelancers have years of experience.

By marketable skill I mean an ability to do something that has value to your potential clients. Having a degree or knowing a programming language is not a marketable skill in itself as a freelancer (it might be if you are looking for a job as an employee). Clients pay for solutions to business problems and added value. Period. No client will ever tell you they need 2,000 more lines of Javascript by next month. For your clients the best indicators of your skill come down to:

• Someone they trust tells them you are reliable and have the necessary skills.
• You can show that you’ve solved similar problems before.
• You have lots of experience in their business domain, or something related.

Whatever you think of as skills now – programming languages, frameworks, operating systems, etc. are not skills your clients actually pay for. All they care about is how you can help them with whatever problems their business faces. They won’t always present their problem in those terms – it’s part of the job to identify business problems, gather and organize requirements, and come up with solutions.

Clients expect freelancers to produce something of value immediately – you have to hit the ground running. There’s no onboarding or probation period. If you don’t know what you’re doing right away you are wasting the client’s time, and your own, and no one is going to pay you to get up to speed for very long. Be honest with yourself and your actual skills and competency as your clients will perceive you.

The two main requirements – that you have marketable skills and that you can solve actual problems and add value immediately – imply that you should identify some niches to specialize in. “PHP programmer” is not a niche or specialty. Think in terms of what businesses need. Example niches (from my own experience):

• System support for small law offices
• PCI compliance consultant
• WordPress web site scaling and security
• Logistics and inventory control software
• PR1ME/Information minicomputer 2x - 4x performance improvement guaranteed

My first consulting niche, back in the early 1980s, was performance improvement for PR1ME/Information minicomputers. Back then companies usually had one or a few large vendors supplying all of their hardware and software, and they would get tied to a vendor, like PR1ME (long since out of busines). I figured out some disk drive configuration tricks at my full-time job and thought I could sell what I had learned to other companies using PR1ME minicomputers. That niche and the follow-on work we learned about was big enough to support me and my business partner for several years.

Identifying a niche doesn’t mean you are constrained to one narrow field of work or clients. It’s a way to get in the door. Once a client trusts you and you’ve done good work for them they will talk to you about other things you can do for them. Don’t limit your options by focusing too narrowly, and don’t be afraid to work outside of your niche.

The rest is pretty simple. I’m a US citizen so I only know about the nuts and bolts of taxes and so on as an American. It’s up to you to figure out setting up an appropriate business entity (sole proprietor works for most people), your taxes, insurance, work environment, equipment, etc. You will find plenty of articles online, books, companies telling you to set up an LLC or corporation, spend time with complicated bookkeeping and accounting, to retain an attorney to write contracts. You will find seminars and companies selling startup and support services to new businesses. You don’t need any of that stuff to get started – those are problems of scale you don’t have yet. Concentrate on your marketable skills and getting clients at first, worry about looking like a “real business” and hiring accountants and lawyers when you actually have those problems. I’ve been a sole proprietor for over ten years and the only trappings of a business I bother with are my web site and some cards I designed and ordered online.

How do I get marketable skills?

The best way to learn a skill is by doing, preferably in a real-world situation. Programming in Ruby on a real e-commerce system is better than doing side projects with Ruby, or reading a book or taking an online course about Ruby. I read comments from programmers all the time saying they learned a new language in a week from an online course, or asking which books to read to learn more languages. It’s hard to translate that kind of learning into a freelance gig, because the potential client’s problem is not lack of Ruby code, it’s that their e-commerce system is not calculating shipping rates correctly, and it happens to be written in Ruby.

I’m not saying you shouldn’t study and learn new things – I read books and take online courses too. I’m saying that books and classes aren’t sufficient to develop a marketable skill you can confidently sell to clients. The traditional path for freelancers was years of full-time employment. Now there are open-source projects anyone can get involved with, working with more experienced programmers on something with actual users.

I think a lot of programmers – especially inexperienced programmers – have the idea that learning a language or framework is the hard part. Finishing a Udemy course in Ruby on Rails is not 90% of the way there, it’s more like 10%. Mastering any technical skill is hard and takes years of real practice – see Teach Yourself Programming in Ten Years. And even if you have mastered (for example) Ruby on Rails, you still don’t have a skill you can market as a freelancer.

If you can’t list your marketable skills in a way that doesn’t look like a résumé/CV – a list of languages and frameworks and tools – think about the actual work you’ve done and what business value you added, what accomplishments you can use to communicate your skills and expertise, and list those instead.

There’s no shortcut to developing marketable skills. You don’t necessarily have to spend ten years in a cubicle farm, but you do need some actual skills and the ability to present and sell yourself to potential clients. And you have to deliver. If you can deliver no one will care where you went to school or how you learned Ruby. I think programmers and software developers are especially prone to pigeonholing themselves and focusing attention on languages and frameworks. I see résumés/CVs all the time that leave no impression of what business domains the person might know about, or what they’ve actually accomplished. You may be able to land an entry-level job with that but if you try to freelance you’re going to lose to experienced people, and you’re going to disappoint your customers and maybe damage your reputation.

I only want to do sexy fun work

I hear this a lot from programmers. The work is out there but they are holding out for a sexy startup gig, or they only want to do something they think is fun and personally fulfilling. I didn’t have this luxury when I was a young programmer (I started in big enterprises where you didn’t get much choice), and now I’ve outgrown the idea that my job and work have to be the main source of fun and satisfaction in my life. I like solving problems, having happy clients, learning about different businesses, delivering more than I promised, and getting referrals. I don’t really care all that much about working with the language or framework of the month. I mainly do maintenance work on legacy and abandoned code, some of it fairly old, some of it new (developers moved on to something more fun). Plenty of programmers turn up their noses at that kind of work. I’m not going to tell anyone to suck it up and do the work, but I will suggest that as a new freelancer trying to build up a client base, contacts, and skills you might want to take on things that aren’t so fun. Learning to read and work with and fix someone else’s code is a great marketable skill. It’s also easier to measure and show the value you add to your clients. I wrote more about this in The joys of maintenance programming.

I don’t want to push maintenance work on everyone. New development can be more fun, and some old code is terrible to work with. But consider that around 75% of new software development projects fail or never go live, and the process can be stressful and frustrating. I think the percentage of failures is even higher if the team and managers are inexperienced and/or don’t understand the business domain. You are much more likely to end up in conflict with your client with a green fields new development project, and more likely to end up holding the bag with a big unpaid bill. If you are working with a functioning system that has some problems you have a better framework to work with, the tasks will be better-defined, results more immediate. About half of the projects I’ve had in the last ten years were green fields fiascos abandoned by the original developers after conflict over money and schedule with the client. Read The Mythical Man-Month by Fred Brooks for everything you need to know about green fields development.

How do I find clients?

Finding clients is what beginning freelancers fear the most, especially software/technical freelancers. It requires the people skills and sales tactics that many programmers find difficult or distateful. If you can’t talk to people, listen, communicate, and just want to keep your head down in your IDE, freelancing is not for you. The technical stuff is almost never the hard part of a freelancing gig. The hard parts are getting good clients, understanding their problems and requirements, delivering what they want, and building a solid relationship and trust.

At this point almost every business relies on computers, web sites, technology. I will include non-profits since they have the same needs as for-profit businesses. As a sole freelancer or member of a small team you should set your sights on companies that operate at a scale similar to yours. If you have expert specialized skills and a name in your field you can get gigs at big companies and government agencies. Until that happens concentrate on small businesses. You will be dealing directly with the owner or someone who reports to the owner (less red tape and faster decisions). And you can take advantage of the severe shortage of talented programmers/developers.

Big companies and sexy startups can attract more applicants than they need (or at least a lot more than small un-sexy companies). They pay better and have better benefits. With so many people already working for them they can usually address their business problems with their own resources (though big companies and startups will hire freelancers with recognized expertise and specialized skills). For every Twitter and Snapchat there are thousands of small companies that don’t get any acceptable applicants for their job openings. Most smaller businesses can’t afford full-time IT or software development staff, and they couldn’t keep such expensive employees busy, so they already outsource everything. Until you have a reputation and skills that give you your choice of clients and projects, you need to find work that is acceptable, work you can do to hone your skills.

I have two long-term clients in a hot job market who have been trying for over five years to hire a local person full- or part-time. They aren’t sexy or particularly fun places, and they aren’t going to make a programmer rich with equity. They’ve kept me on all this time because they can’t attract qualified people. Like many small businesses and non-profits they have the work and the money to pay for it, but the talented people go to the sexy startups and big employers. I’ve seen the responses they get to their ads, and when you remove the recruiters (small companies can’t afford those fees) and the foreigners needing work permits (small companies can’t afford those either) they are left with people fresh out of school and people who have obvious problems reflected in their job history (personality disorders and even alcohol/drug problems have been apparent in interviews). So they keep me on. This gigantic hole in the job market just keeps getting bigger and bigger, at least in the USA.

Here’s how I get clients, best channels first:

• Referrals from current clients
• Referrals from other freelancers, not necessarily tech freelancers (graphic artists, designers, system admins have all been good sources for me)
• People I know: family, friends, former employers and colleagues, classmates
• People I meet: at meetups, user groups, social events, trade shows, etc.
• Recruiters: they are plugged in to the local job market, but unless you have a relationship they won’t tell you much
• Online presence: blog, articles, answering questions (and not just technical questions)
• Contacting companies directly when I see a problem or read something about them
• Online ads (mainly Craigslist)
• Gig marketplaces and online job boards, i.e. UpWork, Fiverr, etc.

I actually don’t use UpWork, Fiverr, et al. to get clients and I never have, but I included them here because they are the first place new freelancers often look. I do use those marketplaces to hire people to do piecework and small gigs.

I am represented by an agency now, 10X Management, and they are my main source of new projects. Before I was represented by 10X Management I found clients on my own.

New freelancers seem to invert that list and start at the bottom, with online gig marketplaces and job boards, blindly sending in résumés and filling out applications online. They compete with thousands of people all over the world (who can live on $15/day) for crappy piecework jobs that don’t lead to more quality work. If you are auctioning youself online, applying for a job, or sending a résumé to a jobs@ email address you aren’t trying. The competition is always most fierce at the bottom so get out of the kiddie pool and learn how to get clients.

Notice that six out of nine of the sources I listed involve talking to people. You can’t succeed just passively sending out inquiries to ads you see and hoping a client drops in your lap, especially not when you are starting out freelancing. Eventually clients will come to you and you won’t have to do as much work finding them. If you don’t like dealing with people, introducing yourself at a party or meetup, asking people what they do and listening to them, and only hang around with people who are just like you, get out of your shell. Or find a business partner who has people skills. Freelancing is a people business, a big part of it is sales and client relations and communication. Don’t tell yourself you suck at that and give up – learn those skills. Good communications and people skills are the best marketable skills you can have. Even better than knowing AngularJS.

If you don’t have any current clients, start with people you have worked with before, anyone you know who owns a business. Make it your goal to meet one new person every day. Introduce yourself, say “Hi,” sit at someone’s table at a deli during lunch, sit next to someone at a bar. Almost everyone feels shy and hesitant to approach strangers, it’s not just you, and you can train yourself to get more comfortable with it. I was lucky to do sales/marketing support for a computer hardware company for a couple of years, going on sales calls and working trade shows, seeing how the professional sales people do it, and I stopped dismissing soft skills. You don’t have to act insincere or phony, just be yourself. I’m not talking about fake networking or accumulating LinkedIn contacts. Get to know people, expand your circle. You don’t have to hang out with them or invite them to dinner. No one likes to be the victim of an aggressive network builder so don’t try to sell yourself or ask if you can give them your résumé. If you meet someone and think you might want to get to know them better exchange cards or numbers or email addresses and tell them you’d like to get together again sometime.

I’ll add a caveat to the “be yourself” part. When meeting new people or talking to clients, avoid politics, religion, anything too personal or controversial. Learn what not to say. Dress appropriately and don’t look like a slob who lives in a basement. Appearances matter. Learn to make eye contact without being creepy – practice with a more social person if you have to. Learn to listen and not interrupt. Laugh if you want but the guy with great tech skills and poor social skills who shows up to meet a potential client wearing a Metallica t-shirt, interrupting and telling the client what they should be doing, is not going to get the job over someone who can listen, talk, and dress respectfully. When you are meeting a potential client or trying to make new contacts your goal is to make a good and memorable impression, not to bludgeon them with your Python skills.

Always get the name of the decision maker. You don’t want to send “Dear business owner” emails or try to get past the receptionist. When someone tells you about an opportunity, get the name of the person who will make the decision, write it down, don’t misspell or mispronounce it. If you see a job ad figure out the name of the company (sometimes not easy), then Google or call them to find out who is responsible. I’ve called companies and told them I’m with Dell or Microsoft and need to know whom to send an upgrade notice to, just to get a name. Then I call back and ask for that person by name, and if the receptionist asks what it’s about I say I’m returning their call. I learned these somewhat underhanded bypass techniques from professional salespeople. The goal is to find out who is actually doing the hiring and get through to them to set up a quick meeting or phone call. No points for being shy or going through channels.

Another technique I’ve used a couple of times is to see where employees go during lunch. A lot of places have delis or restaurants nearby where groups of employees go, especially on Fridays. You get to see what the dress code is too. Then I go in and ask if they work for the company I’m interested in, tell them I’ve been trying to get through about a job opening I heard about, and ask to join them. I’ve amazed myself at how well this works – people love to talk about themselves and will welcome the chance to tell a stranger about their job and workplace. This works for getting full-time jobs as well. Almost every company has job openings, many of which are never advertised.

What if they don’t want to hire a freelancer?

Every job opening is a freelancing opportunity, no matter what the ad says. When a small-ish company advertises a job opening they are not just trying to fill an empty chair. They are advertising that they don’t have enough people (or that they don’t have anyone in a lot of cases) to deal with some of their business problems. Keep that in mind when you talk to them: they are the buyer, and they want to buy. (Big companies and especially government agencies do advertise and hire to maintain headcount because their budgets are based on staff size.)

Potential clients may tell you that they really want an employee, someone to work in the office, and they aren’t interested in hiring a freelancer. In sales jargon getting over this hump is called overcoming objections. The actual objection is usually that they are worried about taking a risk on someone they can’t control. So you need to reduce the perceived risk and push the need for control (over an employee) to a partner/peer relationship based on mutual benefit and trust.

You can’t overcome the “we want a full-time employee” objection all the time, it’s not worth getting upset over it or trying too hard to change their mind. When I hear this common objection, I first say “I know a lot of \[programmers, web developers, system admins, etc.\]. If you can tell me more about your business requirements and your top pain points I can ask around and maybe send someone your way.” I defuse the freelancer question and get them to tell me about all of the problems they’re having. Then I say “I could help you with some of those things until you find a full-time person. If you want me to have a look at your \[system, code, application\] I’m happy to tell you if I can help and how much time I might need.” Nothing fake or underhanded, just get them to think about problems and costs rather than their perceived need to fill an empty cubicle. Very often having a talk about business problems and pains and offering to help will get you the job. Don’t try to offer specific solutions or get pressured into setting a price at this point, just say you need to learn more and see what you can do. I don’t charge for the time I spend doing this kind of requirements analysis for new clients. I just want to get my foot in the door, and that might require a couple of hours of my time for free.

If that approach doesn’t work, leave your card, tell them you’ll ask some friends who might be looking, and tell the client to give you a call if they need any help while they’re looking for a good fit. You don’t have to talk them out of hiring an employee or sell the wonders of freelancing. You just have to leave the impression that you listened to their problems and needs and that you are willing to help just a little, for free, by asking around and helping out in the meantime.

You should ask around. If a potential client hires someone you referred, and you didn’t try to charge the usual 33% recruiter fee, you now have two friends who owe you a favor: the business owner and the person you sent to the job. Think long-term: freelancing is a relationship business.

Because I have attorneys in my family I learned a long time ago that law firms are totally dependent on computer systems, and that small law firms don’t have full-time IT staff. They either outsource or they have that one guy in the office who can keep a file server and printer running. I once walked around an office plaza full of small law firms and just asked at each one if they needed help with any computer problems. I got three clients that way, one of which I had for a couple of years and made quite a bit of money from because they were unhappy with the company they depended on for support, who was ripping them off.

“We already have someone doing that”

Because small businesses often outsource their software/IT needs to local consultants and support firms, you’ll hear this quite a bit. Just because they have someone already doesn’t mean all of their needs are being met, or that they are happy with the relationship. When a company has a vendor already, the relationship gains inertia over time and it’s hard to get the client to change, even if they are telling you how their current developer or support firm isn’t delivering. There are two opportunities when you hear this. First, without badmouthing their current developer or support firm, ask about their requirements and problems and pains and ask if their current provider can help them with those things. Second, try to find out who they are using. Very often they are using other freelancers or small contract firms, and those companies are potential clients for you. Never badmouth or insult another freelancer to a client or peer. It will come back to bite you. Sure, there are plenty of incompetent and unscrupulous people out there taking advantage of small businesses. You can choose not to work with them. But they aren’t going to go away because you talk them down, and they may have more connections than you do. It also comes across as unprofessional and amateurish. No business owner wants to be told they’ve been taken advantage of (they may already know or suspect if that’s happening), so don’t criticize what they are doing now or how they got into a jam. You can say “If Acme Consulting hasn’t been able to deal with the shipping charge problem maybe I can talk to them about it,” or “Have your guy give me a call if he’s swamped, I can work with him to get you back on schedule.”

You can get a lot of work from other freelancers and contract development/support companies so establish relationships with them. In my former city (Portland) I cold-called some small companies that do contract development and asked if I could come in to talk to them. They all agreed to meet with me (since those companies seem always to be short of people) and I got referrals and overflow work from them. If another freelancer or contract firm sends work to you, don’t make them look bad, and reciprocate by referring work to them when you can.

I’ve described meeting with clients in a formal situation in their office, but actually it’s almost always better to have these conversations in a casual context. People are more likely to talk about the business problems and what they aren’t getting from their current developer or support person away from the office. I’ve had these conversations at bars, restaurants, trade shows, coffee shops, after meetups and user group meetings. In those contexts once you introduce yourself to a stranger and exchange pleasantries they will ask what you do, or you can ask what they do, and you can take the conversation from there. I have more confidence I could get a new client or at least a lead at an upscale cocktail bar on a Friday evening than I could trying to get a meeting at an office.

Recruiters

I hear people complain about recruiters and getting LinkedIn spam and unsolicited calls. Recruiters get a bad rep because (like programming) the field has a low bar to entry and there are plenty of incompetent and unscrupulous recruiters. But not all of them. A good recruiter is plugged in to the job market (they usually work in a city or region, but some specialize in specific skills and business domains). They know who is hiring, and more important why those companies are hiring. Recruiters frequently have exclusive arrangements, which means they are filling unadvertised jobs or keeping their client anonymous. Many companies outsource finding people to recruiters because they don’t want to spend the time, or they don’t know how, to find and attract employees. You might think recruiters only place full-time and part-time employees, but that’s not the case. Recruiters are paid by the employer (or client), usually a flat fee or a percentage of annual salary (30% to 40% is normal, so it’s a high-commission job). Any recruiter who asks a potential employee or contractor for a fee to get placed in a job is almost certainly a scam, so don’t deal with anyone asking for that.

Recruiters will place contractors and freelancers and they will place people who work remotely. They are matchmakers who get a commission from a successful match, the details of the arrangement aren’t a problem as long as they get paid. If a company is willing to pay a recruiter 1/3 starting salary for an employee they are willing to pay an equivalent finder’s fee for a freelancer. So talk to recruiters, ask your friends about recruiters they respect. Find out what jobs the recruiter is having a hard time filling, and why. Recruiters know the salary ranges too, so you can get that information right away. I’ve been placed in multiple contract/freelance jobs by recruiters. If you get the name of a good recruiter call and invite them to lunch and ask questions about the jobs they are working to fill. In my opinion recruiters are a great resource too many people dismiss out of hand. Make it worth their while and they can give you good information and work for you.

It should go without saying, but if a recruiter reveals the name of a potential employer to you, don’t go behind their back to cut them out of the deal. If a potential client (or employer) suggests this don’t do it. It’s unethical and can only work against you in the long run.

Working remotely

If you looked around my web site at all you may have noticed I live in Thailand. All of my clients are in the USA. I’ve met all of my longer-term clients face-to-face and make an effort to have lunch or visit their office when I’m in town. I have some clients I’ve never met, in cities I’ve never been to.

Because personal contact is important when you’re finding and landing clients it’s harder to do it remotely. On top of having to communicate by email or phone (or video, but pretty much everyone over 16 hates video calls) clients will worry about the time difference and what happens if you disappear. If I was looking for new clients today I would go back to the USA and get them in person. Once you have established a reputation and good contacts and clients and other freelancers who send referrals the remote problem more or less goes away, though I still get hesitation about it from potential clients now and then.

As a freelancer your clients don’t control when or where you work (that’s pretty much the IRS definition of an independent 1099 contractor). They may care, though. I only take on clients that I can plausibly work for remotely, which is one reason I don’t do green fields development or join teams of people at startups. I’ve tried both as a remote freelancer and it’s just too difficult (though I know several people who do it). I don’t want to be on a daily standup video call at 7 am every day (did that for a couple of months) or try to spec out complex requirements for a new system when I can’t talk to stakeholders and users in person. Plenty of routine work is possible and easy online – most companies don’t even have physical servers anymore, everything is in the cloud on a virtual server, and that works just as well from Thailand as from Portland.

Communication

Once you land a client the most important thing is prompt and clear (and professional) communication. Almost every new client I get tells me stories about “the last guy” or “our last software development company” that starts with “They wouldn’t answer emails or return phone calls.” Even more than delivering late or buggy code, clients hate feeling ignored or neglected. There’s no reason today with smartphones and cell service everywhere to be out of reach. You don’t have to be on 24 hours a day, but set the expectation with your client that you will respond to emergencies immediately, and routine issues the same day (accounting for the time difference). It’s not their problem you are half-way around the world, or that you are traveling around. Give your clients who need it a phone number they can call you on in an emergency, and answer when they call. Reply to their emails right away, even if you just acknowledge receipt. Communicating and reassuring them that you are still there and working for them is the number one thing you can do to keep clients. If you go off the grid for three days they will be someone else’s client soon.

If you can’t compose sentences and paragraphs and talk to your clients in language that makes sense to them, fix that. Freelancers who have good speaking and writing ability are going to do better in the long run. In a typical software project maybe 20% of what I type is in some programming language, and the rest is in English, communicating with my client. Sloppy spelling and grammar, amateurish and unprofessional writing, terse two-word responses don’t give a good impression. There’s no excuse for an adult to talk or write like a teenager, so don’t. Take yourself, your skills, and your clients seriously if you want to make a career as a freelancer.

I have been a hiring manager and done résumé screening and interviewing many times in my career (of 40 years). A résumé or cover letter with just one typo or grammatical error goes into the trash immediately almost every time. A dismissive, snide, condescending email to or phone conversation with a client is the beginning of the end of that relationship, and any referrals you may have hoped for.

Competition

There’s more software development work available than people who can do it, so competition is not usually a problem if you position yourself and your marketable skills and learn how to sell yourself. You will compete with employees, contract firms, and other freelancers. If another freelancer gets the job ask the client to send your contact info to the successful bidder telling them you’re available for overflow work – turn the loss into an opportunity and be gracious about it. Don’t whine or argue with the client – if you weren’t the first pick make sure you’re the second pick, because there’s a good chance that whomever they hired isn’t going to work out. When that happens the client will remember you for putting their needs above your ego.

I mentioned earlier that you don’t want to compete for low-end, low-price jobs on gig/piecework marketplaces, and you don’t want to churn out applications and wait for replies. You don’t want to bank on occasionally being the one person chosen out of a few hundred. You are succeeding when you are the only person considered for the job.

Rates and prices

How many times have you read or heard someone ask how much they should charge for their time or for a project? As a freelancer you need to know the market rates. They will fall into a range, probably a fairly wide range. Ignore the lowball prices on UpWork and other gig/piecework marketplaces. Find out what freelancers in your business and region are charging. How do you find that out? You ask them. And you ask potential clients what they are paying (they may not tell you an accurate number but you’ll learn something). If you don’t know any other freelancers join groups online and in person and ask. Just ask for a range. Not “How much do you charge,” but “What’s the range for freelance PHP developers in this area?” I’ve never found other freelancers particularly coy about this as long as you aren’t trying to pin them down. You can also just call contract and freelance firms in your area (because prices can vary widely in different cities and regions) and tell them you are inquiring for a potential client and need some numbers. I have never had a problem figuring out what the normal rates are, I don’t get why people go online saying they want to freelance and then have no idea how much to charge. It’s basic market research to identify your potential customers and determine a price for your product or service.

It’s easier to find out how much full-time jobs pay because salaries are advertised online and by recruiters. Figure 2,000 work hours a year, and the employer has some overhead for employee taxes and insurance. Take a salary range, multiply the low and high by 1.3 (to account for those taxes and insurance), then divide by 2000 to get an hourly rate. Freelancers are generally (not always) paid more than equivalent full-time employees, so it’s helpful to know what those employees are making.

Rates can vary across a broad range, and lots of projects are bid at a fixed fee and account for multiple developers on the project. It shouldn’t be that hard to get a range, even if it’s a big range. For example, ten years ago in Portland, where I used to live, freelance programmers were getting between $25/hr and $150/hr, and the median seemed to be $60 - $90. It’s higher in Silicon Valley, lower in cities that aren’t tech hubs or desirable places to live. You need to know the range so you aren’t quoting crazy low or high numbers to clients.

It may be tempting to lowball to get the job but that works against you a couple of ways. You get locked into a low rate and it’s hard to move it up with that client. And you send the message that you aren’t worth more money. I don’t even try to compete on gig/piecework sites where the rates go as low as $5/hr. I have fairly high (not crazy) rates, and when I get pushback from a potential client (“I’ve seen people who do the same thing for half that rate”) I tell them, politely, that I have seen that too, and if they want to go with someone else that’s fine, but I can’t work for that rate since I have other clients who will pay my usual rate. I tell them I’m available if they need me and leave it at that. Several times when this has happened I’ve received a call or email a month or two later from the client, regretting going with the lowball price. If you set a high but reasonable price for your services and you deliver your client may brag about the awesome high-end freelancer she found to people she knows. Don’t underestimate the power of prestige implied by a premium price.

I never agree to low starting rates that will increase if I deliver or anything like that. I don’t work for equity. I would if I fell into something I really believed in, but 75% of startups go broke and the equity is worthless. I communicate confidence with my abilities and experience, I tell clients I don’t charge at all if I can’t solve the problem, and stand firm on the rate.

In another article on my site, How to work with freelance software developers, I’ve written more about rates and why you should estimate and charge for deliverables rather than hours. I’ve also described retainer arrangments, where the client pays a fixed amount every month for a block of time – turning an unpredictable variable cost into a predictable fixed cost.

Do what you are good at, outsource the rest

None of us can be great at everything, and you should focus your freelance practice on your best marketable skills. Upgrade those skills or learn new skills to increase your client base and income. I wouldn’t take on a project I planned to completely outsource, but I do outsource bits and pieces and small tasks. This is where the gig/piecework sites like UpWork and Fiverr can come in handy. For example I don’t do graphics or illustration – anything beyond simple cropping and resizing I hire out. I am not a WordPress expert (mostly for lack of patience with it), though I can make WordPress do almost anything. I hire out some WordPress work because there are people charging 1/4th what I charge who are faster at setting up sliders or moving WP sites to new hosting providers than I am. You can pocket the difference, but don’t lie to your clients about outsourcing. Sometimes I pass the UpWork/Fiverr fees through to the client (for language translation, converting Word to text/HTML, graphics work), sometimes I mark it up as if I did it. Don’t give out passwords or secure access to your client’s systems or servers to people you outsource to – I’ve seen that happen, and I’ve seen client web sites held hostage by scammers trolling the gig sites. Keep your client’s best interests in mind.

Clients will sometimes drop something on you that you can’t or don’t want to do. Remember that you are solving their business problems, and you can charge them for whatever they need done, whether it’s your specialty or not. You learn new skills and more about the business by doing. If you tell your clients what you will or will not do, or what languages or tools you will and will not use, you are putting handcuffs on yourself and telling your client to find someone else who will address their needs. Last year a client needed an online course rewritten from scratch and translated to Spanish. Not my thing, but I found a freelance writer friend who could write the material, and hired someone on Fiverr to do the translations. I’ve had clients apologize: “I’m sorry, we have this old Perl code no one understands but it’s running part of our call center queuing system. Can you look at it?” It’s not going to kill me to look and figure it out. Don’t be afraid to get your hands dirty with stuff you might not choose to do. If a client wants to pay my rate for me to debug Excel macros, fine. The more things you are doing for your clients the more valuable (and indispensable) you make yourself. That pays off in work and referrals.

Freelancers sometimes start to fall behind and even burn out from taking on too much work. New freelancers may wish they had this problem, but if you are any good it will happen faster than you think. You will worry about not finding the next client and start committing more time than you have. I deal with this a couple of ways. I’m picky now about which clients and projects I take, so I say no more than I used to. Or I try to refer them to someone else. If I like the client and the project and see a long-term possibility I will try to reduce the scope: “I can’t fix all of those problems this month, but if you list them in order by how much they are costing you I can knock out one or two fairly fast.” Clients understand if you’re busy, but they don’t like when you make a promise and can’t keep it. Under promise and over deliver.

Your workplace, equipment, organizing, etc.

Everyone has their own preferences for working. Some people have a separate office area in their home, set regular hours, and have systems for tracking time and managing their day. Others (like me) are more lax about it. I don’t have a schedule or regular hours. Some days I don’t work at all, some days I may work 10 hours. I schedule phone calls if I can but if a client has to call me I take their call. I don’t need any special inspiration or motivators to get my work done. I can solve quite a few issues on my phone and by email – not everything is a coding problem. I automate what I can, like system monitoring, upgrades, and routine tasks my clients have been doing manually and dropped on me.

Final thoughts

I love freelancing. I can’t imagine going back to work in a cubicle farm nine-to-five. I like the freedom of going where I want when I feel like it. I won’t say anyone can do it, because not everyone can. But if you focus on marketable skills, adding business value, and communicating with your clients (and people in general) you can succeed. You don’t have to be a wizard at every (or any) programming language or framework. Most companies don’t need (or want) wizards and ninjas and rockstars. They just need someone who will care about their business and work with them to make it better.

I’m distilling what I’ve learned from almost 40 years of job hunting and interviewing, both as a candidate and as an interviewer. I have experience with programming jobs both in and out of the tech industry, and 10 years freelancing.

My experience ranges from big companies to startups. Big employers will have a more defined process (and it will prove harder to bypass). Smaller companies and startups generally follow a less formal hiring process.

Starting the job search

Posting résumés (or CVs) online and sending out lots of emails may help with the numbers game, and if you have in-demand skills it can’t hurt to get your name out. In most cases a more targeted approach works better.

• Identify companies you want to work for.
• Learn as much as you can about them.
• Use your contacts to give you leads.
• Work with recruiters if necessary, but only with someone you feel comfortable with.
• If you live physically close to the company try to meet people who work there. Go to local meetups and user’s group meetings employees might attend.
• Ask all of your friends and professional contacts if they know someone at the company. I’ve hung around offices at lunchtime and approached people at nearby delis, asking if they work at the target company, and if I can join them. Don’t stalk employees, but there’s nothing wrong with asking people about the place they work – people usually like talking about themselves.

Résumé and cover letter

A good résumé targets the specific job you want to apply for. Untargeted, generic résumés sent out with no cover letter, or a form letter sent to no one specific, are more likely to get screened out. A résumé serves one purpose: getting an interview. Call out your relevant skills and accomplishments and play down the rest. Get to the point, focus on what you offer, and leave out anything that provides a reason to put your résumé in the discard pile.

Proofread, edit, proofread again, then get someone objective (not your spouse or parent or best friend) with good writing skills to proofread again. Spelling and grammar errors leave a bad impression. If you don’t speak the language natively get help from someone who does. I can’t emphasize this enough – many résumés go straight to the waste basket because of seemingly minor errors, the impression of poor language mastery, or even an unprofessional or disrespectful tone. Use clear, correct, cliché-free language, an active voice, avoid jargon, and no obscenities (yes, I have seen that more than once). Complain later about how grammar and spelling shouldn’t matter in life, they do in this context.

• Keep it brief: a résumé/CV highlights your skills, accomplishments, and experience. Don’t write your autobiography.
• Readers understand that you sent the résumé because you want a job, so skip the Objective or Goal paragraph. Leave out your desires, wishes, and dreams too. No one cares and you can easily give the wrong impression.
• List all technical skills, even those you don’t have professional experience with: languages, frameworks, tools, etc. You need to do this to get through keyword screening software.
• Be careful listing startups, don’t give the impression you are just looking for a temporary paycheck and a place to work on your own company. And don’t call yourself CEO or CTO if the company is you and a friend, especially if you are 22 years old.
• Limit the Education section to your degree, year, and school. Your class rank, GPA, classes attended, school projects, clubs, etc. don’t matter unless you just graduated or you are applying for an academic job (which I can’t help you with).
• Translate titles to meaningful descriptions. Unless you used a sword or toured with Van Halen don’t call yourself a ninja or rockstar. If you worked in a job with unusual titles, or have military experience, explain what you actually did in terms relevant to the job.
• Don’t include references, those come later, and it’s assumed you will provide them on request.
• Don’t include hobbies, jokes, quirks, or anything personal or potentially controversial like mentions of your religious faith, your marital status, your children, or that you love dogs and can only eat gluten-free food.
• Try to find out the name of the hiring manager. Don’t shy away from sending directly to a manager – there’s no rule that you must go through HR or send to a jobs@ email address. Call the company and ask for the name of the manager and get them to spell it for you. To whom it may concern should be a last resort. This is where a good recruiter can offer a lot of value – getting you in front of the hiring manager.
• Keep the cover letter brief, get to the point, don’t ramble – everyone knows why you sent the letter and résumé. If you don’t know what a professional cover letter looks like, find examples.
• If your skills don’t align exactly with the job use the cover letter to connect what you have done and can do with the job requirements. If the job requires Ruby and Rails but you have two years Python and Django experience point out the similarities.
• If you don’t have a job, explain your lack of employment in the cover letter. Stay honest. Acceptable explanations for unemployment include: studying to upgrade skills, taking a break to travel, doing freelance work or personal projects.
• Make sure you put your contact information on the résumé and in the cover letter. Include a street or mailing address, a phone number (that you will answer), and an email address you will read (and not something offensive or immature like [email protected]).
• If you send by email send the cover letter in the text of the email and attach your résumé as a PDF. Don’t send MS Word documents unless the company specifically asks for them.
• If you send by postal mail use a decent-quality white paper, not parchment or something weird.
• Include links to your portfolio or examples or work you want to show off. Don’t send attachments – they will get lost, or even worse set off malware scanners.

Never lie or exaggerate in your cover letter, résumé, or during an interview. Don’t omit anything important (but don’t list everything you’ve ever done either). Getting caught in even a slight exaggeration will make you look dishonest and will probably disqualify you.

I found this excellent article on putting a résumé together, with specific advice and explanations: How To Write A High-Impact Resume. I have no relationship to the company or their products.

Interviews

Professional and technical jobs usually require multiple interviews. You may not get to the actual decision maker until you’ve been screened once or twice. You should call HR to ask about the interview process, that’s one thing they can help you with. While you have HR on the phone ask about benefits, vacation policies, background checks, dress code, etc. because you shouldn’t be asking those things in an actual interview.

You have to approach interviewing as a performance. You get a short time to show that you fit the company and the team, and that you can do the work and add value. People make unconscious judgments within a few seconds of meeting you, and until they decide they want you they will look for reasons to pass. Remember that everyone invests time in the interview process, not just you, so respect that.

If you don’t feel comfortable speaking to strangers, or to a group, or if you lock up and mumble or talk too much, get help from a coach or join a group like Toastmasters to overcome your fears. Most people get nervous in interviews and when speaking to groups, don’t worry that you have a special problem. Practice until you can present yourself as confident, friendly, and communicative. Just as résumés go into the trash because of spelling errors, interviews go bad in the first few minutes because of poor communication and interpersonal skills.

• Get there on time, obviously. No excuses. Better to arrive early and sit in the parking lot for 30 minutes than show up late blaming traffic.
• For phone or video interviews get the technology working in advance. Do a dry run. Go somewhere with decent internet if you can’t use Skype or whatever from home. And don’t have distracting activity and noise in the background.
• Make eye contact, mirror body language, smile, engage the interviewer(s).
• You have one goal for the interview: to leave a positive impression that you can fit in and do the job. Concentrate on that.
• Listen to what the interviewers say and respond. Don’t just talk about yourself.
• Don’t ask about salary/pay, benefits, vacation, what hours you will have to work, etc. during the interview. You should already have that information from HR or the job posting. If the interviewer brings up pay (“How much are you looking to make?”) try to deflect that politely.
• Have your references ready, print a few copies. If you aren’t asked, offer to leave those at the end of the interview.
• Dress appropriately. Wear clothes you feel comfortable in, look neat and well-groomed. As a rule of thumb, dress like other people in the company, only a tiny bit nicer. If you aren’t sure about dress code or what people wear to work park outside at lunch or at 5:00 pm and watch. Call and ask HR if the company has a dress code. If you show up in an ill-fitting suit you wore once to a wedding you will be uncomfortable and probably out of place. If you show up in an AC/DC t-shirt with a stud in your tongue (yes, actually happened) I have no sympathy.
• Interviews are not the time to let your personality run loose or display your piercings. Qualities your friends find charming and quirky may be interpreted as weird, unprofessional, or affectations by people who don’t know you. Always err on the side of caution because you don’t know how strangers will react. In this context you need to care.
• Ask questions about the company, the job, the people.
• Interviews are just as much about selling the job to you as you selling yourself, so make them persuade you that you want to work there. Be careful not to get demanding or critical, but don’t forget that they are looking to hire someone. With experience you can shift the interview in your favor getting them to sell the job, but you have to be subtle about it.
• Don’t ask about requirements you might have, like a standing desk, a special chair, that you will only use Linux, or that you won’t answer emails on weekends. You can negotiate those demands after you get an offer. Don’t give the impression that you will be a fussy, demanding, or high-maintenance hire.
• Be careful asking about working from home or remotely. If that’s a deal-breaker for you then you should have researched it beforehand. Mentioning that you want to work remotely in a room full of people who warm a cubicle every day from 9 to 5 won’t go over well.
• Answer questions and try to establish back-and-forth conversation. Don’t talk too much, go off on tangents, or try to joke around.
• If you don’t know something or can’t solve a problem, say so, ask for some guidance. People generally prefer helping than watching you flail around out of your depth.
• Never bad-mouth a previous job or employer.
• Talk about yourself as if you already work there, but don’t tell the interviewers how you would do everything better than they do.

Follow up

Politeness requires a follow-up email or mailed note thanking the interviewer for their time. Some people find this formal and unnecessary, but it can’t hurt. Don’t try to apply pressure to get a decision from them. Don’t try to explain or undo any missteps or mistakes you made in the interview. Stay friendly and polite no matter how you think your interview went.

Ask for business cards at the end of the interview. Write down everyone’s name, title, and how to pronounce and spell their name if not obvious. If you need to send a follow-up but aren’t sure about a name or spelling call the company’s receptionist or HR and ask – don’t risk getting someone’s name wrong at this point.

Don’t continue to send follow-ups if you don’t get a response. Many companies don’t respond at all if they aren’t going to interview you again or make an offer. Sometimes out of laziness or too much actual work to do, sometimes out of fear of discrimination lawsuits if they say the wrong thing. Don’t take it personally.

References

You should have at least three (but no more than five) professional references who will say good things about you. Supply their name, company name, title/position, and phone number and/or email address. Never give someone’s name and phone number/email address out as a reference unless you have their permission first and know what they will say about you.

Usually employers want to talk to your last two or three managers but it’s OK to use a manager in a different department, or a team lead, or a co-worker if you have to. Unless you are young and applying for a first or second job you should not use family, personal friends, or teachers as references.

In general prospective employers don’t do reference checks unless they are planning to make an offer, so finding out your references were contacted usually means you are going to hear back. Not always – some companies have HR check references, or even outsource that task.

Getting an offer

When you get an offer that’s when you negotiate special requirements you may have. An offer only means the company decided they will take a chance on you, so don’t get carried away. This is the time to ask for a standing desk or to tell them you need three weeks off before you start because you booked a trip to Thailand two months ago. This is also the time to bring up working from home or remotely, perhaps after a period proving yourself and integrating with the team on-site. It’s easier to transition to remote work after you establish yourself at the company than to try doing that from the start.

If you got a verbal offer ask for a written offer describing the position, salary, and start date. An offer opens the negotiation, so if you don’t like the salary or terms make a counter-offer. You should know the salary range by this time, so don’t make a ridiculous counter-offer, you can still blow it at this stage. Now you have some leverage and the company has invested time and effort into hiring you, so they will probably meet you at least half-way. If you make a counter-offer and they accept, or you agree on a compromise, you should accept the offer immediately to show good faith.

Conclusion

It goes without saying that I based this article on my own experience and opinions. You will have different experiences, and different circumstances call for adjusting and adapting. You can and should read other articles and books about job hunting and interviewing.

I have been freelancing as a remote developer for over ten years. I specialize in legacy systems and taking over abandoned and unfinished software projects. I see the problems customers have hiring and working with freelance developers. Here’s my advice for avoiding some of the potholes.

This article is about freelance software developers, programmers, and system administrators, because that’s where my experience is. Some of this applies to freelance designers, writers, marketing consultants, etc.

I work through 10X Management, a talent agency for developers and technology professionals. I am available to consult with customers to help them find, hire, and manage freelance developers, and to define and prioritize requirements and measure progress.

Interviewing, hiring, and managing

I’m not going to get into step-by-step instructions regarding the nuts-and-bolts of finding, hiring, negotiating with, and paying freelancers, writing contracts, tax laws, etc. If you aren’t confident that you can evaluate and manage a freelance developer you should use an agency or hire a consultant who has that experience. Software development is expensive and takes time, managing programmers is hard.

Personality conflicts will kill your project and stress your team. Over and over I get involved in situations where the customer and the freelancer clashed personally. If you don’t feel comfortable with the freelancer when you interview them, find someone else. Be aware of your own rough edges and prejudices. Get someone else involved interviewing and managing freelancers. If you won’t be the main person working with the freelancer, get the people in your organization who will be involved in the selection and hiring, and pay attention to any objections they raise. Hiring someone without getting buy-in from the rest of your team gets everyone off on the wrong foot.

Many contract programming shops outsource most of their projects. Be sure you are talking to the people who will be working with you and your staff, not just an account executive or project manager who closes the deal but doesn’t do the work.

Be clear about your expectations, but back off any you can’t justify. For example you may feel strongly that you need someone working on site every day. That requirement will greatly reduce the pool of available freelancers. Is that an absolute requirement (for example, security clearance required, systems that can’t be accessed remotely, specialized hardware, etc.), or is it worry about not having control? Here are some things you should be clear about before you start interviewing freelancers:

• Your budget, total or for the first set of defined tasks.
• Your schedule.
• On-site time required? How much/how often?
• Who will the freelancer report to and work most closely with?
• Is a security clearance or background check required?
• Availability – how fast do you expect phone calls and emails returned? What qualifies as an emergency?
• Who owns the completed product?
• Do you have existing systems that are part of the project?

Before you start interviewing freelancers you need to have some high-level requirements defined. You should be specific without getting into unnecessary detail, and that takes some practice and editing. “Web developer needed for golf equipment retailer fulfilling 10,000+ orders per month, must interface with existing back-end system” is a lot better than “PHP and MySQL programmers with 3+ years experience.” Try to be detailed enough to attract people who will be a good match for your needs, but not so specific that you eliminate too many candidates. Avoid listing technical requirements (languages, tools, years of experience) unless they are absolutely necessary for the job, and you are able to evaluate the freelancer’s competence.

Check references and reputation

When you find a freelancer you like ask for examples of similar projects and references you can contact. Almost no one is going to put you in touch with their unhappy customers, but you should be able to get an idea of the freelancer’s history and reputation by talking to their previous customers and checking them out online.

A nice online portfolio, blog, customer testimonials, and a github profile are all things that a professional developer should have, but those are not sufficient to base a hiring decision on. Has the freelancer worked on projects of similar scale and scope? Online study assignments and side projects indicate enthusiasm but not necessarily professional skills or ability to deliver, so set your expectations (and payment) accordingly.

Business domain experience is at least as important as technical skills

Look for experience in your business domain, or a similar domain, and value that experience as much as checklists of technical skills. You probably are not qualified to determine how proficient someone is with PHP or Ruby. You can judge how much they know about your business domain, or a related domain. For example if you are looking for someone to work on an e-commerce site talk about inventory management, shipping charges, sales tax, payment processing, customer management. If the freelancer doesn’t understand your business or doesn’t ask questions about how it works your requirements aren’t going to make sense to them. Technical skills alone do not solve business problems.

I have sat in on many interviews where the customer and the developer talk past each other. The customer is trying to describe what they need in terms of business requirements, and the developer is talking about languages, tools, scaling, programming process. This disconnect starts at the beginning of the relationship and will, at best, lead to a false start or two, and at worst to a failed project. If you aren’t communicating with the freelancer or you don’t get the impression that they care about and understand your business needs, address that problem early on or find someone else.

Be alert to signs that the developer is going to use your project as an opportunity to try new programming languages or tools. Unless you want to be on the bleeding edge beta testing open source software insist that the developer use stable, mature tools that everyone else is using. It will be a lot easier for you to find people to work on software that is built with mainstream tools.

Define the tasks and deliverables

After personality fit, the biggest determinant of success or failure is how well you describe the tasks and deliverables. This is actually the hardest part of any software development project, so take some time to get it right. Get professional help if necessary. You need to break your project down into small well-defined tasks with clear deliverables. “Set up e-commerce site for my business” is not a specific requirement. What you want to end up with is a list of measurable tasks, in order, so the freelancer always knows what is next and you always know what has been completed. Every task should have an associated deliverable that clearly describes what “done” means for that task.

Defining requirements and setting priorities is one of the hard problems of software development. There are several techniques for managing requirements. Some people prefer the so-called waterfall or BDUF (big design up front) approach, others advocate agile techniques, where software is development in small steps so gaps in requirements can be discovered early without causing a lot of disruption. It’s important to recognize that most customers and most programmers are not very good at gathering and defining requirements, so be realistic. (It’s also important to recognize that no development process guarantees success.

The best way to protect yourself from fights about missed and vague specifications and changing requirements during development is to start by describing the simplest possible thing that will work. Developers call this the minimum viable product, or MVP. You don’t have to define all your requirements in advance: you only need to define enough tasks to get started. With a list of small well-defined tasks and measurable deliverables you can quickly assess if the developer is working out as expected. You will discover misunderstandings and problems early on, and take action to get things on track or terminate the relationship. If you start with a big list of vague, incomplete requirements and let the developer work in isolation for months you won’t know for a long time if the project is is in the weeds, and the developer won’t know if they are not doing what you expect.

It’s hard to resist doing more than you originally planned when the programmer is telling you how easy it will be to add features, make your system into a platform with APIs, scale to Facebook-level traffic. Don’t go down the path of scope creep, it leads to unfinished projects. Insist that the developer do the simplest possible thing to meet requirements, and don’t let them expand requirements to account for things that aren’t likely to happen.

Communicate clearly and frequently, and don’t get caught up in process

Establish ground rules with the developer regarding communication, and be sure you follow those rules. If you expect the developer to answer phone calls and emails within an hour, you should do the same. I hear customers complain about developers who don’t respond to calls and emails for days or weeks. I also have experience with customers who don’t reply promptly, or at all, to questions I send, or who don’t deliver things they are responsible for on time. If you aren’t answering questions and giving the developer what they need your schedule will slip, and you will frustrate the developer. If you delegate day-to-day management or tasks like approving designs or producing graphics to someone else, make sure those things are getting done.

The developer may propose (or even demand) that you use their process and tools. If that makes sense to you, great. But be wary of developers who have a lot of process and make that your problem. As the customer you should be able to discuss requirements and report bugs in person, over the phone, or in an email – don’t agree to only communicate through online project management or bug reporting tools. Whatever process and tools the developer needs to use to deliver is their internal issue, it should not be forced on you, and you should not be paying for time spent on internal processes that don’t have a clear and measurable benefit to you.

Build incrementally

You should agree with the developer that you need to have something working all the time. Don’t allow for weeks or months of development with no visible results – insist on small steps that build the system up bit by bit. Even if you start with a single web page that shows your logo and some mocked-up functionality, that’s better than having nothing to look at. You should also insist that the developer use source code control from the beginning (Subversion or git), that you have separate development, staging, and production environments, and that you are not using real customer information or credit card numbers for development or testing.

Pay for results, not fixed-fee or hourly

Another reason to work from a prioritized list of small tasks is payment. I’m astonished that customers and freelance developers agree to fixed-fee proposals with significant money and time at stake, and the entire project is described in one or two pages of exhibits tacked on to the contract. Unless you have a history with the freelancer that gives you confidence that they will deliver on time and on budget, or you are buying a mostly off-the-shelf solution with very little integration and customizing, a big fixed-fee contract is probably not a good idea. If you go into a project with a fixed-fee arrangement there’s a good chance you won’t know about schedule or cost overruns until late in the project, and there’s also a good chance you will be presented with costly change orders for every small change. Even experienced professional software project managers can’t anticipate all requirements or estimate tasks reliably. Don’t expect that you or the freelance developer will be the exception.

Paying a freelancer by the hour seems like a better plan when you don’t have experience working together. The problem with paying hourly is that deliverables become by-products. The focus on hours spent and time tracking can consume a lot of energy on both sides and lead to arguments about how much time something took or should have taken. I advise against paying hourly, but if you do pay hourly make sure you put a cap on the number of hours you will pay for.

Ideally you should pay for results: finished tasks and working deliverables. That only works if you defined your tasks and deliverables well enough so the freelancer can estimate the work, and you can determine if the estimate is reasonable. As I mentioned earlier, you don’t have to define the entire project in advance and break all of it down into small measurable tasks before you start; you can do that incrementally as the project proceeds. It’s easier for a web developer to estimate “Home page with our logo, main navigation menu, and content areas blocked out, no functionality” than “Start on e-commerce site, show progress in two weeks.” And it’s easier for you to see if the developer is getting the work done as expected and following instructions.

When I start working with a new customer I ask them to list their top five or ten problems, in order, most painful first. If the list is vague I work with the customer to make it more specific: “Web site slow” becomes “Searches with more than 100 results slow to display.” The goal is to have a list of problems we both understand, so I can work from the list, estimate the time I need to fix the problems, and the customer can determine if I’ve delivered.

I prefer to establish long-term relationships with customers and negotiate a retainer agreement, where the customer pays every month for a block of my time they can use for consulting, new development, enhancements, or maintenance. Once you have established a good working relationship and trust with a freelancer working out a retainer is a good way to insure they are available when you need them, and to keep your software development and maintenance costs predictable.

I mostly work on legacy web applications in PHP + MySQL. Usually the original developer is not available, so I have to figure out the code so I can fix problems or add features. For a long time most of the PHP code I worked on was written in the classic PHP style: URLs for each page, PHP code and HTML (and Javascript) all jumbled together, business logic and presentation mixed in the same file. A few years ago I started to see more MVC-style code based on frameworks like Zend, Symfony, Laravel, CodeIgniter, etc. I thought this was a good thing, and that maintaining PHP code that was based on an MVC framework would be easier. My experience, however, has been just the opposite. The classic PHP style is easier for me to understand and refactor even when it has degraded into spaghetti code with PHP and HTML mixed together. It’s easier to work on classic PHP, even badly-written code, because everything you need to know to follow the request/response flow is in one place and reads top to bottom. By comparison trying to understand the thought process behind an MVC application, with the OOP baggage it usually entails, is an order of magnitude harder, and the MVC/OOP code is not necessarily higher quality, more maintainable, or more reliable.

MVC and web applications

MVC (model-view-controller) is a pattern or style of application design that dates back to 1970s and 80s. The idea is to enforce a separation of concerns, decoupling the user interface from the business logic and data persistence layer. MVC doesn’t require OOP (object-oriented programming), but in practice most MVC frameworks are implemented with classes and objects, because OOP can also support strict separation of concerns. There were MVC-style frameworks for PHP before Rails came along, but looking at the current popular PHP frameworks it’s obvious that Rails has inspired and influenced a lot of them.

MVC dates from the pre-web time of mainframes and dumb terminals, and long-running server processes that could update the user interface at any time. The web’s stateless request/response model meant giving up long-running server processes and dynamic interface updates. It’s common for Java- or .NET-based applications to maintain and coordinate state across requests and user sessions, and it’s now possible for the server to push updates to the client (web browser). That’s not how PHP works, though.

So-called MVC frameworks for web applications generally aren’t strictly MVC. Instead they use a router-handler-template-model style that superficially looks like MVC, but doesn’t have some of the important features of MVC. For example web framework “models” are usually dumb layers over a database to do CRUD (create, read, update, delete) operations. Business logic that probably belongs in the model is usually pushed into the controller, and models usually don’t notify the controller or view when something changes in the model. That doesn’t fit the CRUD paradigm or how request/response PHP applications work.

If you look at PHP MVC frameworks what you find are big class libraries that implement some flavor of MVC. The class libraries don’t model a business domain, so the business logic is forced into the framework’s structure, rather than the framework supporting the business logic. Trying to figure out where the business logic should go—controller? model? view?—is a big source of confusion, with no clear right answer. If the main advantage of using a framework is starting with a structure to hang the business logic on, why is it so hard to figure out where that business logic should go?

My experience with PHP frameworks

I’ve used several PHP frameworks in my own projects, and encountered them in code that I’ve taken on for maintenance and enhancement. I’ve written my own PHP framework that is deployed in a couple of real web applications. In every application I’ve run into the same thing: business logic scattered around at all levels, and difficulty figuring out exactly where something happens. Separation of concerns shouldn’t mean fragmentation of concerns, but that’s often what happens.

Real example: Where exactly should I put the business rule that says my client doesn’t accept American Express cards at checkout? In the template, where there’s a Javascript function to checksum the card number and figure out the card type? The template validates things like required entries and which countries my client does business in—why not which credit cards they accept? Or does this rule properly go in the controller that accepts the card number the user entered? Does obtaining an authorization from the merchant processor through an external service API belong in a controller or a model? What if my client decides they will accept AMEX for purchases over a certain amount? Even simple real-world business logic like this doesn’t always have an obvious right place, or it may have to go in several places for usability reasons (or to avoid submitting transactions that the processor will reject, because there’s a penalty for doing that too often).

Another real example: I inherited a web app that displayed a single page of Instagram and Twitter posts matching a handful of hashtags and authors. The previous developer (whom my client had fired after the application was delivered very late and buggy) had chosen to use the Zend Framework 2 for what seems like a fairly simple task. The code was classic OOP ravioli code: many classes with methods doing very granular operations. It was hard to figure out where anything happened because the overall control flow was not apparent, and state was scattered among many objects that had implicit relationships. All of this was wedged into ZF2 for no clear reason. This was tens of thousands of lines of opaque code that didn’t work. I replaced it with a lighter-weight page that just did what it was supposed to do: call the Instagram and Twitter APIs, provide some simple caching, and generate a page of posts. Adding authors or updating the hashtag list turned into simple one-line changes in a config file. The result was less than 1,000 lines, including comments.

This doesn’t mean that every application based on a PHP framework is poorly written or hard to maintain. It means that I’ve worked on a lot of framework-based applications and I’ve seen the same problems again and again.

Is everyone doing it wrong?

Before I started working on web applications I spent over ten years developing and working on OOP enterprise and educational applications, mostly in C++. When these projects went into the weeds or failed, or turned out to be very hard to maintain and extend, the programmers would always blame it on a poorly-designed class structure. If we (or the original developers) could just model the domain correctly in classes and objects everything would be great. The OOP paradigm was never doubted; it’s just that most programmers apparently are too inexperienced or lazy to do it right.

Read the threads online where someone asks how to implement something “correctly” in their PHP framework. If you put any business logic in the template (or view) you are doing it wrong. If you emit HTML from the controller you are doing it wrong. If the model “knows” about user interactions you are doing it wrong. Do business rules that affect presentation, like shipping options based on order total and destination, go in the presentation layer? Do external services like credit card processing or calculating shipping charges go into the model or into the controller? How do I know when I’m doing it wrong when there’s no agreement on how to do it right?

Programming is subject to fashion and religion, based on opinion, faith, and fear of looking stupid or inexperienced. Pick a programming topic and you can read all day about the right way to do it, or a criticism of a supposedly wrong way. These arguments seldom have any foundation based on facts or studies—they are anecdotal and biased. Programmers can be just as certain that their favorite approach is the one right way as they are about spaces versus tabs in source code, with exactly the same facts to base their arguments on: none. The only time I take the right way/wrong way discussions seriously is when they are about algorithms or relational databases, because there’s an actual theoretical and mathematical foundation to base opinions on.

My return to the “mullet style”

Recently I took on a ground-up development project, something I rarely do. Originally it was a debugging job, but the WordPress-based application was such a mess that I had to tell my client they’d be better off starting over, and I don’t tell clients that very often. I decided to revisit my simple MVC framework and improve it with the latest PHP features. I had a single entry to the application to handle global configuration, class autoloader, security, session management, and request routing. I had models that did actual business logic operations on the database (not just CRUD). I had controllers that would process inputs, update the models, and set up variables for the templates. And I had templates that could only contain simple PHP if/then/else and loops. Referring to models or knowing about business rules implemented elsewhere was not allowed. I tried hard to work within the framework and the rules I had made for myself. I refactored the framework two or three times to implement pure solutions to problems I ran into. I had over 7,000 lines of PHP code, a lot of it framework boilerplate and moving data around to enforce separation. It worked, but I was unhappy with it.

I looked at other frameworks, assuming I was doing it wrong, but my architecture was very close to the other leading frameworks. I got some good ideas but never shook the feeling that I was trying too hard to force the application into the framework, and spending a lot of time enforcing what I thought was the right way (or at least not the very wrong way that would invite abuse on StackOverflow and github).

What finally pushed me to abandon the framework approach was realizing that I was flipping through too many files to follow the flow of control, and that too much of my code was there only to comply with the framework’s separation of concerns. Having business logic scattered around in multiple files and classes is painful because I work on a 13″ laptop with vim. Maybe MVC frameworks are easier to work with on multiple 23″ screens, but on a small screen the buffer changes and screen splits cause too many mental context switches. I wanted to see everything that was happening on a single page, so I could keep it in my head and think through it.

Now I have under 4,000 lines of PHP written in a more readable and, I think, maintainable style. URLs map to individual files (with some simple Apache rewriting to support pretty URLs like /email/inbox mapping to email\_inbox.php). Every PHP file that corresponds to an HTTP request is written in the classic PHP “mullet” style: business in the front, party in the back. Request handling and business logic, including querying/updating the database through the models, is at the top of the file, followed by HTML that can only use variables defined in the same file and a handful of global utility functions. There’s no embedded SQL or database operations—that’s still in the models, and once the boundary is crossed from PHP to HTML there’s no reference to model functions or any embedded blocks of PHP code. It’s now easy to find where every request is handled and to follow it through to the response (HTML output) in a single file. Common PHP functions and HTML code (page headers and footer, style sheets, Javascript) are in separate files. There’s no more boilerplate, and no more long blocks of code that simply copy validated user inputs or a database query result into a structure to pass to a template renderer.

Of course this is all just my opinion, based on my own experience, biases, and work style. If PHP MVC works for you, great, but you may be surprised to find it’s not as robust or maintainable as you think it is. There’s still no silver bullet when it comes to software development.

Postscript: Don’t throw the baby out with the bath water

I’m not advocating writing spaghetti code or ignoring good programming techniques. It’s possible to write good or bad PHP code whether you use a framework or not. Keep in mind that the popular frameworks do some useful things for you that you will have to do yourself if you don’t use a framework. You have to protect your application from common attacks: SQL injection, XSS (cross-site scripting), CSRF (cross-site request forgery), session hijacking, cookie tampering, insecure API endpoints and AJAX functions, etc. Learn about these attacks and use PHP’s built-in functions or a library you trust to protect your application. Treat all inputs as suspect, whether they come from an HTTP request, a cookie, a database query, or an external service. Use a database library such as PDO, never interpolate variables into a SQL statement. Sanitize everything that is output as HTML or JSON. Make sure your web server blocks attempts to access git or svn repositories, or anything else in your document root that shouldn’t be accessed. Build in error reporting and logging from the beginning. Don’t use global variables. Keep it clean and simple, err on the side of caution.

Selected comments

Cindy on 22 March 2015 at 8:58 am

Overarchitecturing makes things a lot harder to understand in my experience too. I thought I was crazy for wanting to remove layers most of the time. I work in ASP.NET MVC these days, and just a controller class, the Razor template and and class to send data between the two makes for code that’s easier to understand and maintain in my opinion.

John on 23 March 2015 at 6:38 am

I enjoyed the 2nd last paragraph before the Postscript. Have you considered to make a diagram of that?

Also I’m astonished when I see all this Java overarchitecturing coming to JavaScript. We seem to go from IE6-days JS to heavy frameworks that go out of control eg. AngularJS.

Regarding changes that affect both client and server-side code, I’d go ideally for a globally scoped config file that is read for both purposes.

Greg Jorgensen on 23 March 2015 at 11:43 am

@John A lot of this applies to Javascript as well, thanks for pointing that out.

I don’t have a diagram or any code samples. I assumed that anyone who has worked with PHP has seen what I call the “classic” style, but maybe there are people new to PHP who have only seen the MVC framework bloat and think that’s the only way to do things.

Over-architecting is an age-old problem in programming, and it’s a problem that programmers mostly bring on themselves. It’s not unusual at all for me to get involved in projects where the programmers spend more time solving technical problems they created, or dealing with excessive complexity in the programming and build environment, than they spend solving business problems.

Tom on 8 April 2015 at 12:47 pm

Great post. I think about this all the time especially the part about where the business logic goes with current MVC frameworks.

Dale on 10 April 2015 at 8:32 am

Finally! I’ve been waiting for an article like this for a long time. I started to dig into MVC world about 2 years ago and I don’t think I’ve ever really understood the hype surrounding it…(im referring to the PHP community) I do agree that the design principles surrounding MVC are important… however, everytime I try out a new framework I find myself spending countless hours researching how to use it / how to extend it or how to modify it and I usually just drop it and move on to the next framework. I sometimes feel that that doing things the “right” way turns into some pretentious rambling just so one can feel like their part of the new “cool kids mvc crowd”. I ask myself countless times “is this really all that necessary?” when trying to use a php framework. “Do I really need to work out all of the Service Container bullsh!t? Where the f\*\*k should I instantiate this class?? Oops I mean ‘service’….” … by the time I figure all of this out I could’ve read the entire encyclopedia brittanica…twice!! Oh and not to mention all the hip cool terminologies that get used… annotations, migrations, depency injection, ioc container, interfaces, service providers…Give me a break! I just want to start building my site now please? ! Anyways… rant over… once again… thank u for the article… a breath of fresh air!

trustradius.com on 3 November 2016 at 8:19 am

I was just thinking this morning that I cannot believe I haven’t seen the last Harry Potter movie yet! I’ve heard good things about it, so I’ll have to watch it soon. I also want fro-yo now after reading this post. I wish that I had a place nearby with dairy-free options other than sorbet.

BobT on 17 April 2015 at 3:08 pm

I couldn’t agree with this more. I am a freelance developer myself, and I have built and maintained several (decent sized) apps from the ground up in PHP (intricate commerce, custom CMS, etc.).

Given many years of experience working with an approach similar to yours while trying to use various MVC frameworks has led me to completely agree with you. I thought I was doing it wrong at times. Turns out I wasn’t.

I much prefer straight-forward, easy to follow and read code to be much more maintainable. It’s solid & secure OO code which is good for the client and good for the developer.

That isn’t to say it’s okay to throw “modern” coding practices out the window, but to use what makes sense where it makes sense.

Thanks for the great post!

Noah on 21 April 2015 at 7:35 am

“Ravioli code” as the opposite extreme of spaghetti code – brilliant. Let me also contribute “lasagna code” – layer after layer after layer after unnecessary layer, once again to support “separation of concerns” as an end rather than a means. And then there’s lasagna made out of layers of ravioli – not some new super unhealthy dish from Cheesecake Factory, but the actual state of a large majority of architectures, for web apps and beyond. (As a former lower-stack Java EE guy I can attest to this.)

That’s just the beginning of what I like about this piece. I have been a silent reader of your blog since late 2013, and this is one of your best.

Greg Jorgensen on 22 April 2015 at 2:00 am

@BobT and @Noah — Thanks!

Karen on 18 May 2015 at 4:22 pm

I have been trying to get my head around all these “new” ideas in programming after decades of coding procedural style in a dozen or so languages, including almost 15 years of writing PHP spaghetti code that I understand. I was told by all the smart people that I not only needed to do OOP, and not only MVC, but should use a framework to force myself to “do it right”. I can see some advantages to the new ideas, but I just haven’t been able to get my brain to think that way naturally – your comment about where the heck to put which piece of business logic rings so true! I thought I was just a hopelessly inept dinosaur who had done too much hand-coding in my formative years, until I read your article. Thank you for courageously challenging the assumptions – it’s a breath of fresh air. It might still be that my main big DB interface should still be refactored in MVC and even a framework, but I don’t feel like a dunce for thinking that the old way is in many respects easier/simpler.

Ahmed M on 8 February 2017 at 1:50 pm

QUOTING: “Now I have under 4,000 lines of PHP written in a more readable and, I think, maintainable style. URLs map to individual files (with some simple Apache rewriting to support pretty URLs like /email/inbox mapping to email\_inbox.php). Every PHP file that corresponds to an HTTP request is written in the classic PHP “mullet” style: business in the front, party in the back. Request handling and business logic, including querying/updating the database through the models, is at the top of the file, followed by HTML that can only use variables defined in the same file and a handful of global utility functions.”

That is a breath of fresh air finding someone as successful and talented as you sharing my same thoughts about php MVC frameworks. My freelance jobs are usually small/medium ones, and I hate being ‘forced’ to do it the ‘correct MVC’ way. I tried CI and Laravel and I genuinely believe that using these frameworks results in more confusion and spaghetti code than the good old, straightforward OO code just as you said.

Thanks for a great post, and a boost of confidence in a time I started to doubt my ability for not being able to get comfortable using a MVC framework.

Juris Trusevičs on 26 March 2017 at 5:56 am

Wow,

I’ve been thinking about all this for 5 years at least and all this time I just hated the idea about being forced to learn Laravel ,Yii or whatever crap they have declared to be the best MVC ever. Today I still don’t care how this Laracrap, Symphony or Ci works, I decided to create my own simple MVC framework without things like ORM, ROUTING and DEPENDENCY INJECTION CONTAINERS. I feel like my “framework” should take less than ~200 lines of code. And if a VIEW code inside a controller makes my happy , without doubt I will put it there. And Routing is Apache/Nginx problem , not mine.

Thanx a lot for your article!

Here’s a list of things I know, or at least know about, as a web developer. I’m sure I’ve left a lot of things out. Web development is a large and complex collection of technologies, tools, languages, protocols, and services. I started programming for the web back in 1995, so I’ve been able to adapt to changes and learn new tools as they were released. If I had to learn web development from scratch today I’m sure it would take me a long time to master even a few of these things.

If you are doing a new project from scratch you’ll get to pick and choose your tools and “stack.” If you are coming into a team that has already made these decisions you’ll have to adapt. If you work with legacy code you’ll have to get familiar with more languages, frameworks, libraries, technologies, and programming styles.

Web applications usually have two big pieces: the front-end code that runs in the browser, and the back-end code that runs on the server.

The front end (browser side)

You’ll need to know HTML, in detail. HTML5 is the latest but you’ll probably have to know how it’s different from HTML4. You also need to know how styles work: CSS, the box model, inline styles, inheritance, specificity, resets, the differences across browsers and browser versions. You will probably need to know about grid systems, and responsive layouts. Most likely you’ll need to learn one or more of the popular front-end frameworks: Bootstrap, Foundation, Toast, Yaml, etc. You may have to know about CSS preprocessors, like Less and Sass.

Almost all modern web applications use Javascript, a lot, so you need to know Javascript. You will probably have to learn jQuery because it’s everywhere. There are other Javascript frameworks and libraries: React, Angular, Ember, Backbone, Prototype, YUI, GWT. You will have to be careful that the Javascript tools play well together, and they work with your HTML/CSS framework. All of the popular frameworks and libraries offer add-on modules and extensions for things like input validation, modal dialogs, drop-down menus, animations, and so on. You will have to learn some of those, too. Javascript manipulates the contents of the web page through a standard abstraction called the Document Object Model (DOM), so you need to know how that works. Browser events such as clicks and keyboard presses can invoke Javascript event handlers. To interact with the back-end server from Javascript you will need to understand AJAX. Once you start using Javascript you will want to learn how to use the browser developer tools, DOM inspector, and debugger.

If you’re lucky you have some design skills, or you are working with a designer who understands the web. Otherwise you’ll be responsible for layout, typography, colors, image formats and optimization, for a start. Customers often don’t understand that a web page is not a Photoshop proof, that they can’t put a 40MB TIFF file on their home page, or why their favorite font is not available to everyone else.

You probably have to know something about the different kinds of devices people use to browse the web and how that affects design and front-end implementation. Desktop and laptop computers, tablets, and phones have obvious and subtle differences. There’s no hovering on most touchscreen devices, for example. High-resolution Retina or 4K displays may give you grief. Mobile users often have slow connections and data rate caps and limits, so you need to think about that and test on all kinds of devices, with wi-fi and cellular connections.

Somewhere along the way you need to understand the HTTP protocol, at least what request/response and statelessness means. You need to understand cookies, browser caching, expiration, content types and MIME. You have to know about mixing content retrieved with HTTP and HTTPS. You will probably need to understand synchronous vs. asynchronous requests.

The back end (server side)

The server stores and serves the HTML and CSS and Javascript files to the browser. A web site that only has HTML, CSS, and Javascript is called a static site. Non-trivial web applications have code running on the server to generate the front-end code dynamically based on user inputs, session state, database queries, and external services. The web server may run one of the general-purpose web servers such as Apache, Nginx, IIS, Tomcat, or it may integrate the server with the application, like NodeJS. There may be multiple web servers handling different types of content: proxy servers, CDNs (content delivery networks).

The web server runs code that may be written in any number of languages, and many web applications use more than one language. PHP is widely used, so is Ruby with the Rails framework, Java, the various .NET languages (C#, VB.net, F#), Javascript with NodeJS. You may run into code written in Perl, C, C++, Go, Erlang, Python, or something else. All of the big server-side programming languages have extensive libraries and components you will encounter or want to use.

Almost all web applications use a database or some kind of datastore accessible by the web server. MySQL, MS SQL Server, PostgreSQL, SQLite, Oracle, DB/2 are all relational database management systems (RDBMSs) that use the relational model and some dialect of the SQL language. You may use a non-relational (so-called NoSQL) database, and there are many of those: MongoDB, CouchDB, Cassandra, Redis, Memcached. Many web applications use more than one database or data storage tool. The database management stuff may be partially hidden behind an ORM provided by your application framework, but it’s likely you will eventually have to understand the details of how your data is stored and retrieved, how data integrity is assured, and how the data is backed up and secured.

External services give you access to all kinds of things: sending and receiving email, credit card and check payment processing, analytics, maps and location services — the list is endless. If your web application uses external services you will have to learn about APIs, REST, authentication, JSON, XML, RPC, SOAP. You may have to use or write a scraper or crawler that extracts data from sites that don’t offer an API. You’ll probably have to work with curl/libcurl at some point when getting your server to talk to an external service.

Besides the languages there are programming techniques, styles, and workflows you will have to understand: object-oriented programming (OOP), maybe functional programming, certainly imperative programming, test-driven development (TDD), Agile, SCRUM, etc. You may need to use a debugger for the server-side code.

The web server has to be hosted somewhere. It may be a server in the office, or co-located at a hosting facility. More likely it will be a shared, dedicated, or managed cloud instance at Amazon, Rackspace, Google, Microsoft, or any number of cloud hosting providers. Hosting decisions can limit your choice of languages and tools, how much control you have over the server configuration, and things like email handling, DNS, bandwidth limits, and so on.

Security and legal/regulatory issues

As soon as your application goes live on the public internet it will be attacked by bots and hackers. You have to understand common vulnerabilities like SQL injection, cross-site scripting (XSS), cross-site request forgery (CSRF), cross-domain request issues, CORS, session hijacking, spambots. If your employer or customer doesn’t have a system administrator you will have to know about firewalls, keypair authentication, and securing your servers. You have to keep up with security threats and patches to the server operating system and every part of your stack.

If you accept credit cards or electronic payments on your site you will be subject to PCI compliance audits. Other types of personal data you collect and store may be subject to other compliance and audit rules (HIPAA, COPPA). Every country your site does business in will have its own legal and regulatory requirements.

Miscellaneous

You will need to master at least one text editor. You will have to know how to get your code uploaded to a server with FTP, SCP, or some other deployment tool. You should be using source code version control: git, subversion, Mercurial. Eventually you will have to learn your way around UNIX, the file system, the shell, tools like find, grep, vim, emacs, nano, ssh, cron.

You will have to understand domain name registration, DNS, probably SSL certificates. You will probably want to monitor your server for crashes and outages and spikes in activity than can signal a denial of service attack or something else wrong.

If your site is available in more than one country, or your customers may not all be English readers, you have to know about internationalization (i8n) and localization and deal with translations and multi-lingual content. You will also have to understand timezones and date/time conversions. Depending on where your users are you may have to deal with country-specific issues, like China blocking most Google contents (goodbye Google-hosted fonts and Javascript libraries).

All of the languages, libraries, frameworks, and tools you choose will be updated and patched frequently, so you have to stay on top of that while making sure updates don’t break your application. Even moderately-complex web applications can have tens or hundreds of dependencies at different layers in the stack. Dependency management and managing versions and updates can turn into a risky and complicated problem all on its own, and of course there are tools to learn for that.

A large number of web sites are based on content-management systems: WordPress, Drupal, etc. You may have to know the advantages and shortcomings of those platforms, along with a lot of the stuff listed above.

If your application get significant traffic and users you will need to think about bandwidth (especially costs), performance, scaling. Decisions that made sense during development or with little traffic can come back to plague you when your site gets loaded.

I was asked today on Twitter about how to find work as a digital nomad. My comments are too long for a Twitter reply and may be interesting to people who don’t follow me on Twitter, so here goes.

How I became location-independent

After almost thirty years as a professional programmer working in offices and cubicle farms I started taking on freelance projects on the side, while I was still working full-time at an office job. I found small web site and database projects through word of mouth, by contacting small businesses with broken web sites, by writing articles on this blog, and by presenting at local user group meetings. I got referrals from programmer friends, web site design firms, and even from recruiters. It didn’t take long to build up a steady stream of work.

About the same time I started concentrating on fixing broken web sites and the underlying code, rather than developing new applications or sites from scratch. I explained why that works for me in my article The joys of maintenance programming. My decision to focus on debugging and maintenance was informed by what I saw as high demand and less competition, my ability to figure out and fix legacy code, and by the increasing difficulty I encountered trying to get hired at startups, which I attribute to my age and unwillingness to work crazy hours for pizza and beer.

I took a contract with a company that had a profitable and almost-working web-based business. They had fired their original developers over a contract dispute. The company needed immediate help fixing and enhancing their application, they didn’t make me work at the office, and that turned into a five-year relationship. With a steady income I was able to live anywhere I could get an internet connection.

I was living in Portland and happy to just work from home, but I wanted to travel more. With my kids growing up and moving out I had fewer reasons to stay in Portland, or to stay anywhere. A little over two years ago I gave up my apartment, sold almost everything, put some stuff in storage, and took off on my motorcycle. I was able to stay with friends and family, used airbnb, and sometimes splurged on hotels. I lived in Palm Springs for a month at a corporate condo no one was using during the baking summer. I went to Belize for a few weeks, and loved it, but internet connectivity is poor there and I didn’t get much work done. I stayed in Las Vegas for a couple of months, taking advantage of cheap hotel rooms and food. I was looking for someplace to go to learn to scuba dive when a Thai bartender suggested Thailand. I did some research online and the next day bought a one-way ticket from San Francisco to Bangkok. I’ve been in SE Asia ever since.

Let me explain the specific suggestions from my Tweet:

Be flexible

Working on your own, whether you freelance or have a regular job, requires both discipline and flexibility. You need to get organized and make sure you meet your commitments. You have to adapt to what your clients (or boss) needs so they don’t worry about you working remotely. That might mean staying in one place for a while to concentrate on a project, and taking phone calls at 3am.

You also have to be flexible about traveling and living in a foreign country. Not everything will go as planned. You may not be able to stay somewhere you like because of visa restrictions. High-speed internet at your hotel may be terrible, and in some places reliable electricity is not a sure thing.

Be flexible about the kind of work you will do, and the work conditions. If you are the guy who only wants to write brand-new Python code and you need three monitors and a special chair you will find the digital nomad lifestyle tough. I work from coffee shops, hotel rooms and lobbies, beds and sofas, lounge chairs on the beach, trains, and airplanes. I have an apartment in Bangkok so I am mainly working from home, but I travel on average two weeks every month. I work on whatever web site or application I can fix, whether it’s decaying PHP 4 or brand-new C++ that went into the weeds.

Be flexible about where you go. You may hear about some great place or read about an expat paradise, but find out that it’s not for you. Maybe the weather sucks, or it’s too expensive, or you don’t make friends. Don’t get stuck. Go somewhere else. The world is a big place and you will find places that are interesting and fun to visit and work from.

Some people can’t adapt when things don’t go their way. Some people are easily overwhelmed by not understanding the local language, immigration bureaucracies, getting money from banks, or missing family and friends. You will meet these people everywhere you go, complaining and worrying instead of adapting and enjoying. Working remotely in a foreign country is not like going on vacation. Be honest with yourself, and start out slow with an escape plan in case you don’t like the location-independent lifestyle.

Line up work in advance

If you have a job already find out if you can work remotely. Start out working from home, then go on a trip somewhere you can work from. If your boss doesn’t have a problem with you visiting friends or family and working remotely it will probably not be a big deal to work from Panama or Thailand for a few weeks.

If you don’t have a job or you do freelance work you don’t need permission to work remotely, but you do need clients who don’t care where you are, and you need to have a reasonably steady stream of work. I get work through an agency now but before I signed up with them I had a good roster of clients who sent work my way.

It will be harder to get clients when you are traveling. You won’t legally be allowed to work in the countries you travel to, and even if you find local work the pay will probably be terrible. You can find work online through freelancer sites, but you’ll be competing with people who can live on a lot less than you can. That’s why I advise lining up work in advance. How you do that depends on the kind of work you do. I wrote an article about freelancing as a programmer: Tips for successful freelancing.

There are agencies and recruiters who specialize in matching freelancers with projects, seek them out and establish a relationship.

My clients are mostly small business and non-profits that don’t have an IT staff, and don’t need full-time programming or IT support. Put yourself out there and start asking people you know about small organizations that need some work done. I’ve even found new clients by sending emails to companies that have obviously broken or old web sites, telling them I can fix their problems.

Get banking set up, and other practical considerations

If you work for American or European companies and get paid in dollars or Euros, and live somewhere with a weaker currency and lower cost of living (like Thailand) you can work fewer hours and maintain your standard of living. Or you can work more hours and build up savings. It’s easy to spend too much on travel — airfare, hotels, etc. get expensive, so plan to stay in one place for a while and then move on. In many countries it’s easy to rent an apartment for a month or two, with furniture, kitchen, and internet service.

Getting paid, paying bills back home, and getting money in a foreign country can get complicated, so sort that out before you go. Pay off your bills if you can, and simplify what you can’t. Set up your bills so you get statements by email or online and arrange online bill payment with your bank. Open a bank account that won’t charge you crazy foreign transaction or ATM fees (I recommend the Charles Schwab Investor Checking account). If your bank offers chip and pin debit cards, get one, though you can still use a magnetic stripe card in most places. Get a backup credit card that doesn’t charge foreign transaction fees. I get paid by some clients by wire transfer or direct deposit to my US bank account, and others pay through PayPal. If you open a PayPal Business account you can accept credit card payments (for a fee) and you can get a PayPal Business debit card that comes in handy. I talk to quite a few travelers and expats who have chronic problems getting money from their clients into their hands, so sort that out before you leave, because it’s close to impossible to open a US or European bank account remotely.

If you are an American citizen you will have to file Federal tax returns. Learn about that, information is free on the IRS web site. Depending on your state of residence you may have to file a state return. If you plan to go nomadic you might want to change your state of legal residence to a state that doesn’t have an income tax.

You will need a mailing address in your home country. I use Traveling Mailbox. For a small monthly fee you get a US mailing address, scanning and online mail management, forwarding (including overseas), depositing checks, and package receiving. Many expats use parents, family, trusted friend for handling mail. You may want to get a lawyer in your home country and give them power of attorney, in case you need to have something signed and notarized. Sending documents back and forth by FedEx is expensive and you will have to go to a consulate or embassy to get notary service.

If your passport is close to expiring, renew it before you leave. Many countries won’t issue a visa or admit you if your passport expires in six months or less. If you need a visa for your destination, or you want to stay longer than a visa on arrival allows, get a visa before you leave. Research visa requirements online at official government sites, and don’t trust guide books or postings in online forums. You don’t want to be denied entry because you don’t have correct and current information.

Check if your health insurance covers you overseas. For Americans the answer is probably no. You can buy a travel insurance policy for catastrophic problems. In most countries paying out of pocket for routine and minor medical or dental procedures is practical because costs are a fraction of what you pay in the US. For long-term travel look into expat medical insurance through companies such as Cigna and BUPA, and pay attention to what is covered and if you can use private hospitals. If you have a medical condition or need medications get your prescriptions, on paper, and ask your doctor about generic or equivalent versions sold overseas.

If you need immunizations for wherever you go get those at a travel clinic or from your doctor, and be sure to get documentation. In some countries, such as Thailand, immunizations are advised but not required for entry, and it’s cheaper to get them once you’re here.

Take photos of everything: passport, visa, prescriptions, credit cards, birth certificate, and other important papers and keep them on your smartphone and backed up somewhere like Dropbox.

Travel light

You can find lots of opinions on what to take, what to leave behind, how to pack, the best luggage, and so on. Everyone has different requirements and preferences. Traveling very light works for me. I don’t want to have to check luggage or carry a huge backpack around in tropical heat and humidity. I don’t want to be a theft target. I’ve been minimalist for a long time and I don’t own a lot of things or get attached to stuff. When I first planned to go nomadic I bought a big 42 liter travel backpack at REI, but when I had it filled up with stuff I found it too heavy and bulky. I returned the big backpack and determined to use the 25 liter Ogio Drifter backpack I used for commuting on my motorcycle. I still use that pack for traveling, though I can no longer fit everything I own into it.

I use a 13″ MacBook Air. It’s light, reliable, and has great battery life. I don’t use a lot of applications: Chrome and Safari, Terminal for remote work, the command line (I’m a long-time Unix user), Vim for editing. I use Google Docs for spreadsheets, drawings, and word processing, with offline editing enabled. I use Parallels for running Windows, mostly for Internet Explorer testing and for clients who have Windows-specific applications. I use LastPass for managing the hundreds of passwords I need for multiple clients. I have StrongVPN for watching Netflix and dealing with the occasional web site that cares that I’m not in the US (bank and credit card sites sometimes do IP geolocation). I carry a USB ethernet adapter and retractable ethernet cable, and an HDMI adapter and cable for plugging into a TV.

I have a Nexus 7 tablet I mainly use for reading books and magazines and playing games. I don’t think of the tablet as an essential piece of gear. I carry a my MacBook power adapter, a small USB charger, a fairly hefty battery backup, and a couple of USB cables. I normally use the camera in my phone, and I have a GoPro camera I use sometimes.

I don’t carry a lot of clothes. Travel clothes can look silly and mark you as a tourist. On the other hand good travel clothes are comfortable, rugged, and easy to wash and dry in a hotel room. Before you leave go to REI or equivalent for some basic items like shorts, a pair of presentable trousers or a dress, walking shoes, a couple of t-shirts, a long-sleeved casual shirt or blouse, and a hat or head covering. It depends on where you go but in SE Asia it’s hot, humid, and rains frequently. I recommend Icebreaker shirts, ExOfficio underwear and clothing, and REI’s house brand. Jeans are heavy, take a long time to dry, and I think they are too hot for tropical climates. Shoes are important, bad shoes will make you miserable. After some trial and error I’ve settled on Keen Clearwater CNX sandals because they’re comfortable and waterproof (and sold in Bangkok). You can buy t-shirts, shorts, swim trunks, flip-flops anywhere and in most places dressing like the locals is cheap and you will blend in better.

I carry an inflatable travel pillow, a travel laundry line for hanging clothes up to dry, a small flashlight, small quick-dry towel. Hotel shampoo works fine for washing clothes. You can buy soap, shampoo, toothpaste, insect repellent, sunscreen, etc. almost anywhere, don’t bother packing a lot of that stuff — you can’t take large bottles of anything in your carry-on anyway. If you have an electric shaver or clippers or any other electrical appliances make sure they work on both 220 and 110 volts, don’t hassle with a bulky, heavy converter. Power plug adapters are not always necessary, and they are available locally for less than a dollar, you don’t need to buy those in advance.

Clean out your wallet. Loyalty cards, airline cards, business cards won’t be needed. Take photos of cards you will need and keep them on your phone or computer. Split your debit/credit cards up so they aren’t all in one place in your luggage.

Don’t worry too much about what to take. You won’t need as much stuff as you think you will, and you can buy most things you need at your destination. Haircuts, shaves, hair wash and blow-dry, laundry may be cheaper to pay for as necessary — that’s certainly true of Thailand and most of SE Asia. For comparison, a haircut costs 100 baht (about $3) in Bangkok, and a straight razor shave is 80 baht.

Stay in touch

When you work remotely it’s very important to stay in constant contact with your clients. You need to answer emails right away and be able to take phone calls. Clients will lose confidence in you if they don’t get fast responses, and the time difference is not their problem. I can handle a lot of issues with my cellphone. Just replying quickly to assure your clients that you got their email and you’re working on the issue is enough to keep them from worrying that you’ve disappeared.

I use a Nexus 5 phone. It works everywhere in the world and I can swap SIM cards easily. Prepaid phone and internet plans are fairly cheap in SE Asia, and easy to set up — you can do it 24 hours a day right at the Bangkok airport. The Nexus 5 also serves as my internet connection when I don’t have decent wi-fi: I can use it as a wi-fi or Bluetooth hotspot, or plug it in to my MacBook and charge the phone while I have it tethered over USB.

If you have a smartphone get it unlocked before you leave. Don’t worry if your phone provider won’t unlock it, though: it’s easy enough to unlock your American smartphone in Bangkok, Singapore, Hong Kong, etc.

I use Skype for calling the US, they have a cheap plan that allows me to call any US landline or cell number, including toll-free numbers, from my computer or phone. I use Google Hangouts for voice and video chat and instant messages to my G+ contacts. I have a Google Voice number that goes to voicemail, for banks and credit cards and other companies that need a phone number. Google Voice is also my outgoing SMS provider so I can send free SMS to phones in the US. I use Twilio for very cheap forwarding of a US phone number to my Thai cell number, that usually has better voice quality than Skype since it works over the cell network.

Final thoughts

Going nomadic is easier than most people think, but it’s not for everyone. You need to have skills that you can sell. You need the discipline to work from anywhere and not get distracted. You may get lonely living in a strange place where you can’t speak the language. You will meet all kinds of people, some of whom will try to take advantage of you. Finding work is a constant challenge, and you have to manage your finances carefully.

Be prepared for friends, family, and co-workers to dismiss your plans as silly and unworkable. Don’t worry, you will meet people who are traveling and working and you will make friends and learn from them.

You can find hundreds of digital nomad web sites and bloggers, there’s plenty of information and advice out there. Research as much as you can, but try not to overthink or get worried about minor details.

I’ve learned a lot and had ups and downs since I went nomadic, but I don’t imagine ever going back to a desk job or giving up on seeing as many new things as I can. I’ve been able to reduce the hours I work while improving my quality of life. I keep my skills sharp. Best of all I’ve met some amazing people and made friends I never would have known if I was sitting in an office.

This article got some great commments. Scroll down to read those.

I’ve worked on big projects, small projects, in huge teams and by myself, in fossilized federal agencies and cool Silicon Valley companies. I have learned and used at least twenty programming languages. I’ve lived through waterfall/BDUF (big design up front), structured programming, top-down, bottom-up, modular design, components, agile, Scrum, extreme, TDD, OOP, rapid prototyping, RAD, and probably others I’ve forgotten about. I’m not convinced any of these things work.

\[Edit: Let me explain what I mean by writing methodologies “don’t work.” I mean they don’t deliver a predictable or repeatable software development process in and of themselves. I don’t mean that using a methodology dooms the project. Most software development methodologies try to make programming a more engineering-like process, and in that regard they fall short.\]

How can you tell if a methodology works?

Whether a methodology works or not depends on the criteria: team productivity, happiness, retention, conformity, predictability, accountability, communication, lines per day, man-months, code quality, artifacts produced, etc. Every methodology works if you measure the right thing. But in terms of the only measurement that really matters—satisfying requirements on time and within budget—I haven’t seen any methodology deliver consistent results.

My own experiences are anecdotal, but they are shared by almost every programmer I know. It turns out that anecdotes are all that anyone has: rigorous studies of software development methodologies haven’t been done because it’s impossible to control for all of the variables.

Try this thought experiment: Imagine two teams of programmers, working with identical requirements, schedules, and budgets, in the same environment, with the same language and development tools. One team uses waterfall/BDUF, the other uses agile techniques. It’s obvious this isn’t a good experiment: The individual skills and personalities of the team members, and how they communicate with each other, will have a much bigger effect than the methodology.

In his 2003 thesis People and methodologies in software development Alistair Cockburn concludes “People’s characteristics, which vary from person to person and even from moment to moment, form a first-order driver of the team’s behavior and results. Such issues as how well they get along with each other and the fit (or misfit) of their personal characteristics with their job roles create significant, project-specific constraints on the methodology. This result indicates that people’s personal characteristics place a limit on the effect of methodologies in general.”

Our own worst enemy

When I started programming in the 1970s software development was tightly controlled by management through a hierarchy of project managers, business analysts, and senior programmers. We did not get to pick the language or tools. I worked on some big, complex projects in companies that worked this way. Some succeeded, some didn’t. Now it’s common for the programmers to choose the methodology and style of working, along with their languages and tools. Analysts are not part of most programmer’s experience anymore, and project managers have been reduced to “team leads” and “Scrum masters,” neutered of any managerial authority and not really in control of anything other than rituals dictated by team consensus.

Strict management, in love with Gantt charts and schedules and documentation, was reduced to “stakeholders” and “users,” and then abstracted away into “user stories.” It’s common now for me to get involved in a project that seems to have no adult supervision. Surprisingly, left to themselves programmers don’t revert to cowboy coding—they adopt or create methodologies stricter and more filled with ritual than anything I experienced in 1980. In fact programmers today can be much more rigid and religious about their methodologies than they imagine a 1970s-era COBOL shop was. I now routinely get involved with projects developed by one or two people burdened with so much process and “best practices” that almost nothing of real value is produced.

Once a programming team has adopted a methodology it’s almost inevitable that a few members of the team, or maybe just one bully, will demand strict adherence and turn it into a religion. The resulting passive-aggression kills productivity faster than any methodology or technology decision.

Does anything work?

My own experience, validated by Cockburn’s thesis and Frederick Brooks in No Silver Bullet, is that software development projects succeed when the key people on the team share a common vision, what Brooks calls “conceptual integrity.” This doesn’t arise from any particular methodology, and can happen in the absence of anything resembling a process. I know the feeling working on a team where everyone clicks and things just get done. What I don’t understand is why I had that feeling a lot more in the bad old days of BDUF and business analysts than I do now.

I think programmers should pay much more attention to listening to and working with their peers than to rituals and tools, and that we should be skeptical of too much process or methodologies that promise to magically make everyone more productive. Maybe social skills come harder to programmers than to other people (I’m not convinced that’s true), but developing those skills will certainly pay off a lot more than trying yet another development methodology.

\[Giles Bowkett published an excellent take-down of Scrum that ties in with this article. He covers the shortcomings of Scrum and how “agile” is practiced in real life in more (and funnier) detail than I got into here: Why Scrum Should Basically Just Die In A Fire

Comments

Keith on 10 February 2014 at 7:45 am

Amen Brother.

Gordon Morehouse on 10 February 2014 at 7:49 am

Heck, I’m green enough that I went straight from cowboy coding to something resembling Agile and then achieved my ‘grizzled veteran’ stage of programmer incompetence (search that, it’s funny) with Agile. The only time I’ve seen it “work” in any real sense is on projects undertaken with programmers who are also my friends. Otherwise – at best – it can wring a little more customer-focused work out of a dysfunctional team. I don’t know if this was true in BDUF days, but it’s my sense that in Agile, team dysfunction is usually lack of a product manager and/or terrible executive management before personal conflicts get a nod.

Paul W Homer on 10 February 2014 at 8:03 am

My experience is similar, what is often behind a high-functioning team is talented people who communicate well about a shared vision. The methodology helps, but if the other characteristics are missing than things won’t function. I have found that an overall culture of engineering at the organizational level is necessary for the initial creation of teams. The same group of developers working in a bad environment just won’t gel.

I do worry that the overall gamification of software development has shifted the focus away from producing quality results and onto less critical goals. What counts in the end is the quality of the system and whether or not it actually makes the user’s lives easier or better. Everything else is secondary.

Fred Zisk on 10 February 2014 at 8:24 am

The only method that seems to work is Waterfall. You have to know what you need to produce, design it thoroughly, and know the technology and solutions you are presenting. If there are any ‘unknowns’ things go awry. Only very experienced devs (10+ years) can spot hidden unknowns that deep six a lot of projects.

Bjorn Louser on 10 February 2014 at 8:36 am

“Surprisingly, left to themselves programmers don’t revert to cowboy coding—they adopt or create methodologies stricter and more filled with ritual than anything I experienced in 1980.”

Programmers today artificially inflate the complexity of their work under the guise of ‘best practices’ in the hope that it will create job security and lead others to perceive them as experts in the field.

It’s just the delusional thinking of generalists whose contributions to the product are as well regarded within the company as the efforts of the office custodial staff.

The lack of fear for the evil that lurks beneath the fingers of a subpar programmer leaves this neutered ‘profession’ with no recourse but to build monuments to itself through a religion of methodology.

Daniel Meyer on 6 September 2015 at 11:37 am

Your statement was so accurate I had to send you this response. I have been programming and in development for 35 years. From COBOL to .NET. I have worked for major technology corporations. Our current batch of programmers are “pretenders”, that do not respect or care about development, they are more concerned with CTOA’s so they can make money. It got so bad that I went independent, and I am now working on my own. I absolutely love the freedom of being able to create my own programs faster and more efficiently than any of the past development teams I have been on in the past five years. So to all the “pretenders” out there. Thank you 🙂

Chuck Norris on 10 February 2014 at 9:40 am

I agree that a common vision is needed. It’s also interesting that MVVG (mission, vision, values, and goals) are some of the first things that B-schools teach.

Jon on 10 February 2014 at 9:44 am

I like your article, and I get it. In my personal experiences, I have found that scrum is successful when it is used to reinforce a single coherent vision and to build a tight integrated team.

It does not work when it’s used as a magic machine of happy fun time productivity…. Sprints and stories don’t translate into more productivity. Focus and teamwork do.

Ashish Jain on 10 February 2014 at 9:50 am

I am the one, who is big fan of agility and least fan of any methodologies. Recent pattern I have observed (and currently facing), stakeholders wants to use newer methodologies to hide basic problems with projects, like not enough competent team, not enough clear goals for quarter or year. They think, using agile practices, things will fall into place automatic. That’s what everyone says.

In end, developers are spending 50% of their time in managing tasks, not really doing it. And little time they get, they just finish it off.

Tom Moxon on 10 February 2014 at 9:52 am

Nice Article Greg, I couldn’t agree more. However, I would also add that “the Vision thing” matters the most – someone who can visualize all the pieces, protocol, cogs, and gears working together. IMHO, without that – all big projects fail…

calvin on 10 February 2014 at 9:54 am

The only “methodology” I’ve read that I have seen consistently work in action is the one described in the mythical man month. (the author used a surgical team as the model) A single technical lead with full authority to make decisions, with a next tier assistant, associated technical staff, and a non-technical support person. the achievement of the team is then determined by the leadership of the team. the size of the team and project complexity is then limited by the leader and her ability to understand the problem and assign tasks. it prevents death by committee, it creates ownership, the size of the team is optimized. methodologies go into that, but only that they serve a purpose for the team and the production.

mark andreeson, steve jobs, elon musk. are famous examples of this model. And while I don’t know if it’s in MMM, the leader should arise, and not be appointed by management. The team knows who should be leader, there should be no conflict of title and ability. nothing dooms a project quicker than a leader who cannot lead a technical team. it’s about the act of an ability to take charge vs being put in charge.

noname on 10 February 2014 at 10:09 am

collaborative work sucks, esp in a git environment

Chuck on 10 February 2014 at 10:38 am

The issue with methodologies seems to lie with the misconception that it’s a cookbook that doesn’t require any thinking, deep or otherwise. Too often I’ve heard people say “just follow the methodology and it will just work” without understanding that methodologies are just frameworks that need to be tailored to the problem at hand. If the cookbook approach worked and required no additional thinking then we wouldn’t need people to build things because it could simply be automated. Having said that, there are a large number of things that likely should be automated so that people can focus on solving problems, not inventing new boilerplate methodologies.

topher on 10 February 2014 at 11:07 am

I always think of the following quote when someone is advocating that a tool/process/methodology will solve all the problems: “Life is pain, Highness! Anyone who says differently is selling something.”

Brooks was right, there is no silver bullet.

What I’ve seen work:

1. Get the smallest number of the smartest people possible together.
2. Give them a clear goal.
3. Get out of the way.

Gates VP on 10 February 2014 at 11:10 am

The reason for all of this conflict in “what works” with Software Development Methodologies is simply that there is no one “software”. The methodology for creating the software depends necessarily on the the purpose of the software and the needs of the people using the software.

Facebook’s Zuckerberg famously touts “Move Fast and Break Things”. That’s a reasonable idea in web apps that are continuously deployed 10 times / day. It’s completely useless while building Mars lander software or the banking software that makes sure you get your paycheck. Clearly a bad idea if you’re building the next Therac machine, right?

But lots of software lives in between. Could healthcare.gov have succeeded with “Move Fast & Break Things”? Well ostensibly that’s exactly what they did and in a year or so it will probably be functional, if only by virtue of trading off massive user headaches.

And as much as some groups would like to tout things like TDD, I’ve never met a database that ships with its own mocks, meaning that systems will fail regardless of their ability to pass unit tests. That stated, some form of automated testing has proven to be an improvement in basically every environment I’ve seen it used. But it has a real cost trade-off.

At the end of the day, the tooling and methodologies you use are tightly coupled to the product you’re developing. In many cases, managers and academics are still trying to quantify the trade-offs being made.

And all of the this is regardless of the team involved. Clearly, good vision and the right caliber of team will help smooth over the methodology problems. But methodology is about enabling “everyone” to succeed, not just the all-stars.

Ayan on 10 February 2014 at 11:24 am

I’ve struggled with the idea of finding the most efficient usage of time. Even after reading the comments and this article, I’m still disheartened from looking for the “perfect model”. Maybe that’s just how other perfect things work: they don’t exist.

However, and in in the tiniest part, your point on developing social skills might be just that: the ‘perfect’ plan. Of course it’s not perfect because us, the people are not, but it’s personalized and adaptive. Listening, analyzing, and measuring how your team works best, while being knowledgeable about all these ways of working sounds like a great plan. What we need to measure, how alert we need to be to our teams, and how we can grow and learn fast, while having the experience to avoid the pitfalls, is paramount in this methodology.

Anyways, thanks for the read and for sharing your experience.

Hein B on 10 February 2014 at 12:01 pm

Biggest problem is a bunch of introverts not communicating.

Clark on 10 February 2014 at 4:12 pm

We’ve discussed this a lot lately where I work, so I have a long stream of thoughts on the issue. I can relate to poor team communication, having worked with developers who either do not communicate enough and put the team out of sync, and also those developers who turn certain practices into a religion and demand strict adherence.

I would agree that a methodology that consistently satisfies requirements on time and within budget is one that “works,” but only for projects in which requirements never change, architectural decisions never need to be revisited, and time/budget requirements are immediately obvious at the beginning of the project’s lifespan (this sort of project would be a great fit for waterfall/BDUF).

Requirements change frequently, of course. How much effort should you spend nailing down a specific, achievable deadline at a stage in the project when you know the least about what the end product will look like? Would a 4 week delay be a deal-breaker? How about 8-12 weeks? Are the customer’s margins so narrow that any development methodology failing to meet or beat a somewhat arbitrary deadline is not “working”?

Improving developers’ listening and interpersonal skills might do more for productivity and success than finding the right methodology, for many teams. This absolutely doesn’t mean finding and following a good methodology is not important.

You would also be right to say that a driver’s skill and attention to the road and other vehicles does far more to get the vehicle safely to its destination in a timely manner than his choice of which route to take, or the presence of signals and signs that require adherence to certain traffic laws. Many freeways have too much traffic. Sometimes alarming amounts of your drive time is spent waiting at traffic lights at intersections. Does this diminish the importance of building roads, placing traffic signals, and requiring every driver on the road to follow a common set of rules and procedures while driving? At most, it seems to me these inefficiencies call for a new round of evaluation and improvement, so they become things that facilitate your safe journey to a desired destination, rather than a hindrance.

I’m not forgetting the possibility that a skilled HR department can put together a dream team of developers that bring with them a depth of technical skill and near precog-level interpersonal skills, allowing effective collaboration to begin on day one of each project. They are obviously hard to find and recruit, and this requires specific language when posting job openings (”rockstar”, “10x”, “superstar”, “ninja”). Barring such a team, it’s probably safe to assume we’ll continue working on teams with developers of mixed ability and communication skills.

I believe that scrum/agile methodologies can be extremely effective when implemented in a way that makes sense for the team/company. We don’t need to be overburdened with processes and SOP’s. Ceremony for the sake of ceremony shouldn’t be an unavoidable consequence of our process, and the use of an underlying process doesn’t preclude effective communication.

Jakob Martienz on 10 February 2014 at 5:22 pm

Glad people like you are wrting about this.

So True:

“Once a programming team has adopted a methodology it’s almost inevitable that a few members of the team, or maybe just one bully, will demand strict adherence and turn it into a religion. The resulting passive-aggression kills productivity faster than any methodology or technology decision.”

Ricki on 10 February 2014 at 6:31 pm

Great article! After 10+ years of coding let me share the one programming methodology that has worked somehow consistently, at least for me, my style and the skills of the help I hire. I call it “the funnel”. These are the assumptions:

1. Specs will change: guesstimate, lie about the due date and charge double.
2. Mostly we work on business-type software, process automations, so what we normally develop is a CRUD with some custom logic on update events (oversimplification, I know). Hence: deliver a prototype super fast (even with incomplete/unknown data models) and let the user “discover” the missing pieces of the specs while using the prototype.
3. Repeat cycle 1-2, funnel the client into the desired solution.
4. When analyzing the initial specs, I identify the “challenges”, the real programming, the things we cannot count on a framework/plugin to solve, the stuff we have not done before. I put someone to work on just that task and plug it into the final product when ready. Recent real example: client wants a mobile app that scans docs/reads OCR/persists on user account. Everything aside from the “read OCR” is pretty much a copy/paste client/API CRUD, we can do that in a few days. However, the challenge we work with one programmer, best ninja for the job, outsource if necessary, and when ready plug it in.

Why I think this works?

1,2 and 3 are repetitive tasks, these can be treated as engineered software development, a factory of code, measurable, predictable.

4 is creative work. The “art of programming”. It cannot be put under “project management”. If lucky it comes out in a few days, normally a few weeks or even months. Programmers are not robots (sorry managers this is true!)

Finally challenges can also be non-programming tasks: ever had to deal with 20+ rounds of UI graphic design approval? What about designing the right scalable infrastructure? Security? These challenges while not programming are nevertheless crucial for project success. They require a different methodology depending on each case.

Aahz on 10 February 2014 at 9:08 pm

I agree with you strategically but disagree tactically. That is, over-arching methodologies fail, but there are things we as a discipline have learned over the years that can be applied in almost every environment. Just a couple of examples:

• Use source control. Preferably with a culture that encourages/forces decent commit comments.
• Have some kind of automated testing — you can’t refactor easily (or sometimes at all) without it.

scaringella on 10 February 2014 at 11:17 pm

Same problems in France ….. Programmers come from the engineering/scientific field most of the time. That’s the problem.

Lonny Eachus on 10 February 2014 at 11:39 pm

In my experience, some of them (waterfall for example) don’t work simply because they’re dysfunctional as designed.

Others don’t work because very few actually do them right. For example I worked in one “Agile” shop that was great. People were good, it got the job done, and people actually did it as they were supposed to.

I worked in another shop that called itself “Agile” but in reality was anything but. They just used the informal trappings of agile as an excuse to really have a policy of “We will throw any features we want at you at any time. And, oh… no, you don’t get any time to do any of that useless refactoring stuff.” It was even worse than just regular top-down.

Guess which of those two I liked working for?

Micky Kurniawan on 10 February 2014 at 11:58 pm

Nice share 🙂

Magnus Gustin on 11 February 2014 at 1:13 am

Great post! I have a similar background and experiences. I like to compare to the world of team sports (say soccer, since I am european). An offence and defence strategy will of course help, but it all comes down to having the right players and a good coach who will stand up for the team. You want to have the right mix of people – workers who don’t ask too many questions, and the creative guys that will find the unexpected ways.

markG on 11 February 2014 at 1:21 am

I’ve worked in software development for close to 40 years now, started as a programmer, and been a CIO/Head of IT three times, including at corporates and start-ups. I’ve worked in and with some high quality engineering teams and used, introduced, varied, applied several different methodologies.

I would not say that software methodologies do not work, they can and do, what I do think is that it is simply not realistic to expect that a single method will be able to fit all known problem domains.

Software development is not an empirical science. You cannot explicitly define or quantify many of the tasks, unlike, say, a production line, or in construction where there are well proven industry formulae for certain tasks. So from the outset one must accept that no single methodology is likely to prove a panacea.

Organisations ideally should have a range of tools and methods to suit the business problems that they are likely to face, and select the most appropriate method during the preparation and evaluation phase, and the resources to be assigned to the project should have the maturity and flexibility to be adapatable. Any good developer worth their salt will choose the right tool for the job at hand. Similarly, in my view, any good delivery organisation should select the right mix of methods for the problem to be solved and the project manager and developers work to that method. In most organisations there is not likely to be significant variation in methods in any given year, but over time methods change, they have to. 20 years ago I was developing client-server applications in the oil industry, the same methods and techniques used their have morphed and changed over the years.

Having an adaptable approach is an ideal; it is very hard to make such a thing reality in the large corporates, mainly because their quality and compliance departments have choke holds on methods and processes, guarded by those whose annual bonus depends on adherence to process. Smaller companies often have the flex, but not the experience to make such an approach work. At two of the IT departments I set up, we agreed (myself, the architects and the developers) to use an appropriate mix of waterfall and agile methods, and that the project initiation process would involve resources qualified to make an appropriate assessment. Similarly at an organisation I worked at where there were 50 running projects concurrently and again we introduced a flexible approach.

As an aside, I do not think any single method can lay claim to be productive with repeatable results for all types of problem, there is no perfect approach. Consider for example football teams; they can be said to use methods, usually dictated by the experience of the manager and the quality of the resources. But there is no single demonstrably reliable method of guaranteeing success – the best teams are those that meld method and the individual talents into a cohesive group who work well together and enjoy their work. And that is probably the key, enjoyment. If you are surrounded by peers who, like you, enjoy the work, understand the methods and know the strengths and weaknesses of them, it will probably work, whether it is agile or waterfall.

Amit Gupta on 11 February 2014 at 2:17 am

Excellent article. It gives us insight how little has been done int he science of product development. I like author’s view on bringing how human factor, give more priority can change the course of entire product development. May be those methodologies were inspired by product development in other industries in the beginning. However, a bunch of people building product, require more of a better interpersonal chemistry, than the process itself. In modern days, where we see excellent products coming out of dorm-rooms and by even some amateurs, it’s the apt time to re-think the process oriented development methodologies.

KevB on 11 February 2014 at 3:03 am

Hate how the word ‘methodology’ has been highjacked to make following a ‘method’ sound more scientific.

If you are following a procedure, or talking about a defined procedure it is a METHOD.

If you are talking about the study of methods then it is ‘methodology’. Basic High School English, if you add ‘-ology’ to the end of a word you are talking about the study of.

Jacques on 11 February 2014 at 3:15 am

Your conclusion is the first point of the agile manefesto: individuals and interactions over processes and tools

Francis W. Porretto on 11 February 2014 at 3:26 am

There are any number of reasons why software methodologies — at least, those purveyed by institutions such as the Software Productivity Consortium and ISO — don’t deliver as hoped or expected, but one that must not be overlooked is that they keep changing. We’re overrun with methodologies du jour, and fads that accompany them.

We’re largely ignorant about how the programming mentality works. There’s some irony in that, programming being about the imposition of computerized order on chaotic human practices. Nevertheless, it’s so — and until we grasp the structure of that mentality (which might never happen) we’re doomed to flounder in our attempts to structure our approach to programming.

I recall, from a discussion of alternative currencies and currency management some years ago, a commentator saying that “Competition is a proper response to ignorance.” Well, the million-plus Americans working in software today are competing, in a fashion, through their individual approaches to their work. Perhaps the best thing we can do is just watch, and try to learn from those who do it better than we.

Ken McNamara on 11 February 2014 at 4:06 am

Success in any group effort is directly related to leadership.

Consider Steve Jobs – or more elementary Vince Lombardi.

Lombardi takes a failing team and in one season takes them almost to the championship.

How?

He understood football better than anyone else – and concentrated on the fundamentals.

AND he knew how to lead.

Try this book:

http://www.amazon.com/That-First-Season-Vince-Lombardi/dp/B004H8GM12/ref=sr\_1\_1?s=books&ie=UTF8&qid=1392120078&sr=1-1&keywords=that+first+season

Penguinman on 11 February 2014 at 4:25 am

Hi, you listed a few bad methodologies amongst the two good ones: ‘Top-down’, and ‘Agile’.

That is; people have to use them, not just say them.

something like this:

Focus on what your software needs to do, and hack it together quickly. Don’t worry about language, or get caught up on objects, patterns, or coding style. If there are any people on the team that argue about which editor is best, or serial port monitor application is best: Fire them. Also, if someone mentions SLOC counts to monitor productivity; Fire them.

If the software is new, or related to a new product, there is 50% chance it will fail no matter what.

So, fail early, get something working for the customer quickly..i.e hack it together with existing products, existing source. Give them 70% of the functionality, or even the 10% functionality that is most important.

Start getting feedback and test results from the end user early (within a week or two) Also, this can flow into progress payments.

Even If the product IS a success, it will take 50% longer than expected , change five times and be riddled with bugs. Avoid all these problems, by getting something into the customers hands early to test. So they can change their mind early and often.

Divide the project into small milestones, chunks..at the end of most is a workable release of the product. and feedback from the end user. Be it one application in one language, or a string of applications and services cobbled together performing whatever you need to perform. etc.

I hope you get the gist. I think the difference between (Top-down, Agile, product functionality focus) and (Bottom up, structured , methodology, code-factory focus) is: millions of dollars and months of work.

Forget about the last project,

Start with the new requirements, and find the best and newest tools to meet them.

(In my humble opinion)

Mike Breeden on 11 February 2014 at 4:31 am

This is a great article. I think I can add a couple useful comments. (Programmer about 20 years) I recently worked in a very progressive software development shop. I say that positively, because they were smart, dedicated people. We used Agile. They were good, so I wanted to see what Agile offered. Like the author said, it’s about how you use it. I did see one weakness. They seemed to lack any overall vision, such as you base Waterfall or BDUF on. Agile is fine, but I suggest you add that vision to it.

Another point I say is that I have often had the task of fixing Programmer Art and Programmer Junk. Not sure which is worse. It is not surprising that a highly intelligent developer creates art, but it can be a problem to avoid. We need business code or you’re gonna need someone like me.

Remember, from every strength comes weakness and from every weakness comes strength.

Sean Farrell on 11 February 2014 at 4:50 am

If you look at it methodologies are primarily about communication. The Waterfall Model tried to get everything onto paper so that it was obvious. The agile practices like stand up meetings or kan ban are about enabling communication between developers and ensuring that everybody is in the loop.

If you have a good team people will just talk to each other and any additional forced communication is perceived as a burden. On the other hand dysfunctional / semi-functional teams the methodology sort of force them to communicate, which raises the odds that the project will succeed.

I think most methodologies are mostly about average results. That means you can get bad to average developers to produce average results, but it comes at the cost that stellar developers are also slowed down to the average level.

I personally think the best thing you can do is get a small group of stellar developers, ensure that they work together and get out of their way. (Yes it sort of was said here already.)

Greg Jorgensen on 11 February 2014 at 4:53 am

@Jacques: I am a big fan of the Agile Manifesto. My experience in so-called “agile” teams is that it’s a buzzword that appeals to management and an excuse for more process.

@KevB: Thanks for schooling me on the correct usage of “methodology.” The dictionaries I consulted have mostly succumbed to the same error, but keep fighting the good fight.

Mike Valverde on 11 February 2014 at 5:07 am

“satisfying requirement on time and within budget”

I have never (in 15 years of software development) worked on a project large or small that the project due date was not already determined and promised to executive management before we even began working on the project. It seems that the due date must be the most import aspect of a project, since it is always determined before requirements, scope of work, features requests, design and implementation plans.

A Scrum Master on 11 February 2014 at 6:00 am

I feel I must defend the agile / Scrum camp here.

The blog post boils down to: Independent of the process in place, the team must function as a group. The author says that when a “few members of the team, or maybe just one bully, will demand strict adherence and turn it into a religion”, then “The resulting passive-aggression kills productivity faster than any methodology or technology decision.”

Well, that is really not much of an insight, I think that goes without saying. But how can you deduce from this that “Scrum, TDD, OOP”.. do not work? Let’s just look at Scrum here.

Almost all of the Scrum Master training, and most of the communication in e.g. Scrum alliance mailing lists, revolves around human factors like that. Technology is not an issue there, tools to a certain extend (but much more tools for maintaining your Scrum artifacts like backlog and burndown charts, not development tools). Every Scrum Master worth the job title is very well aware of group dynamics like these and will try to restrict/rechannel them.

Promoting a common vision is a major part of the job, every sprint even is supposed to have a “theme”. Many will try to make architecture a team effort, try to find an iterative approach to it etc., and avoid to have one team member dictating decisions.

I have had both successes and failures with Scrum, where the reasons for the failures could be tracked down to ignoring some important guidelines despite knowing better. Today I would say if done right, Scrum & lean development work very well.

I think the author has a) not really understood what Scrum is about (because it addresses exactly what he says needs addressing, group dynamics, common vision and such), and b) has worked with a Scrum Master that was maybe not focussed, or experienced, or trained enough. One should not condemn Scrum on those grounds.

Greg Jorgensen on 11 February 2014 at 6:19 am

@Mike I have had the same experience with schedules. That’s how business works, though. The problem isn’t usually the due date. The problem is that requirements change and expand against the due date. Programmers bear some responsibility for agreeing to one change after another and not alerting management when it’s clear the schedule is unrealistic. Management is often unwilling to listen to the warnings, and choose to throw more people at the project, which of course makes it later.

Steve Naidamast on 11 February 2014 at 6:24 am

Bravo !!!

As a senior software engineer who started in the field in 1974 I know exactly what the author of this essay is talking about. Like design-patterns, most software development methodologies are solutions looking for problems. There is only one way to design a software system or application and that is the tried and true method of the “waterfall” approach, which when taken down to its barest essential primarily demands up-front analysis and design, then coding, which is subsequently tested.

No software development technique since has been able to get away from such a basic format no matter how much you spice it up with fancy terminology or try to change the perspective on different aspects of such techniques.

If any of you follow senior software engineering analyst, Stephen McConnell, you will note that in his classic 1996 text, “Rapid Application Development”, he demonstrates 13 known software development techniques, all of which in the end implement some level of the waterfall approach.

Good software is a result of competent individual professionals working together or alone to make a quality product. Business development environments always promote “development by committee”, unless of course you are the only developer on staff. In any event, a “committee” has never designed anything of worth providing the reason for why most business software is so poor in quality. But every now and then you do get a business team that produces something of quality. Unfortunately, that experience for me has been a rarity in close to 40 years in the field…

Greg Jorgensen on 11 February 2014 at 6:25 am

@Scrum Master: You’re agreeing with me. If a Scrum project works it’s because the Scrum Master has the management and communication skills required to focus the team, reign in personalities and egos, deal with management, and “promote a common vision.”

Someone with those skills can lead a successful project using almost any methodology.

I don’t mean to insult your skills or training, but you must know that it’s pretty easy to get certified as a Scrum Master. I’ve attended Scrum training, I’ve worked with several Scrum Masters on different projects. Those projects succeeded or failed because of the team, not because of Scrum. Using Scrum techniques can certainly help a functional team succeed, but it can’t help a dysfunctional team not fail.

l&S on 11 February 2014 at 6:29 am

@Ricki….that,s true….i to use that though am new in the software developing world.

Greg Jorgensen on 11 February 2014 at 6:33 am

@Steve Naidamast Completely agree. Even so-called RAD and Agile projects have a BDUF because that’s what management is comfortable with. The design may not be detailed or thought through very well, and it may be dribbled out to the developers bit by bit.

That’s been my experience with most Agile projects: the team is not discovering requirements or working from use cases — they are just getting requirements spoon-fed to them so they “don’t worry” about the big picture and get distracted.

I’ve worked on both BDUF/waterfall and agile (small a) projects that succeeded. Looking back on those projects I remember the people and the conceptual integrity, not the process.

Winston on 11 February 2014 at 6:36 am

My first real experience in a “team” environment came in my 15th year of programming – I’m now in my 40th if that means anything. I was looking forward to some actual design specs to work from and a common goal. Those illusions were dashed within a few weeks. My indie experience – considered a negative during the interview process for that job – proved to be my best asset, i.e. being able to do a major makeover on short notice, and being able to deal with huge shifts in the scope and direction of the project and taking it all in stride. As well as keeping a wary eye on a part of the project and the incompetent that was working on it, and preparing myself to take it over when that person inevitably quit or was fired. He did, and I did. My cynicism about the methodology was at first considered “negativity”, but then later elevated to prophetic genius when all of my predictions about release dates etc. came true. As if someone’s a genius for saying it will rain in Seattle. I’m convinced programmers as a group fit the definition of insanity: doing the same thing over and over expecting different results. Oh the methodologies are different, but the “this time we’re going to use a methodology and make it work” mantra never dies.

When the last vestiges of the original Gantt chart lie smoldering in ruins, that’s when it’s time for me to get to work. I’m in my element. And I’ve never had to personally torch the chart to get there. They have a habit of spontaneously combusting, usually rotting from the inside out but sometimes a spectacular flare-up that scares the crap out of everyone from top to bottom. For many, it’s the end of their world. For me, it’s freedom.

John Keklak on 11 February 2014 at 6:47 am

No doubt most would agree that software development is mostly “research”. @Greg Jorgensen Even most of your work fixing web site code consists of research. To have even half a chance of estimating rationally, it is necessary to first accept the research nature of software development.

See Chapter 3 of “Introduction to Software Development”: http://bit.ly/M8wxS1 (pardon the rough state of the text; it is currently only a draft version)

@KevB @Greg Jorgensen It seems we also need another word for ‘technology’.

Anon on 11 February 2014 at 6:51 am

Agile is no more and no less than anthropomorphic observation of the human behaviour capable of producing good results. It does not show you how to get it. It just gives you the awareness to recognise the processes and people behaviours you are likely to find in a well functioning group of people. if you happen to be in one or see one take note and think yourself lucky.

Greg Jorgensen on 11 February 2014 at 6:57 am

@Winston Thank you — you said it better than I could but I know exactly what you’re talking about.

Captain Kirk on 11 February 2014 at 7:23 am

While I started in the 80s as a Cowboy Programmer, I think I have seen most of this. It is complicated.

Vision: Knowing what you are building & Why! Clarity: Knowing what you are NOT building. Focus: Having access to the people who will use it and train you in their jobs enough to “get it”.

Then you have Conceptual Integrity (is there a metaphor or elevator pitch maintained by the architect so everyone understands the system will have limitations but strengths as well)

A Clear definition of success (which should include how fast the response times need to be, because I am tired of people finishing a system that takes 30 seconds to add a new record and think it is fine when it is a keypunch type application).

Finally, you need competent people, time to do the right thing, and management to stop picking RANDOM dates.

The toughest part is the schedule, and unless you are building the same widget you have done over and over (or management is willing to spend 30% to get an accurate estimate \[see cone of certainty\]), having lots of unrealistic deadlines and no ability to get the message to the stake holders is RISKY.

Many years ago, we estimated (rough) that it would take use 2man years of effort to take on a new business line. The owner then counted EVERYONE in the room (6 of us) as full-time developers and concluded that we should be done in 4 months (on top of our existing workload, of course).

I suggested we label everyone in the office a programmer and we could have done tomorrow at noon! Somehow, I was the one being belligerent!!!

There are many places to place the blame, but for me it starts with management not understanding the process.

If a Secretary told her boss that she would 500 pages of hand written notes in 2hrs, he would not believe her.

Yes, a detailed plan of attack, and thorough analysis are required. But that should be DONE BEFORE the project actually starts! I find too many companies consider ALL of that to be part of the project, and they want the ESTIMATE first (with a target they already have in mind, and no limits to what they will ask for, of course).

I feel the world has gone mad. The places that succeed often control the definition of success (MSFT ships offices with HUNDREDS of unfixed bugs)

Arun Sunny on 11 February 2014 at 7:41 am

Good post .The comments made by experinced professionals are rather more informative

bob on 11 February 2014 at 8:14 am

We use BDUF where I work, and I too get the feeling of working on a team where everyone clicks and things just get done. That’s not to say it’s completely fault-free. The designer and project lead spend hours each week in design meetings, hammering out one page at a time which gets sent down to the development team. Even so, we often get those pages done in time with the designer and work gets done at a quick pace. Another instance of BDUF failing to deliver is when the development team and when the designer were not in agreement on how to structure pages on a site and we had to restructure the website. Luckily, this happened early in the process so it didn’t cost too much in terms of time, but it was a learning experience.

We are looking at ways to improve the process, but that leads me to think that something like BDUF should just be looked at as a template. Once you start working with your team you tune the process, adding what works and removing what doesn’t work.

araybould on 11 February 2014 at 8:42 am

Methodologies don’t deliver the results their adherents expect because their expectations are based on a fundamental misunderstanding of software development: it is not a form of manufacturing, where the goal is to produce many replicas of the production prototype, it is a design-like activity which concludes with the creation of the production prototype. Success depends on knowledge and good judgement, and methodologies cannot substitute for this; at best they provide guidelines for competent people.

Mark on 11 February 2014 at 8:51 am

Wonderful vindication of DeMarco & Lister’s “Peopleware” (1987).

Dan Sutton on 11 February 2014 at 8:53 am

Absolutely right. Another way to look at it is like this: just as one should work out what the problem is before choosing a language in which to write the solution, the same goes for the development method: as you say, no one method suits any problem: to choose a development method irrespectively of what you have to solve is just as idiotic.

There’s something more, though: in many cases, these potted methodologies do little more than shield the incapable from detection: as someone who’s been programming since the ’70s, you’re no doubt as aware as me that most of today’s programmers simply don’t understand the basics — they run around spouting acronyms and “leveraging functionality” – whatever the hell that means – but they can’t write a simple sort algorithm: “There’s a class for that”… so, from that point of view, the best development methodology is education.

BrainiacV on 11 February 2014 at 8:53 am

I agree with everything you said, and many of the comments.

I’ve always seen “methodology” as a way everyone (mostly management) to turn off their brains and think the process will guarantee results instead of the efforts of the team.

It’s what I call “Industrial Age Thinking” as opposed to “Information Age Thinking”. Industrial processes are based on the idea that you can come up with the ideal way to manufacture widgets through design and/or time/motion studies.

That doesn’t apply to software development. Programming is driven by cognition and you can’t force everyone to understand everything at the same time or degree.

You can try to make things easier, but there is no guarantee what you do this time will work the next.

Which is why I view software development like jazz, you go with what works at the moment.

robb on 11 February 2014 at 9:15 am

Very thought-provoking article. IMO, it comes down to this: People, process, tools….in that order. Without the right people, any process is worthless. With the right people, it can help alot.

James Ashley on 11 February 2014 at 11:24 am

Why don’t software methodologies work?

Because you aren’t doing it right and nobody else is either?

The saddest thing is when even the people implementing an agile process have no real faith in it. I had a project where SCRUM was done off of a visual studio template and after about three weeks no one was even adding stories or moving cards anymore — and worse, no one seemed to care. They still extolled the virtues of git over TFS, tho.

Oh — and the project took twice the time projected, went way over budget, functionality was cut, people were thrown under the bus (including most of the design team) and, it turned out, the iOS expert leading a big portion of the project didn’t actually know how to build iOS apps, etc. Good times.

Thomas Phaneuf on 11 February 2014 at 12:03 pm

Good article. And I agree with the author and most of the comments. Any methodology (yes, more accurately, “method”) should be looked at as a prospective path, not a silver bullet. It can’t replace the overall vision, or the compensate for problems with people in a development team, or address all the unknowns or even make coffee. But it is a starting point. It can also help guide the development and help keep it moving towards a goal. Methodologies, if used right, are a good thing – but not magic, and not a guarantee. However, if there is more focus on the development method used than the project itself, then things will go awry very quickly.

I like what Topher wrote and it is worth stealing and repeating:

1. Get the smallest number of the smartest people possible together.
2. Give them a clear goal.
3. Get out of the way.

Andrej on 11 February 2014 at 12:55 pm

Non professionals + Good Methodology = fail

Non experienced team of construction workers will not build a skyscraper

WyattMobile.com on 11 February 2014 at 2:09 pm

All Software Methodologies are a desperate attempt to include stupid people in the game. From SCRUM to TQM, they are all an attempt to allow someone with a communications degree to be included. Those people are cheap to employ, and owners are convinced it is a matter of having enough idiots who can tell a programmer what to do.

Cassandra Andra on 11 February 2014 at 4:24 pm

Claro! Animated cartoons often depict a group of IT planners gathered around a blackboard with arrowed lines drawn in every direction, criss-crossing dotted lines and unfathomable schematics, meaning haplessly confused. How to create more content over time is a driving function of the world. Even this example generates five seconds of entertainment!

Though I don’t think the purpose is to include poorer programmers in the game, I rather think it is to create a sideshow. It’s something else that can be “done” and therefore is. Short of the sense to not destroy the world, “do override” is major problem in the logic of the mind. A better law would be, “A system at rest should be left to rest.”

Most people in a room empty except for a button, would push it. Then, a bouncing ball in the same room would inevitably push it. I agree, it’s just one more thing that IT persons can do – is philosophize about work. Hmm…

Matt on 11 February 2014 at 4:27 pm

I’ve been a developer for 20+ years. Like the author I’ve seen all the methodologies and lots worked / failed in many different situations.

The one common thing I’ve noticed is the owners (managers, stakeholders etc) do not know how to document what they want or don’t really even know what they want.

I’ve started using a few simple diagrams that I borrowed from a book called ICONIX by Doug Rosenberg. I found that these simple diagrams (Use case, Domain Diagram, Robustness) were all I needed to bridge the huge gap.

It gave us all a common nomenclature (domain diagram) it helped managers quantify the invisible (like when we’re working on things they can’t see, like databases, views, procedures, functions, etc) it helped us to estimate (count the bubbles and multiple by some coefficient)

It forced project owners to do more than draw up a screen in photoshop and then believe that since they can see it it must be really easy to build. It also forced them to think about what they are asking so that they don’t make illogical or contradictory requests.

So, no matter which methodology, increased communication of what the software should do, not how it should do it, and the god forsaken question “how many hours is that going to take” have always been signs of success for me in my career.

Good developers self monitor and don’t need a stupid scrum meeting daily.

Peace,

-Matt

David Slick on 11 February 2014 at 9:19 pm

Your thoughts definitely resonate. Particularly the fact that there’s not one silver bullet. All too often “Agile” is used as just that, a silver bullet to solve other problems deeply embedded in the company culture/leadership. Agile doesn’t solve those problems, but I’ve seen it help shed light on them so they can be fixed. But it still takes mature teams/orgs to do something about them. It also enables you to get customer feedback and integrate that so you can adapt to build things incrementally toward your ultimate value goals, but it takes a company that has customers and is willing/able to access them.

Also, I’ve seen great code written using TDD and without any unit tests. But on average, the code I’ve seen unit tested is of higher quality, if TDD’d even better, and it mostly stays working over time.

I think it’s important to be clear about any methodology is meant to give you. Do you need that? Are you willing to pay the costs associated to get it? Is it worth it? Do you have teams mature enough to handle it? Do you have management skilled enough to deal with the output?

And more importantly than the practices, agile is about the values. You share those values outlines in the agile manifesto? If not, don’t bother, you’re just wasting your time.

Stephan Schmidt on 12 February 2014 at 12:07 am

“rigorous studies of software development methodologies haven’t been done because it’s impossible to control for all of the variables.”

No, because it’s so much easier to go by anecdotes than do some multi-year research on a topic.

Greg Jorgensen on 12 February 2014 at 4:29 am

@Stephan Schmidt: Maybe “easier” to go by anecdotes, but when you can’t control for the biggest variable — people — it’s not possible to test the effectiveness of a methodology. There’s no way to have the same team do the same programming task using different methodologies and from that determine which works best, unless you have a Men In Black memory eraser.

Edub on 12 February 2014 at 8:20 am

“Once a programming team has adopted a methodology it’s almost inevitable that a few members of the team, or maybe just one bully, will demand strict adherence and turn it into a religion. The resulting passive-aggression kills productivity faster than any methodology or technology decision.” Excellent point!!! So, So very true!!! I have been subject to the bullying of system engineers and DBAs who seem to base their adherence of a “policy” based on how they feel that day. Policy is in quotes because either they have never read the policy (just hear say) or in absence of a policy they make up their own rules.

Tim on 12 February 2014 at 1:15 pm

#4 Fred Zisk: “The only method that seems to work is waterfall.”

Having completed well over a dozen implementations of enterprise systems in the last decade, I can say with complete confidence that waterfall is a project killer. There indeed is no silver bullet but Agile comes the closest to acknowledging the realities of modern business.

Tom on 12 February 2014 at 2:49 pm

Would you ask a biology graduate to remove your appendix? Would you travel in a jet plane designed by a team of mathematicians? Would you work in a chemical plant designed by chemistry grads. Wonder why the development processes don’t yield better results? Just look at the people.

Roughly half of all software developers have degrees in computer science. Most of the rest do not have degrees related to computing. I did some work for a government agency and discovered that one developer had a masters in Eastern European folk dancing. Hopefully he minored in Java or something similar.

Training shows you how to do something while education broadens your mind.

The people who graduated in physics or math have some change, but even the people who graduated from Computer Science were educated by the Science Faculty. The Science faculty does not train people in product development. There are no courses in development metrics, configuration management, project management or quality assurance. Basically the vast majority of the people producing software today were never prepared for this role by their education. They are essentially all self taught.

And they do not know what they do not know!!!!

As for the methodologies that do not work, I have very seldom seen an accurate description of the so called waterfall model. Advocates of other methods us it as a straw man to show how much more reasonable their approach is. But the waterfall model just says requirements before design, design before construction, construction before testing. All engineering processes follow this model at some level (although the TDD people seem to want to reverse this). Basically, most of the people involved in software development have not really studied any development approaches and most groups adopt a given method as the result of the preference of the loudest person in the group.

Another really bad thing about most software development processes that are actually in use is that they are oriented towards building features. While that is what users are interested in, it is not what engineers produce. Engineers produce designs for things which is software translates into components. In a feature oriented development, anyone can change anything which typically forces code merges and the associated defects that merging introduces. It also forces everyone to be familiar with everything which is okay if the system is 10,000 lines of code but is a disaster when the system is 500KLOCs or bigger. Nobody can understand enough of such a large system to safely make modifications to arbitrary parts of it. And if everyone is responsible for all of the designs of all of the changes they make to all of the bits they touch to implement their assigned features, nobody is accountable when there is no design.

Failure is not left to chance. And even when a project fails, it is often deemed a success. Look that the Chrysler c3 payroll project that gave us agile development.

initdotd on 12 February 2014 at 7:06 pm

One word, plastics! All kidding aside the common vision is critical. As it develops and the truth begins to show itself, you end up with, regardless of what you call it, requirements. Idea to concept to something well defined that indeed can be measured and deemed success when reached. Methodology has nothing to do with finding the truth although some would argue it helps you get there. I’ll buy that. The point is you learn as you go and what you measure as success will directly impact the way you choose to define it.

Rick Mowen on 12 February 2014 at 9:48 pm

Yes – I heartily concur with this interesting treatise on the fundamental nature of practices in the area of software development postulated by this intriquing author. I myself have developed several similar methodologies. My first foray into this area led me to develop “yum-go-go” a work framework which simply states that the relative work accomplished is directly proportional to the size of one’s breakfast. It was adopted throughout the many large organizations and led directly to the implementation of work sponsored cafeterias. Having worked mostly at smaller companies, I never did directly benefit from those advancements which led me to develop over time “obscure-o”, which postulates that the amount of equipment one’s office, working or otherwise, is directly proportional to one’s worth as a team member. Lastly, my hobby as a collector of Americana, as well as my involvement in Linux (which I pronounce Lin-ish as a tribute to my own personal involvement in the erstwhile os), has given me the perspective to write “Working Without Effectivity” which is an explorations into the field of “Malleable Task Management using Waterwings”.

Keep up the good work!!

A Scrum Master on 13 February 2014 at 1:26 am

@Greg Oh yes, I’m not arguing that you don’t need a well-cooperating team, or that process would be more important than the people on the team finding that spirit and “clicking”, as you put it. I was just surprised that you dismiss the agile / Scrum approach on that basis, since from my perspective that’s exactly what they’re about. As someone else pointed out above, after all the first sentence of the Agile Manifesto is “Individuals and interactions over processes and tools”. Isn’t that also what you are saying?

And of course anyone can take a two day seminar and get a Scrum Master certificate. But I find it hard to imagine that within those two days, you would not speak about group dynamics, about Tuckman’s stages of group development, about handling “difficult” people, about how team members have different expectations and motivation, about retrospectives and how to make them effective, and how to set an atmosphere where everybody speak their mind. I’m not saying that you are given solutions for all of those challenges, or that everything then subsequently just works, but at least you’ll have some awareness of these issues.

Which brings me to my last point: Those projects you mention you are involved in routinely, those where “almost nothing of real value is produced” – are Scrum projects among those? If one of these projects was a Scrum project, how did the Scrum Master handle the situation? When nothing of real value was produced, clearly there must have been sprints where the sprint goals where not reached. It’s hard to imagine that the Scrum Master wouldn’t address that at all. Was the burden of too much process etc. never brought up in a retrospective? Being agile is about adjusting the process – you sprint, find out what went well and what went badly, encourage the former and try to reduce the latter. If in your project things didn’t work out and you just kept doing them anyway, well, then that wasn’t really an agile approach, no matter what your managers might have called it.

I really think most of what you write is correct, but the headline for me just isn’t. “Why is everything that has been found out about collaborative software development in the last 20 years being ignored in the projects I work in?” would be a more appropriate, if less catchy, title.

Alessandro Koterle on 13 February 2014 at 3:19 am

After having thought it over for a while before leaving a comment, it came clear enough to me that this article is lacking too much in giving clear boundaries and contexts and then it could be likely misleading. Indeed, seems that even the author has recognized it and retreat a bit by editing original post which lead me to consider this article more seriously. Having been worked for many years on many small/big projects, with huge teams, adopting any of existing methods, processes, practices, technologies doesn’t necessarily mean having got them right or at least not giving them enough attention. Claiming that no methodology works is like saying ‘No girl/boy-friend is good enough for me’ should I live alone then? or ‘None of any car is suitable for me’ should I walk a lot then? or ‘None of programming languages fit my expectations’ should I learn and use them all together ? (guessing whether learning well enough more than ten could be actually possible).

Furthermore, should we all discuss on a mix of several relevant topics such TDD, OOP, Agile, RAD, Scrum, Human interaction all at once in an misleading if not erroneously ordered unlikely recipe?

More than that, reading the article, I have personally perceived that it is mainly focused on people ant that leads me to to think how often people run into troubles with his peers (job mates). Please note that Alistair Cockburn thesis reports several answers we all could further discuss on and not just keep them as a generic and ‘pessimistic’ conclusion over human beings. Indeed he also answers that: “Every project needs a slightly different methodology, based on those people characteristics, the project’s specific priorities, and the technologies being used. This result indicates that a team’s methodology should be personalized to the team during the project and may even change during the project.” So he is saying something slightly different from what this article aims to.

I perfectly agree on ‘Strict Management’ issues, on ‘No Silver Bullet’ concept and on what ‘Two-different-programmer-teams’ thought experiment aimed to claim at and also that human beings personalities and interactions obviously will ever make the difference. People interaction over processes is another good point I agree upon.

But there is a quite big discrepancy between arguing on such issues and claiming that most of or even all methodologies will fail. Furthermore, who or what will exactly fail? How long a project should last before stating a given process/methodology makes it fail? Who or what lead the change in adopting a new process or methodology and who or what actually caused the failure?

Scrum has been mentioned as well. Scrum is not a methodology but rather a framework providing teams with some guidance in adopting proven ‘agile’ practices. It’s up to managers, software development teams, hardware teams, even sales get a try and work hard to achieve continuous improvements.

Here we go, Is the continuous improvement a ‘typical programmer’ aims at?

Mr CSharp on 13 February 2014 at 1:05 pm

I agree with this blog on the most part. No one methodology works alone. You got to take the best of every methodology and adapt as the situation comes along. Definitely some methodology and standards are better than none. The thing you’d hate the most, especially on a mid to large sized teams is when you have numerous code achieving the same functionality and that similar code is copied and pasted everywhere. Then, when the functionality changes, multiple code changes are required throughout the app. That’s what you call cowboy programming. Methodology and standards that are too strict definitely kills the spirit on a team level since rules cannot be bent to allow for flexibility. Think of it this way, software development is like the Matrix. The Matrix keeps everything in-bounds, which is a good thing. The Matrix can also warp to adjust for other things and it’s not rigid. The Matrix is your friend in the software world, unlike the movie where it is the enemy. 🙂

Gnum on 13 February 2014 at 2:35 pm

All you saying is the rephrase of the old statement – “Any system is only as good as it’s parts combined”. Of course no method can substitute the actual skills, effort and genuine commitment. You can’t expect to take a bunch of sloppy inexperienced, willing to work for a higher salary, workers, call them developers and with the magic of some Methodology produce good software in short time for small cost. Of course you need to have an adequate skills and sufficient level of commitment in a first place. This is all only a premise for methodology.

Methodology comes to play when you DO have a good team with good skills, well committed, with right attitude and sufficient social skills. When you have all that then you have a choice of how exactly you organize your work, and then it comes to the choice of pattern and the process.

Saying there is no need of pattern or the process is effectively saying that the team is either not in a state when it is ready for any organized work. In other words it means you don’t have a team. You have some may be well talented individuals, may be even in the same room, but that doesn’t make them a team.

Until you have a qualified team the question of methodology or the process is an abstract. It is like choosing the route of commute before getting the job. It doesn’t matter which way you come to work of you don’t have a job. It doesn’t matter how you dress for work if you don’t have a job. It doesn’t matter what you decide your daily work routine to be if you don’t have a job. None of these choice will make any difference on your income if you don’t have a job.

Same story with the process, methodology and the team. If you don’t have a functional team – no process will help. When you do have a team – the choice of methodology and the process will make a whole world of difference then.

Gene on 13 February 2014 at 9:13 pm

I had to weigh in on this excellent article. As said directly above, the comments made by the experienced professionals are of more value than the others but it is good to see technical professionals debating this subject.

I coded, then project managed, then directed but when you have your own business and it is really your money that is being lost and your firm’s name that is at risk with your customers, the learning curve steepens. Following are a few things I’ve learned and some quick thoughts on the matter.

Clearly no one methodology fits all projects. I have a lot of trouble listening to individuals who think one method is the best (sometimes the only) solution. The overall approach must fit the type of project, type and criticality of the application(s), complexity, timeline, acceptable level of risk, customer’s team, customer’s business controls, technology team’s familiarity in the application area, available tools, diversity in the team, customer’s maturity with the process, the team’s professional maturity, acceptability of frank and open challenges in the various environments, communication skills among the members, experience the team has with each other, proximity and access the team has to each other and all the various constraints whether technical or project oriented. I’m not being glib here making this list. This was just a quick flow and is not organized but if you really dig into projects you will see that every one of these things and more factor into the equation.

This is why each project should start with an approach yet remain flexible as more is learned about all of the above. Long academic debates about waterfall vs. agile are interesting and much can be learned, but at the end of the day a general approach needs to be determined and then some principles developed to control the project.

Principles allow the project to be controlled as tightly or as loosely as fits the overall landscape. For example a principle may state “no code shall be final-tested by the same individual that did the programming” and “subject matter experts shall control the final testing” and “subject matter experts shall be named and will formally have defined functional responsibilities”. Obviously much care must be taken developing and applying the principles and they must be comprehensive but statements like these are easy for everyone on the team to understand and it is easy to measure compliance. And principles can be loosened up to be more of “guidelines” or tightened down to be more of a “specification”.

So after my 30+ years at this on projects small and large and in some very critical environments I have come to the same conclusion a neurosurgeon who was my doctor told me one time “the more experienced I get the more I realize the less I really understand”. The key word is understand because certainly the experienced professional knows more than the less experienced, but how much is really understood of a complex subject as this, with humans as the primary variable, is the elusive part which really matters. So my conclusion is to start with the best approach you can given the experience level in your team, consider using principles you develop instead of very rigid structures, encourage debate and be willing to listen, learn and adapt but keep the eye on the goal. Most importantly keep moving forward without dwelling too intently on trying to control every element of the process. Oh and the obvious hire and provide motivation at all levels for the very best people you can afford!

Stefan on 14 February 2014 at 12:56 am

I’ve found that all software processes have two things in common:

1. plan your development in managable chunks, over managable intervals of time
2. reassess your achievements in regular intervals and adapt your assignments accordingly

The projects that were good at these two things also went reasonable well. Those that didn’t went way over budget or failed. That said, I recall only one case of failing at the first 😉

Piotr on 14 February 2014 at 2:44 am

The answer is easy – the “problem” is not because of methodology, the “problem” is because of skills… New methodologies are invented to “legitimate” lame development instead of facing the issue directly – poore skills. Really good developer can be successful even with waterfall up-side-down or whatever. The thing is that education is tough, requires lots of effort and time, so it’s being skipped in our greedy world.

H.M.Müller on 14 February 2014 at 7:20 am

All “engineering” (design-level engineering) works like that .. what’s “the problem”? Talk to people designing (not building!!) a next generation car engine; or an airport (even a small one); or a new assembly line. We always go to the limit of what is possible – that’s the whole idea of that “progress enterprise” mankind has been doing at least for the last 400 years.

All teams I have worked with in the last 30 years have used methodologies – i.e. abstract ideas about how work (thinking, designing, communicating, writing code, testing, etc.etc.) should be done: They had their rules – sometimes purely local ones, but most of the time a mix of external and local ones they adapted – knowingly or unknowingly – to their environment. And they worked to the level they worked – depending on the environment (much/little money, much little/time, good/bad managers, …) and the problem to be solved (standard/risky, large/small, …).

This article shows once again that programmers (me included, in my younger years) are a bunch of people that believe that they can rule the world with what they think. This is simply wrong. But of course it is still worthwhile to try to organize oneself and the people around – from team to project/department to all stakeholders – so that the enterprises we start come to an end that is as good as possible (measured on that very multi-dimensional scale of “good”\*).

• There are even projects where it is a success if they fail … I could tell a few stories about that.

H.M.

Tim S. on 14 February 2014 at 1:03 pm

Definitely agree. This was one of the best programming articles I’ve read in a long time. I’m sharing this with my-workers as our management is looking to implement stricter red-tape/processes in our environment, even though all is well.

CMS on 14 February 2014 at 11:36 pm

I agree it comes down to the talent. The “not reverting to cowboy coding…” comment grabbed my attention the most. My initial reaction was skepticism and admittedly some disdain. However, who am I to question your life’s experience.

My reaction stems from the following. There are oceans of garbage development in the industrial automation business. Which I believe is a somewhat different craft, but surely shares many fundamentals. Per my life experience most of the people I’ve run into that deride specs, processes, documentation, meetings, etc. often do the worst quality work, AND can’t see it! And even when forced into more structured work processes, it only helps so much, because the values still don’t match.

So I come back to my opening, the talent, and good leaders/mentors are the most important part of project success.

I shouldn’t complain about bad coders, they are good for business in our arena. But it disheartening at times being immersed in rotten systems all the time.

ricky on 15 February 2014 at 3:15 pm

You left out an interesting if expensive option that I’ve seen work. Plan to throw one away: In particular, first build a BDUF thingy a as a prototype; Document the hell out of every function and module; Use the documents and running code as spec for the next version.

Andy on 16 February 2014 at 2:44 pm

IT Consultants can take much of the blame for corrupting methodologies. It is often…no always in their interests to pad out a project with extra staff, extra process and to take as long as possible. Their version of “good practice” is all about profit…and it works depressingly well.

I have seen $800M spent on a data warehouse that was designed to fail. (raw XML in relational tables with one column!) 80% of the “team” had never coded yet they were convinced they were “high level” design geniuses. I can design a nuclear power plant at a “high level” and it is similarly useful!

Why is all this failure and waste not reported? The ones who hire them cover their reputations and call the project a “success” then claim their bonus.

EG. A large Telco spent over a decade trying to get a “flexible” billing system to work…they gave up and hired the same consultancy to replace it with a new one. This seemed baffling until you realise how the “alumni” system works: You scratch their back…

IT is productive, it creates wealth. Like leeches to a bloated swimmer, the consultants and pretenders tap into the wealth. Programmers are now beast of burden, to be tapped and carefully contained by the ladder climbers and pseudo-techs.

Jacek Dalkowski on 17 February 2014 at 3:20 pm

Best methodology is good will and common sense. Unfortunatelly, it is not popular enough 😉

Phil W on 18 February 2014 at 11:29 am

That opening paragraph nails it – in decades of programming that has become the problem. There’s a hole in the hierarchy between the managers who say “We want this” and the developers. The middle guys who could translate requirements into development strategy, plans, code, objects or whatever (the Powerpoint to coding guys) have gone. I think managers were never happy about their technical objections or their salaries after years in the business, so a bunch of largely clueless programmers beneath a manager who can’t actually guide them technically has become the norm. I keep seeing this in many setups and the results are as predictable as you’d expect, regardless of methodology or tools.

Daniel Garcia on 18 February 2014 at 7:21 pm

“I think programmers should pay much more attention to listening to and working with their peers than to rituals and tools” Do you think pair-programming could help in some degree to “work with peers”? Just curious to know your opinion.

nooshin on 19 February 2014 at 1:52 pm

hi,I’m doing a project and I need information about the software engineering methodology developed in 20 of the last 50 years to know please help me in this regard

Brian Shreeves on 2 April 2014 at 7:09 pm

Greg, I realize this post is not old, but chances are your experiences have a little more mileage. Considering that you have realized that personalities, teamanship and proper leadership are crucial to the success of a methodology, therefore the product. Have you recognized ways to aptly put together a well functioning team?

John Hunter on 23 August 2014 at 6:10 am

I think the key bit there is “in and of themselves.” Certain methodologies are excellent in the right situation. But I do believe things need to be molded to the organization.

I do think agile-like methodologies are very good for most (if not nearly all) situations. But different situations will require somewhat (to very) different practices within the agile range of options. I wrote about my experience

http://management.curiouscatblog.net/2014/05/29/building-a-great-software-development-team/

Erin Shellman on 26 August 2014 at 7:02 pm

I wonder if maybe you had that feeling of conceptual integrity in the bad old days because people don’t stay in their jobs very long anymore. With such high turnover it’s got to be hard to really develop a high-functioning, super productive team.

Gregory McIntyre on 26 August 2014 at 7:03 pm

I agree.

Question: Is agile not meant to be taken with a grain of salt and applied wisely and perhaps adaptively to the circumstances at hand?

If you read the agile manifesto, that seems to be the implication. Perhaps it’s the eroding of the original spirit as larger corporations adopt a top-down authoritarian approach to agile that goes against the entire point of it, that agile becomes silly.

Doug Reeder on 27 August 2014 at 7:00 am

I’d say, rather, that the real goal is something beyond process that we can’t yet describe. The goal of any process is to get programmers in the state of mind where they write good code.

Isidro López on 27 August 2014 at 10:21 am

The point is: would two IDENTICAL teams within IDENTICAL conditions get the same results with a waterfall and with an “agile” methodology?

When I say “the same results” I’m thinking about everything: the quality of the code, the satisfaction of the customer, the satisfaction of the team itself, the economical cost, the duration, etc.

Obviously, it’s scientifically impossible to prove it (because two things can not happen at the same time… I think 🙂 ), but all the experiences that I had myself and saw around scream out the same answer: NO. I would definitely bet my own life that the overall outcome would be mucho more satisfactory following agile-like methodologies rather than waterfall ones.

Oh, sure, I forgot to write down “IMHO” at the very beginning of every sentence that I wrote :-p

I get paid to take on technical debt. In my work I see a lot of hard-to-maintain code, and I see many of the same avoidable problems over and over.

I specialize in debugging, fixing, maintaining, and extending legacy software systems. My typical client has a web site or internal application that works, more or less, but the original developer isn’t available. Business requirements have changed and the software hasn’t kept up. Or my client has something that is “almost finished” but they parted ways with the developer after using up their budget and schedule. Usually there’s a list of missing features and bugs.

My typical client usually has been told by other developers that they need to throw everything away and start from scratch. Most programmers don’t like maintaining code, and they especially don’t like maintaining someone else’s code. When writing code programmers often ask the wrong questions when they talk about maintainability—see Matt DuVall’s article The myth of maintainability for a good description of how that happens.

Here are some things you can do in your own software projects to keep me in business:

Don’t use version control

I’m always surprised to find big projects written in the last few years that aren’t under source code version control. When you don’t use version control the next programmer to come along has to figure out which files are part of the current system and which are obsolete or backups. The next programmer won’t have any commit messages or change logs to get the code’s history from. I covered how not to use version control in my article Introduction to Abject-Oriented Programming.

Customize your development environment. A lot.

Don’t make it easy for the next programmer to start working on the code. Require specific versions of languages and tools, and make sure they conflict with the versions that come with the operating system. Customize your Eclipse or VisualStudio or vim environment like crazy, then write macros and scripts that only work with that setup. Don’t have a disk image or scripts to replicate your development environment, and don’t bother writing any documentation—it will be intuitive.

Create an elaborate build and deployment environment

For a web site deployment to a staging or production server should look like one of these:

svn up
git pull
hg pull

Programmers can argue about simplicity and elegance in code, and then turn around and build the most elaborate and baroque build and deployment systems. This is one of the off-the-radar things that programmers do without their client or project manager reviewing or understanding it, so it easily gets out of control. While you’re chaining together eight different tools with various scripting languages remember to leave out the documentation.

###Don’t set up testing/staging platforms Changing a production system is exciting. Don’t bother with a testing/staging server. Instead have secret logins and backdoor URLs to test new features. Mix test data with real data in your database. Since you aren’t using source control keep copies of previous versions just in case. Don’t build logging into the code. Don’t disable outgoing emails, credit card authorizations, etc. during testing.

Write everything from scratch

Don’t bother with a well-understood framework like Django, Rails, or CakePHP. You can write a better templating engine, ORM, sort or hashing algorithm. I scratch my head when I see code with comments like “faster than native dictionary method” or “replacing PHP library function because parameter order sucks.”

Add dependencies to specific versions of libraries and resources…

Throw in as much third-party code as you can. Link to as many shared libraries as you need to. I’ve seen code depend on large external libraries just to get access to one function. Modify the source code of the third-party libraries so they can’t automatically be updated to a newer version, but don’t bother forking or keeping your modified versions under source code control. I will be able to diff your version with the original and figure it out.

… But don’t protect or document those dependencies

I get more urgent calls because of updates and upgrades gone wrong than for any other reason. A seemingly innocuous WordPress upgrade, Linux package update, or new jQuery release will trigger a chain reaction of failures. Don’t make your code check for the specific version or modified copy of your external resources or third-party libraries. Don’t even add a comment to remind yourself.

Use a bunch of different programming languages and stay cutting-edge

Every day HackerNews and Reddit buzz with cool new languages. Try them out on your client’s time. Any decent programmer can learn a new language in no time, so if the next programmer who gets to work on your code is a newb that’s not your problem. The boundaries between the languages, the incompatible APIs and data formats, the different server configuration requirements are all fun challenges to overcome and post about on StackOverflow. I’ve actually seen PHP web sites with pieces of Ruby wedged in because everyone knows PHP sucks and Ruby is better. Half-baked caching, aborted Rails and Node.js projects, and especially NoSQL solutions (“It’s more scaleable!”) are my bread and butter.

Where’s the programming advice?

It won’t matter much if your code is beautifully object-oriented or shiny and functional—programmers pretty much always see spaghetti code when they have to maintain a legacy system. I am good with diff and grep and Ctags, tracing through code, refactoring, and debugging. I’ll figure out the code. The most beautiful, elegant code is still hard to work with if there’s no source control, too many dependencies and customizations, and no testing/staging facility. It’s like finding one clean and nicely-decorated room in a house straight out of “Hoarders.”

Related and possibly interesting: The joys of maintenance programming.

In my career as a programmer I’ve seen lots of projects go off the rails because of strict adherence to some practice, rule, or fashion. It may be something the entire team has bought into, like OOP or TDD. It may be something a single member of the team bullies everyone else about, like tabs vs. spaces or braces style. Even a programmer working alone can sabotage a project by honoring fetishes above productivity.

Here are just a few of the fetishes I have observed wasting hours and days of programmer time:

• Spaces vs. tabs, how much to indent each line
• Braces style
• CamelCase, mixedCase, names\_with\_underscores, etc.
• Variable naming conventions, especially Hungarian notation
• No function can be longer than 50 or 100 lines, or no function can be more than one printed page
• Never use GOTO, eval, operator overloading, singletons, or some other language feature deemed evil
• Functions can have only one return
• No global variables
• Pure OOP, pure functional
• Design patterns
• TDD (test-driven development)
• Gannt charts, Agile, Scrum, Extreme, User Stories
• Never use SQL stored procedures, or always use SQL stored procedures
• SQL vs. NoSQL
• Never use PHP short tags
• All web interfaces must be RESTful
• HTML must validate with W3C
• Strict XHTML

Not all of these things are necessarily bad ideas. And it is important for programmers on a team to follow some guidelines so their code is understandable and plays well with the rest of the system. The problems arise when a programmer or a team decides that purity according to an arbitrary rule is more important than delivering a working solution. If someone is paying you to develop an application or solve a business problem it really isn’t important how the source code is indented. It is important to the customer that the programmers not waste many billable hours debating whether singletons are bad or not, or if all of the variable names follow one person’s favored convention.

The problem is similar to cargo cult programming. It’s bad enough when newbie programmers do things without understanding why. It’s worse, I think, when experienced programmers turn their own preferences and biases into fetishes and then force themselves and everyone they work with to follow them, even when that means putting the project at risk.

For the last few years I’ve specialized in debugging and fixing code, a lot of it abandoned when the developer and the customer divorced over budget and schedule overruns and missed expectations. I see code that doesn’t even work sprinkled with self-congratulatory comments referencing the GoF patterns. I see badly-written replacements for library functions that exist only because the programmer decided that the standard version’s name sucks. I see specs full of justifications for technical choices that don’t mention important business rules.

I also see lots of code that doesn’t use my preferred indentation, tabs, braces style, variable naming, etc. Spending time “fixing” the code to satisfy my aesthetic sense would be a waste of time and money for my client. I’ve figured out that code that wasn’t written to my tastes is really no harder for me to read and maintain than my own code.

Methodologies, techniques, and best practices should be understood and used to solve problems, not worshipped as rules that must be obeyed no matter what the cost. If you are spending more time debating naming conventions with your colleagues (or yourself) than you are writing code that solves a problem you are wasting time with programming fetishes.

Comments

mike on 30 September 2013 at 9:14 am

Coding out of a fetish is stupid… coding with reason is logic.

Eldc on 30 September 2013 at 11:27 am

There still is usefulness in having a consistent coding style. Enforcing things too strictly is certainly a problem, but having gotos everywhere is still going to make things difficult. As always, seek balance: http://en.wikipedia.org/wiki/Perfect\_is\_the\_enemy\_of\_good

Anton on 1 October 2013 at 5:18 am

You forgot that code line should not be longer than 80 symbols.

Greg Jorgensen on 1 October 2013 at 10:30 am

@Eldc I agree that a consistent style is useful, though not as important as some programmers think it is. Agree on some ground rules and move on. Do code reviews and if someone is writing truly unreadable code do something about that.

@Anton I haven’t forgotten, just blocked it out. The 80 line limit came from punch cards and was perpetuated by dumb terminals for way too long.

Dave Goldman on 2 October 2013 at 12:10 am

Greg, your use of blank lines between left-justified paragraphs is simply wrong. Proper paragraphs should be first-line-indented (5 spaces or 0.5 inches, depending on medium), without extraneous vertical spacing.

I’ll be happy to discuss this at length.

Greg Jorgensen on 2 October 2013 at 9:08 pm

@Dave I have compiled a list of references and style guides refuting your statement about indentation. I’ll be happy to send those to you separately and then arrange a series of meetings to discuss. I don’t have enough free space on my hosting account to upload all of the materials.

Gjorgji on 3 October 2013 at 7:33 am

Nice post, I had experience with a really Nazi-style senior java developer. I coded some functionality for couple of hours and then send the patch to integrate into the system. I can tell you that we spent good 1-2 hours, just on style of one comment.

A good engineer know how to apply good coding practices, bad developer just blindly follows them.

Vasiliy on 4 October 2013 at 7:49 pm

I haven’t seen any projects “go off the rails because of strict adherence to some practice, rule, or fashion”.

I have seen projects that became irrelevant during development or soon after, I saw a project that could not find enough customers. But I don’t remember any serious problem happened due to strict following of a rule.

Disclamer: I am not a big fun of coding style guides and rules.

Sherial on 6 October 2013 at 8:06 am

Thanks a lot for sharing this. It covers a lot of questions that were bugging me for some time. I started a blog recently about my journey to become a professional programmer, decided that it might be cool for people to see the entire process. Please let me know what you think: http://www.syntaxthis.com.

Paul on 11 October 2013 at 2:44 am

“I can tell you that we spent good 1-2 hours, just on style of one comment.”

@Gjorgji the more worthwhile discussion he should have had with you would have been, why is there even a comment their in the first place.

John on 16 October 2013 at 4:21 am

Modern IDEs make implementing a common style as easy as pie.

The successful projects that I have worked on have all insisted on (and implemented) a basic, common, style. I agree that spaces vs tabs and braces styles shouldn’t be the most important thing – but when the team can’t even agree on those how on earth will they agree on the more detailed stuff.

I’ve seen plenty of devs arrive on a well established and well run project and insist that they are an “artist” and should be allowed to choose their own style. It RARELY works out well.

Eclipse on 4 November 2013 at 11:17 am

With right tools such as Eclipse or InteliJ you can format all java files according to your style with one keyboard shortcut press. So this will not take you any time at all. Just initially 1-2 hours to setup common style.

Edward Patel on 11 February 2014 at 5:14 am

Think this also comes down to “open mindsets” vs “closed mindset”. Maybe it’s more “close mind” programmers around… http://aynrandlexicon.com/lexicon/open\_mind\_and\_closed\_mind.html

Angelito on 9 April 2015 at 8:01 pm

Excellent post.

Nowadays many developers spend more time discussing techniques (often unnecessary) instead of spending time writing really good code.

And if you do not use their techniques, is an amateur, beginner, asshole…

I’m your next potential dream boss. I run a cool, rapidly growing company in the digital field, where the work is interesting and rewarding. But I’ve got to be honest about some unfortunate news: I’m probably not going to hire you. … If you want to survive in this economy, you’d be well-advised to learn how to speak computer code. – Kirk McDonald, president of PubMatic, writing in the Wall Street Journal, 10 May 2013

This is another article beating the “everyone must learn to code” drum. Employers can’t find enough people with programming skills, schools aren’t turning out enough engineers, jobs at these cool companies are left unfilled because American students are too lazy or short-sighted to spend a summer learning “basic computer language.”

If only it was that simple. I have some complaints about this “everyone must code” movement, and Mr. McDonald’s article gives me a starting point because he touched on so many of them.

A dilettante is not a programmer

That isn’t writing at all, it’s typing. – Truman Capote

Learning a little bit of programming is probably worse than not learning it at all if you are going to be working with professional programmers. Anyone who has worked in the software business has had to deal with a manager or marketing or sales person who took a BASIC or Pascal class back in college. Although their exposure was shallow and short, they will refer back to it as if their mastery of a FOR…NEXT loop qualifies them to make decisions about programming projects. Consider Mr. McDonald’s example: “Suppose you’re sitting in a meeting with clients, and someone asks you how long a certain digital project is slated to take.” How exactly does knowing a little bit of Python help answer that question? I’d rather hire someone who said “I can’t tell you offhand but I will find out,” and then consulted people who can help estimate.

There are lots of people who have training and degrees in computer programming who can’t program: see Jeff Atwood’s article Why Can’t Programmers.. Program?. I can’t see how adding more amateurs to these ranks is a good thing. I specialize in debugging and fixing code that is abandoned or unsupported by the original programmer, so I see plenty of what happens when someone who half-understands programming tries to develop real programs.

Programming languages are not programming

Teach yourself just enough of the grammar and the logic of computer languages to be able to see the big picture. – Kirk McDonald

The grammar and logic of computer languages are not the big picture. Programming languages are not the hard part of programing; not even the most important part. Programming is methodically using software to solve problems — business problems, rocket guidance problems, selling ads online problems. Programming languages and APIs are tools to write programs: they aren’t programming. By analogy you know you are in a group of amateur photographers when they have $3,000 cameras but don’t know what an f-stop is or how it affects depth of field. Professional photographers talk about composition and lighting, not camera features.

Good programmers can construct, hold, and analyze complex models in their head. They can translate business requirements into code. They know how programming languages and software tools can be applied to solve a problem.

Not everyone has the talent or inclination to program

I never learned to skateboard. I can’t dance. I can’t play a musical instrument. I struggle learning foreign languages. I know people who can do those things well. They apply themselves because they are interested and motivated, in the same way I applied myself to computer programming, for years, until I got good at it. I used to tutor college students and teenagers, and I learned that very few people find programming interesting or fun. Students sit through the courses because they have to. Their experience paying me to write their Fibonacci sequence assignment while they tried and failed to understand recursion will be cited later in their career as programming expertise. Maybe they will estimate important projects based on that half-remembered exposure to Visual Basic.

I’m not saying programming should be exclusive. I’m all for the kind of curriculum upgrading and partnerships with business that Mr. McDonald supports. Anyone who wants to learn programming should be able to, and experienced programmers should help and mentor. But let’s not kid ourselves that everyone wants to program, or will have the combination of talent and interest to get good at it.

It’s setting people up to fail

Most people who try to learn to program are going to fail or lose interest. Telling high school and college students and job seekers that they don’t have a chance because they didn’t learn to “speak computer code” is creating a phony bar to entry. At best it’s a distraction, like business students forced to take a programming class they will forget about. At worst it’s another demoralizing episode—like trigonometry was for a lot of us— where students are told they must master something that most of them will find uninteresting and difficult and inapplicable to their life.

In most jobs some level of familiarity and comfort with computers and technology is required. But you don’t need to know how to write code to do most jobs. I’ve worked in ad agencies and most of the employees didn’t program but could do the job they were supposed to be doing just fine.

Programmers have worked hard to learn their skill

So the book can only be talking about a superficial familiarity, not a deep understanding. As Alexander Pope said, a little learning is a dangerous thing. – Peter Norvig

Suggesting that just anyone can learn programming in a summer spent with Python tutorials and getting “acquainted with APIs” trivializes the amount of real time and effort required to learn programming. Peter Norvig wrote Teach Yourself Programming in Ten Years, and I think he got it right. Malcolm Gladwell came to similar conclusions in “Outliers.” As a professional programmer with 35 years experience, much of it continuously upgrading my skills, I’m a bit offended at the suggestion that everyone and their mother can be programming after watching some videos online.

The idea that programming is something anyone can learn if they just sit down with a book and type examples is not just offensive to programmers—it’s a dangerously misleading idea for management to hold. It feeds the idea that programming is a rote skill that can be trained into anyone, and that programmers are therefore more or less interchangeable. Many of my friends and almost all of my colleagues are programmers, but I would flail for a while trying to do their jobs if I was rotated in as a new “resource” or headcount. Programmers tend to specialize. Our underlying expertise and experience, and our innate interest in programming, means that we can do another programming job better than someone fresh out of school, but we aren’t interchangeable.

So what do I tell my kids?

I tell my own kids to invest in learning useful skills, whether the skill is programming, plumbing, writing, or fixing cars. Learning any skill well enough to make a living from it takes a lot of time and commitment. I also teach them that credentials are not the same thing as expertise, and the more competitive the job market gets the more skills and experience will be valued over credentials, especially as we figure out how degraded a lot of credentials really are. I teach my kids that they can learn anything they are interested in, but I can’t make them interested in any specific thing. And I tell them to be skeptical of advice like “Everyone must learn to code.”

Comments

oldtaku on 12 May 2013 at 1:40 pm

For me the first major separator between a programmer and a non-programmer (even one who abuses code) is that they’ve internalized ‘If and only if A, then B’ and then chaining that. People don’t naturally think this way, and it takes real effort to learn. Once you’ve learned it, it then takes real effort to remember that most people don’t really grasp cause and effect, which is why you are having trouble communicating with them, and why his/her code is the equivalent of flailing around randomly.

Of course there’s much more (solving problems). But that’s a big one you are not going to learn just by watching some videos.

TechForProgress on 12 May 2013 at 1:54 pm

Just because McDonald argues that everyone should learn to code, does not presume that each and everyone of them will consider themselves an expert. I took a drawing class freshman year, but I don’t go into the design department saying that I’m an artist, and my opinion outweighs theirs. Those who do will get weeded out of their position appropriately.

Your argument that most of us don’t use trigonometry later in life so we shouldn’t learn it because its demoralizing is a fallacy at best. Not to mention how defeatist your argument of “well, you’re going to fail, so don’t bother trying” is.

We should all be exposed to the compendium of knowledge and see what sticks. Some people may not realize their affinity to programming until they’re exposed to it.

I’m not an artist but when the designer comes to me and asks me what I think, because of that intro drawing class, I can comment on proportion, lighting, and aesthetic. Because of that trig class, when I try my hand at home improvement, I know how to cut an angle into a piece of wood. I don’t presume to be a carpenter though.

The challenge of knowing what you don’t know is always present, whether you are ignorant, a novice, or an expert. But that’s just me.

-=jwk=- on 12 May 2013 at 1:58 pm

“This is another article beating the “everyone must learn to code” drum. ”

No it isn’t. It’s a plea for all students to have some idea how programming works, which sounds like a good idea to me. It’s not about knowing how to code an API. It’s about knowing what an API is , understanding the limitations and being able to listen intelligently while the real developers explain which functions the API is capable of. Mr McDonalds theses is that the best way to learn this is to try and write one yourself. I am pretty sure anyone who had (tried) to write a function call will understand the impact of changing it much better than someone who has not. I haven’t programmed since Turbo C for DOS (well, little SQL when needed) but because I understand the basic mechanics I am a much better manager for it.

Rex Kerr on 12 May 2013 at 2:04 pm

You make good arguments for why not everyone should be taught programming poorly. But you can remove everything you said about programming and replace it with “math” or “writing” and the logic would be unchanged.

Programming in today’s world should be a core competency so that people can better understand how the world around them works, and so that any time they need to free themselves from a tedious intellectual task, they have some hope of doing so.

Not all people who know how to write will be writers, but a literate society has many advantages over one that is not–you can ask people to send you information in writing, for example, and have some expectation that they actually can. Not all people who know how to do math will be mathematicians or economists or data analysis, but a numerate society has many advantages over one that is not–you can expect a basic level of efficiency from calculations, and outrage or acquiescence to political decisions to be made from a position of knowledge. Not all people who know how to program will be professional programmers either, but giving people more tools to take charge of their lives and be more capable is, if you can do it well, a huge plus.

That we can’t get math and writing instruction right is a problem. Let’s fix those, and then note that computation is ubiquitous and being able to instruct our machines is important.

Ruben Dias on 12 May 2013 at 2:53 pm

Perhaps it is not programming that we need to learn — after all, it is mostly about solving a problem with a given set of tools (be it the language, or whatever libraries you chose to use/learn). I would argue that we are not being taught in schools how to approach a problem and solve it. We are being taught that you may encounter this set of problems, and you may solve them with this set of solutions. If you’re clever, you can see that those solutions can be adapted to other problems, thus translating into problem-solving skills and not just “I solve this by doing that because I was taught”.

Unless the man does mean everyone should learn how to code. In such case, I mostly agree with you.

Brian Dunbar on 12 May 2013 at 3:25 pm

I’m a systems administrator, twenty years experience.

Now, I’m not a programmer, but I have spent my entire working life writing code to do stuff: systems management tasks, mostly, with the languages you would expect: perl, bash, and so on.

I like my job, and what I’m doing – bossing computers around is fun- but I have a small, growing, dissatisfaction with it.

In your experience, can people from a ‘sysadmin’ background transition to a programming one?

Timothy on 12 May 2013 at 3:36 pm

Anyone who senses that either both authors are correct xor both authors are incorrect, must appreciate what was not said but well understood: Good programmers are smarter than and frustrated by most everyone else. If programming is a skill in high demand, it is necessary to intersect smart people, potential interest, and skilled educators. In the context of programming skill, resources spent outside of the Venn is necessarily waste. The two authors only disagree on scope and perspective: the optimistic vs the pessimist.

Timothy on 12 May 2013 at 4:11 pm

@Brian, generally yes. That you are asking implies that you have not tried. Given opportunity in an ideal environment, not having tried implies lack of interest. Lack of interest translates to: No.

Why don’t you just bite the bullet and jump in? Sounds like you’ve got the basics of logic and tools. Maybe you could find a small simple open source project that interests you and seems to have competent developers; Read the code, scratch an itch. Or start with the hello worlds, sorts, graphics, then solve some curiosity, like calculating the dates of Easter, write an AI that plays tic tac toe, calculate the odds of poker hands, or rewrite one of your slow bash scripts in a compiled language.

kredit report online immediately afterlife on 7 February 2017 at 11:32 pm

Average Joe wrote, “Ren:The major differences between races and ethnic groups is that ethnic groups refer to subgroups within the same race. For example, Swedes and Norwegians are different ethnic groups who belong to the Caucasoid racial group. Because Swedes and Norwegians come from the same racial group they are going to be more closely related than different racial groups such as Caucasoids and Negroids.”This is circular, but somehow I don’t think you’ll understand why, even if I explained.

Ryan on 12 May 2013 at 4:55 pm

Steve Jobs says everyone should learn to program:

http://www.youtube.com/watch?v=mCDkxUbalCw

Danyel Lawson on 12 May 2013 at 6:53 pm

My 20+ year programming career resembles something closer to the theatre of the absurd. You should consider yourself lucky if your career hasn’t been one bad manager after another with most if not all either megalomaniacs or ill informed liberal arts majors or people who think speed to delivery is king.

The framework as programming language and requiring experience in one is the latest expression of a deep misunderstanding of what is the role of a software developer. No one would ever require a carpenter or plumber to use their layman’s opinion of what tools they think someone in that profession needs. But you can’t get work these days without proving you have experience using some neophyte’s latest version of their college project. That you could probably rewrite faster than you could learn about all the immature features and poorly thought through design principles.

In the current work environment professional software developers are treated to the new prototype tool of the week by competing software developers whose aborted attempts at delivering the penultimate tool based on the current meme of web server as multilayered over engineered object orientated database obfuscator for those with phobias about working with a database while simultaneously having no understanding of the mysterious inner workings of HTTP and a deep rooted aversion to working directly with POST data much less so any documented file format or protocol.

The framework AKA use my Dremel approach to programming requirements is only the first of many insults and obstacles flung in the path to delivering the optimal and correct solution to the programming task placed in front of a developer. The recurring theme as insult I have personally been bumping into for years from coworkers/managers is the I could have written what you wrote faster than you. Which completely ignores the fact that I just fixed said insulters code. Which is the first sign that this is not the work environment I was looking for/start interviewing now.

Brian Dunbar on 12 May 2013 at 7:15 pm

@Timothy

“That you are asking implies that you have not tried.”

Yes, and no. Over the last few years I have schooled – or tried to school – myself in interesting languages and tools: Ruby, ROR. I liked common LISP, Racket and Clojure. Dabbled with node.

“Why don’t you just bite the bullet and jump in? ”

Good question. I’ve done a half-dozen ‘apps’ that solve various itches. I don’t think I am, yet, good enough to contribue code that other people would use.

I have talked to a few companies about actual paid work: shopped my resume around. I’ve gotten nothing back, yet, except polite interest: sounds great, like your enthusiasm, we’ll get back to you.

Antoine on 12 May 2013 at 7:39 pm

Your concern about stating the nature of programming as the praxis of specialized knowledge is well-founded.

1. Bloated ego of coworkers/clients : things already are hard enough
2. Human mortality beats human curiosity : no-one has the time to be a true Jack of all Trades

These are fair and understandable.

Yet, WHAT ABOUT ANALPHABETISM ?

As I see, you define and defend “programmer” in the fashion of a writer with the term “writer”.

Algorithms are not merely a tool, they’re a cognitive process, a way of life, a unique angle of approch to problems, just like sentences.

1. I know words, and I humbly accept that I’m no writer ; it’s easier on my ego because alphabetism is ubiquitous.
2. I loved every second I spent learning words

Let : Analgorithmism, aka Analgorism = lack of knowledge of the algorithms.

Context : I need to communicate on/using algorithms.

Proposition : I won’t hire an analgorithmist.

Counting and reading are mandatory skills today. Algorithming (duh) tries to wiggle its way in the list. Programming shouldn’t, of course.

Algorithms are awesome. They deserve to be known.

Timothy on 12 May 2013 at 8:18 pm

@Brian “\[not\] good enough to contribute code”. Bullshit. Example: Timer I like didn’t store previously selected duration and wasn’t ready to ‘go’ again after stopping. Simple and appreciated solution but required getting into the guts a bit. Alternatively, make your day job tools better, maybe off the clock.

It doesn’t sound fair or realistic to expect an employer to hire you with the expectation that you’ll learn the trade on the job. Get your flying wings, build up some code, and then get a new job.

If I needed a mechanic/cook/artist/programmer and you came with no experience, the only way I’d let you into the shop to learn is if you were working for free. Which is not a bad idea actually, but if you wanna keep your day job, open source is a good education and resume builder.

Jhonson on 12 May 2013 at 8:26 pm

The major point missed is the ability to communicate with computer related people. Understanding and able to tell a programmer what is needed implies the understanding the fundamentals of programming. If you don’t know say, English grammar and composition, you can still talk but cannot write cogently to communicate with others. To communicate with an auto mechanic, if you can outline the problem in his language such as, I got the engine light on and it is time to change my timing belt is better than telling I have my engine light on which leads to cheating etc. So, instead of arguing about the merits of learning a programming language, algorithm, memory management(data structure) and some hardware knowledge, dismissing them as “not needed” is not worth any thing.

Pseudonym on 12 May 2013 at 8:27 pm

@Brian Dunbar:

Your apparent humility says to me that you’re exactly the sort of person who should be programming. We need more engineers, not more prima donnas.

Steve Nordquist on 12 May 2013 at 11:11 pm

Watch, it’ll turn out middle schoolers could only ever have been taught Perl6, J, obscure vertical market frameworks, and lawn tennis; the singularity will be set back 5 years straightaway. Because grass stains. Also, human advancement. Java RMI and VM will eventually be boiled down to better preschool toys (…than the mere 55-core 1TiB tablets of the day, somehow…) or 560-level courses, whichever’s fungible.

Benjamin Ondari on 12 May 2013 at 11:34 pm

Should anyone and everyone Know How to program? The motivating philosophy behind this push to get everyone into the coding bandwagon is simple.The belief that the person who is is experiencing a need or problem will most likely be the one who understands it the most and hence could come up with the most accurate solution.More often than not hiring a cs programmer to solve a business problem ends up with having to adopt a less than accurate solution as these techies end up creating cool but useless solutions.Its more or less like the story of blogging and web design,in the past a web presence was the preserve of the tech elite or those who could afford to hire someone to develop and maintain(including updating)their website,however with the advent of easy blogging software anyone with basic computer skill can build and maintain a blog,what has this created?we now have quality posts in all areas or expertise ,doctors,politicians,academia e.etc can share their knowledge online. I am a professional programmer by the way.

Zinc on 12 May 2013 at 11:49 pm

Danyel Lawson — my 35+ year programming career has been greatly enhanced by developers like you who hate working with badly designed tools and preexisting code, which in my experience has mostly been what the job is about. I’m perfectly happy to work with badly designed frameworks and make it work the way managers want, however convoluted, as long as they are paying my salary. Fact is, I’m darn good at it and that’s a valuable skill. Fad new techniques are just another means of micro-managing the job, something I’m perfectly willing to allow whoever is paying my bills to do. Could I do it faster, cleaner, more elegantly and efficiently? No doubt. And I do with that which is fully under my control, or when asked. Do I care if it is or not? Mainly to the extent that it makes my job easier, which does usually include ongoing support of code I’ve worked on. But if a manager wants to badly design the overall structure himself, I’ll offer helpful suggestions, but I see no compelling reason to stand in his way. He’s the boss, and if he wants to drag the project out years longer than necessary because he thinks he knows better how things should be done, I’ve no real problem with that. It’s my bread and butter, after all. Bailing him out of the situations he gets himself in is one of the things I’m good at, and in fact, generally enjoy.

Regarding programming as a “core competency”, I’m not sure I have an opinion on that. I would ask those arguing for it, what would be an example of a skill that you would argue should not be included in core competency? Rocket science? Brain surgery? Atomic physics? Motor vehicle repair? Computer repair? Basic electronics? Integrated circuit design? Why or why not?

Peter on 13 May 2013 at 2:44 am

Learning a small amount of something will either result in overconfidence (as you suggest), or a better appreciation of the complexities of the subject (which could be valuable). I’d look to spot the difference when it came to interview.

David Smith on 13 May 2013 at 3:05 am

Kirk MacDonald would probably get better results if he sought a few qualified IT project managers. He seems to have confused that with a passing acquaintance with high-level scripting.

Patrick Sinz on 13 May 2013 at 4:37 am

I would probably not want to work for M. MacDonald, but mostly because I’m not really interested in his business model. And I agree that knowing the syntax and grammar of Python is not “knowing how to program”. But I would agree that everybody should learn “how to program” at least somewhat. Not in order to be able to write some code, but to understand how to organize their work in general. If you are a manager in Logistics you should be able to think about “while something if this then that but there might be some contention on this resource so …”

If you are a sociology major and have no clue how to organize your data and do some datamining not only you should fail to get a degree, but you will be unable to do or to explain what you need to get to an IT person. Etc ….

You do not need to be able to fix your car to work as a taxi driver, but if you do not understand it you will be unable to choose the right mix between reliability, total cost of ownership, attraction to clients and cost, and each time you have a car problem, the mechanics will be happy to take you to the cleaner. (of course us IT people are nice an patient people, we would never bambouzle our internal client just because we can and are tired of their incompetence..)

Greg Jorgensen on 13 May 2013 at 4:51 am

@TechForProgress – Thanks for your comment. I don’t presume everyone can be an expert either. I just don’t think everyone needs to code to get a job, Mr. McDonald does. And I don’t say that we shouldn’t teach trig or anything else because we might fail. I object to using things like coding, or trig, as phony bars to entry for jobs where those skills aren’t really necessary. Mr. McDonald says that college grads who can’t program can’t get a job with his company and probably not anywhere else — that’s the phony bar I referred to.

@jwk – Thanks. My reading of Mr. McDonald’s article is that he specifically tells college grads that they need to know how to code at some level in “at least” two programming languages. That’s not the same thing as computer literacy or a higher-level understanding of software.

@Rex Kerr — Thanks. I’m not convinced that programming is a core competency. Computer literacy and some understanding of software and what it can (and can’t) do is a core competency for many jobs, just like math and writing. And there are many jobs where math or writing aren’t core competencies. Anyway I didn’t argue against anyone mastering math and writing — I’m all for that.

@Ruben Dias — Exactly, thanks.

@Brian Dunbar — In my experience anyone who wants to learn to program can learn. I also believe that some people are more inclined to be good at programming than others. Programming even moderately well takes a lot of commitment and time. If you have 20 years as a sys admin I’d guess that the logical thinking process and understanding software parts are already there.

@Danyel Lawson — Agree. Watch the old Cary Grant comedy “Mr. Blandings Builds His Dream House” for a laugh at the laymen running a project.

@Jhonson — A good programmer should be able to speak in plain language to non-technical people. Good communication skills are a core competency for programmers.

@David Smith — exactly.

@Patrick Sinz — I agreee that there are higher-level thought processes more fundamental than programming: organizing data, logical thought, thinking in discrete steps. A good programmer has mastered these skills and can use a computer language to express a solution.

Joe on 13 May 2013 at 5:15 am

I personally lean more toward the side that a little too much of the wrong knowledge can be dangerous. One needn’t look further than sites like TheDailyWTF to find countless stories of managers and people in power who make crazy decisions (and feel empowered to do so) by their limited knowledge of programming and how computers work.

But, perhaps that’s more a problem of ego and bad management in general, and had these people lacked their basic programming knowledge, they’d still be bad managers/supervisors/project leads/etc. They’d just screw something else up.

That said, I think there is huge value in stripping away at least a little of the “magic” behind computers. I hate watching people be victims of technology. I’ve watched so many manual, time consuming processes happen that could be automated with just a little bit of thought. It’s like the difference between hand washing clothes with a scrubbing board in a tub, versus using an automatic washing machines. Unfortunately, programming computers is a little tougher than sorting by fabric type, reading the care label, and dialing in a few knobs.

And maybe that’s another angle: There’s (at least) two levels of programming that are worth separating because they’re quite different, but they’re both “programming.” One is high level, basic automation. I want to automate a few basic tasks, hooking these pre-existing, complex pieces together. That is something I think more people should learn. The other is the lower level, “real” programming, that involves creating those pieces.

You can build things with Legos, but someone had to make the Legos.

Greg A on 13 May 2013 at 5:37 am

You’ve completely missed the point. A manager can know how to program and still be a bad manager, but not knowing how to program won’t guarantee that he will be a good manager. Teaching programming fundamentals to non-programmers is not a tool to allow you to replace your programmers. And you should learn how to skateboard or make music. As a general statement, I have invested more time and love into programming than anyone I know, and that probably includes you. If you are somehow even more dedicated to programming than I am (which I find laughably hard to believe, but you never know), THEN IT IS ALL THE MORE IMPORTANT TO DIVERSIFY YOUR EXPERIENCES. Dedicate some time to learning how to swing a hammer or hold a Sawzall. You might still hire a contractor when your bathroom floods! But it will make you a better person, and a better home-owner.

Bozon on 13 May 2013 at 5:59 am

My first job out of college (many years ago.) My boss asked me to write a program to sort some data a special way. I thought he was testing my ability, so I didn’t use the operating system included sorting algorithm. I instead wrote an optimized “Quick Sort” Sorting program taken from Knuthe’s book on algorithms. I was proud of myself. He then scheduled a meeting with me, and proceeded to lay into me because a sort could be written in a couple of lines of code, and no he wasn’t talking about calling the system sort routines. He was talking about writing a bubble sort that he had seen in a magazine print out. He literally thought I didn’t know how to code because I had used many lines of code to sort data when you only needed a couple. I tried to explain that the sort was fast and we needed it to be fast for the large amount of data we were sorting.

I literally almost got fired because my boss thought he knew how to program. I did leave as soon as I could find another job though.

David on 13 May 2013 at 7:18 am

Maybe everybody should learn English. The first sentence of the second paragraph, “If only it was that simple,” should have been written “if only it were that simple.” For more details see the Grammar Girl at

http://grammar.quickanddirtytips.com/subjunctive-verbs-was-i-were.aspx

I don’t know if there are other grammatical mistakes. I stopped reading the article after encountering this one.

JT on 13 May 2013 at 7:24 am

I don’t think learing to program is good because every one should be writing production code for their business. But in this day and age, not being able to program is like not being able to write 1900s.

You can get by, but you don’t really understand how the world works. People who can program will explain it to you, in metaphors, but the metaphors will break. Not being able to program means not being able to really use tools like excel. Not being able to program means not getting the most out of your mac (the CLI is programming). Yeah you can live without programming, but if you work in a digital world, you’ll be much better off if you can program. Sort of like if you worked at a newspaper in the 1900s it’d be kinder handy to be able to read, write and type.

Oh yeah I am no one you know. /. effect.

Greg Jorgensen on 13 May 2013 at 7:25 am

@David Thanks for noticing. It must be hard getting through the day frustrated over common usages that may be incorrect according to a website. It must be tough getting through Shakespeare and Hemingway, too.

I’m sure I made other grammatical mistakes but according to Google Analytics tens of thousands of people slogged through.

Yahoo Answers may not be as authoritative as Grammar Girl, but it did confirm that I’m in line with common usage: “… the subjunctive is disappearing and, at least in spoken conversations, the indicative, was, has gained popularity and is accepted as proper.”

Jon Nials on 13 May 2013 at 8:05 am

@Brian:

I’ve been a sysadmin for 25 years. I’ve also been a developer and managed developers during that time. I discovered that even though I consider myself a plodder as a programmer, some programmers I respect think I’m better than a lot of people I work with.

I think it is easier for a sysadmin to become a programmer than the reverse. We bring to the table some very useful mind sets: Change one thing at a time to see what happens, a certain amount of attention to details that is hard for some to learn, and a viewpoint of “how does this affect the whole system” that some times eludes programmers who are focused on just their piece of the puzzle.

Just find something that interests you and start working on it. However small your contribution it will make you a better programmer and sysadmin.

Josef Richberg on 13 May 2013 at 10:30 am

I learned programming way back in high school, because I found it fun and interesting. It was one of the few subjects that could hold my attention. It still holds my attention to this day, due mainly to he fact that there are so many different ways to do the same thing, in so many different languages (syntax differences).

I have taught numerous students through various organizations and not everyone has the ability to grasp the fundamentals of programming. Learning the syntax for a Python “for loop” does nothing to help you understand the problem was better solved with a “while loop”.

Feynman Fan on 13 May 2013 at 10:40 am

Dude! You’re awesome. I really like your analysis and agree with everything that you’ve suggested. I mostly like your segment because I like when people like McDonald make these kinds of proclamations and when people like yourself really call them on the validity of their proclamation and point out the real problems with their “policy”. Frequently the “policy” just really isn’t that good of an idea and may also reveal their own agendas or mindsets.

For example, like that programmers are interchangeable. I think that shows that he doesn’t like paying programmers and what he really wants to do is cut their rates or resents paying them at all.

And I also really like your replies as well and I like that practice as well in general. In your case you expressed your ideas, listened to the reader’s views and then responded. But your responses weren’t defensive they were very matter of fact and appropriate. Kudos to you again.

Finally, @Greg A? I don’t think Jorgenson has missed the point at all. He has read the article and expressed an opinion. And that opinion and belief is based on facts not ego. But the same cannot be said for your own posting. If you were half as good as you think you are then you would be quite talented indeed. Shouldn’t you be like running a multi billion dollar company or something? Or curing world hunger?

Ryan Hupfer on 13 May 2013 at 12:16 pm

I feel like you’re taking this article a little too personally. I know you can code and the idea of saying that everyone should possess your level of skill might seem a little ridiculous to you (and I would agree). But that’s not what Kirk is saying here.

The way I took this is that he’s actually saying how it would benefit everyone (not only hardcore engineers) to have a good idea of what’s going on under the hood of the technology they’re using everyday. The world we live in doesn’t require you to build a lot of things from scratch and knowing how things can work with each other to add value is a powerful skill to have.

Of course these typs of people don’t need to build APIs or write code all day, but they should know what an API is, why they’re so important and what they have to do with their business. I’m not going to kid myself and say that everyone can learn to code at level high enough to make it their career. But they should be able to hold a conversation with someone who can. (just like how an engineer should be able to hold a business conversation with a sales guy)

Coding is more of an art than anything and it requires a very unique set of skills and interest. But that doesn’t mean it wouldn’t hurt for non-engineers to be able to understand technology to a level that would help them work more efficiently with the engineers and product managers on their team.

Personally I love coding and I even took a job out of college where I coded all day long. After a year or so I figured out it wasn’t really my thing and I ended up moving over to sales and marketing. The skills I’ve learned from the days of coding have only helped me do my job better and I feel like it differentiates me in the market and puts me in the category that Kirk is talking about. Knowing a little code never hurts.

Greg Jorgensen on 13 May 2013 at 12:24 pm

@Joe — well put, thanks.

@Greg A — Thanks for reading and commenting. I think you are arguing against things I didn’t say. I have been programming professionally for 35 years. I spend quite a bit of time keeping my skills up to date because I’m way outside the 20-something age range employers prefer when looking for programing “rock stars.” I may never learn to skateboard but I am competent with auto and motorcycle mechanics, motorcycle riding, plumbing, and fixing things around the house, including with hammer and Sawzall. I don’t know how you got the idea that I’m one-dimensional just because I listed a few things I can’t do.

@Bozon — Thanks. I’ve had similar experiences myself, including almost the same argument over bubble sort half-remembered from college. I tried to explain to the person who brought it up that bubble sort is usually presented as an example of bad way to sort.

@JT — agree that not knowing how to use a computer to get things done can be a crippling deficiency in the job market, and knowing how to program is a good thing. But I don’t think we’re at the point that knowing how to program is a required skill for everyone, nor do I think Mr. McDonald’s “dabble for a summer in Python” approach is a valid solution.

Greg Jorgensen on 13 May 2013 at 12:26 pm

@Feynman Fan — Thanks!

@Ryan Hupfer — Thanks, I agree that computer literacy and some understanding of software and automation are a good thing and increasingly necessary for a lot of jobs. However Mr. McDonald specifically prescribed learning “at least” two programming languages and gave specific examples of what he thinks college grads should know.

TEVIS on 13 May 2013 at 2:50 pm

Thank you. As a graphic design student who wanted nothing to do with animation or web design i was forced to take classes in both as a part of a curriculum. It seems like it may be a good idea to know how those things work but the problem is you cant just take the parts you need. I had to learn programming in flash and how to debug html and install java components into a website. wasting time, money and a chair for someone who wants to be there.

hammer on 13 May 2013 at 4:07 pm

Does Kirk McDonald really think you can understand the fundamentals of computer science by taking a few courses or reading a few book? This is absolute nonsense!

It takes years of hard work and studying just to understand the fundamentals of computer science, and then lots of experience to become a competent software engineer.

The main reason there is so much terrible software is because there are too many incompetent software developers that don’t know the first thing about crafting well designed software.

Keith Bowes on 13 May 2013 at 7:06 pm

These are both interesting articles. I know myself that getting employed is nearly impossible (a lot of employers,not just Mr. McDonald, are defining unrealistic requirements, like a bachelor’s degree to work at McDonald’s). But getting to the level of programming that I do is something I don’t consider it hard work. It’s basically just a case of RTFM to get an idea of how the programming language and compiler work and then use basic logic to translate an idea to a solution through it.

Bruno Garcia on 14 May 2013 at 6:19 am

Perhaps if I could write this well, I’d end up with the exact same text as you! Couldn’t agree more.

richard wilner on 14 May 2013 at 6:47 am

This is the most articulate and well thought-out response I’ve seen to the “everyone should program” movement. It’s also the only article I’ve read that properly attributes the level of talent, experience, and craftsmanship that is a REQUIREMENT for any usable and valuable program.

Bravo.

Bob on 14 May 2013 at 8:08 am

I hear most of your arguments, but in today’s economy, credentials are being used first to weed out the mountains of applications coming in for most openings. The Digital Ad Exec that would hire you does value credentials first, experience coming in at a distant second.

hugov on 14 May 2013 at 11:55 am

Maybe I’ll refer more to science fiction than to close future. I am wondering if this debate is not finally about the idea that everybody should be a programmer with a second “major” in another field – like medecine, finance, law?

Greg Jorgensen on 14 May 2013 at 11:56 am

@Bruno Garcia — Thanks! I was afraid my grammatical errors would drive readers away. 😉

@richard wilner — Thanks!

@Bob — If Mr. McDonald or any employer is screening based on credentials first then a college grad with no programming classes but a summer spent “dabbling” with Python and APIs is not going to get through anyway. Mr. McDonald doesn’t say how he would measure the programming expertise of applicants, just that they have to know how to program. Even for experienced technical managers that’s not easy — see Jeff Atwood’s article I linked or Joel Spolsky’s articles in that vein.

John on 27 June 2013 at 6:52 am

I am a programmer and i have been interacting with lots of clients, over the years. Most of them want an excellent programmer in their fold. Most of the time, I find non programmers leading the so called IT department and those who know a bit are wise enough to advise their employers to get the programming task outsourced.

In order to become a good programmer, the basic ingredient is to be excellent at problem solving, good mathematics skills always help. Lots of top tech companies like IBM hire a fresher on the basis of their problem solving skills and not their knowledge of various syntax of a programming language.

So having problem solving skill is a must.

Programmers love to optimize their code and their tools. Why then is there so much slow and bloated code and so many arcane and fragile tool chains? Because programmers aren’t always optimizing for efficiency. There are many things programmers can optimize for, and a team can quickly end up working at cross purposes when everyone is optimizing for something different. Here are just a few of the things programmers optimize for:

Speed, memory usage, efficiency

What most of us think we mean when we talk about optimizing: making the code as fast and efficient as possible.

Conciseness

There’s a line between concise expression and obfuscation that we recognize when we see it, except in our own code. If you can’t easily understand someone’s code because it’s too terse, and they defend it by citing Occam’s Razor, that line has been crossed.

Complexity

The flip side of conciseness. Some programmers enjoy making code more complicated than necessary, but usually this optimization is a side-effect of forgetting YAGNI (You Aren’t Gonna Need It) or going off into the weeds with a new language.

Obsessive-Compulsive Disorder

When a programmer spends a lot of time lining up comments, converting tabs to spaces, converting all variable names to a single style, etc. they are optimizing to relieve their OCD. Converting all variables to Hungarian notation and forcing the team to comply “for consistency” is an extreme form of this optimization and may require an intervention.

Showing off

Looking ahead to the next code review or scrum, the programmer writes code that they can show off to the team. The code will be just clever or obscure enough to require the author to explain it to everyone and show how clever he or she is.

GitHub, StackOverflow, etc.

Showing off in public, and usually looking forward to either adulation or picking a fight in an online forum.

Job security, technical debt

Code is written with an eye to making maintenance and future enhancements difficult for anyone else. When a programmer is recognized as the only person who can work with a code base they are guaranteed employment.

Ritual, ideology, purity

Everything must comply with a methodology or fad. Programming decisions are defended by appeals to the authority of scrum, TDD, functional programming, etc.

Passive-aggression, being right

Failing to persuade co-workers to do it their way, the programmer changes a big chunk of code over the weekend to force the team to come around. Replacing a crucial piece of the testing or build process with something written in that cool new language is an example of this optimization.

Style, usability

Nothing will be released until everything conforms to the opinion of an expert. Tufte and Nielsen will be referenced.

Novelty

The project is optimized to make a new language or tool seem like a good solution. Programmers read about something cool online and suddenly the project can’t succeed without it.

Testing

The code is optimized so all unit tests are green.

CYA

Not wanting to be blamed for anything leads to code that can be defended against any criticism.

Meetings

Creating roadblocks and diversions in the code to force a meeting about the new problems. Meetings are an opportunity for schedule and specification changes.

Marketing

The code is optimized to meet ever-changing marketing requirements. These are frequently in conflict with each other, to say nothing of core business requirements, leading to HAL 9000 syndrome.

Ignorance

Code has to be written so the least-skilled programmers can understand it and work on it.

Comments

Alex, 12 May 2013 at 6:38 pm

Testing The code is optimized so all unit tests are green.

Isn’t this part of expected optimizations? Along with speed and efficiency it should also, ya know, work.

“If it’s not tested, it’s broken.” –Bruce Eckel

Danyel Lawson, 13 May 2013 at 5:05 am

The road to programming hell is paved with the quotations and rules of thumb of self proclaimed experts. Not one of which seems to be at acquainted with the scientific method.

Scott McClure, 20 May 2013 at 10:38 am

Frankly the quote from Mr. Eckel makes sense. Programming does not fall in the purvey of the scientific method. We are not formulating a hypothesis, we are writing code to solve a problem.

John, 27 August 2013 at 5:52 am

From my experience I can say that, the most visible reason for the programmers not seeking to optimize their code is plain laziness and then many a times there are stifling deadlines, which give little time to organize and optimize the code.

Greg Jorgensen, 13 September 2013 at 11:51 pm

@Alex, this article is satire, and what I’m poking fun at is the kind of TDD that only does unit testing and cherry-picks tests to get that green light after a build. Nothing necessarily wrong with TDD but passing all of the unit tests doesn’t say much about whether the app is bug-free or correct, nor does it insure the app meets requirements. It’s just one dimension of development, and if programmers optimize just along that dimension they are doing it wrong.

I was in a coffee shop in Portland, Oregon and happened to spot Linus Torvalds sitting alone at a window table. I asked the creator of the Linux operating system and the Git source code control system if I could join him. Over the next fifteen minutes we talked about programming and programmers.

Typical Programmer: It’s been 20 years since Linux was released. Now it’s one of the most widely-used operating systems. How does that make you feel?

Linus Torvalds: Surprised, frankly. It was a hobby project I didn’t expect to do much with. As interest in Linux grew I saw it used mainly by the hardcore programmers and computer geeks to separate themselves from the herd. There are a lot of people in the software industry who like to show off that they’re using the latest software or programming language. It’s a status thing, like the people who talk about obscure indie bands or foreign movies. Whether it’s Linux or Haskell or MongoDB or whatever, every workplace has at least one guy who spends most of his time talking about closures and how he is moving his blog to NodeJS so it will scale.

Linux is mainstream now. Did it get too easy?

I don’t think it’s any easier, but there are a lot more resources now. In most ways Linux is more complicated to learn and use than Windows or MacOS. The people using it for servers were already used to Unix so it was no big change for them. They were used to conflicting updates and dependencies and shared library hell. On the desktop I think people lost interest after a few years.

Lost interest?

No one is excited anymore about transparent console windows or Kate color schemes or being the first to post on Slashdot how to get some weird sound card to work. That’s what got the early adopters to switch from Windows to Linux. Now Linux looks like Windows. I can install Ubuntu on my grandmother’s laptop and she wouldn’t know the difference, as long as there’s a Facebook icon on the screen.

What about all of the Linux distros out there? It seems like there are more distros than Linux desktop users.

There are more Linux distros on a single Linux Format disc than Microsoft has versions of Windows. But they’re all pretty much the same thing warmed over. Only the clever and cute names distinguish one distro from another. Once a Linux For Dummies book was published I started to lose interest in it.

You released the Git distributed version control system less than ten years ago. Git caught on quickly and seems to be the dominant source code control system, or at least the one people argue about most on Reddit and Hacker News.

Git has taken over where Linux left off separating the geeks into know-nothings and know-it-alls. I didn’t really expect anyone to use it because it’s so hard to use, but that turns out to be its big appeal. No technology can ever be too arcane or complicated for the black t-shirt crowd.

I thought Subversion was hard to understand. I haven’t wrapped my head around Git yet.

You’ll spend a lot of time trying to get your head around it, and being ridiculed by the experts on github and elsewhere. I’ve learned that no toolchain can be too complicated because the drive for prestige and job security is too strong. Eventually you’ll discover the Easter egg in Git: all meaningful operations can be expressed in terms of the rebase command. Once you figure that out it all makes sense. I thought the joke would be obvious: rebase, freebase, as in what was Linus smoking? But programmers are an earnest and humorless crowd and the gag was largely lost on them.

What do you think of github?

It started as a place for mothballing unmaintained and unnecessary projects, and that is still most of what is hosted there. But it’s turned into a kind of World of Warcraft universe for programmers, where they are ranked by their commits and which projects they have trunk privileges on. I read about a recruiting company built around the idea that github reputation means something, so I guess if you aren’t committing to github you won’t be getting a job at the coolest startups. The good old days of writing FizzBuzz and moving Mt. Fuji during your interview are over.

It sounds like you’ve soured a little on Git.

The first Git For Dummies and Git Visual Quickstart books are going to be out in a couple of months, and that is the beginning of the end as far as I’m concerned. Those books mean the end of git expertise and github reputation as reliable indicators of geek status. Once a technology is adopted by the masses the extreme geeks find something more esoteric. Look at what happened to Ruby on Rails. The people stumbling their way through Rails to-do list tutorials have never even heard of DHH.

So what’s next?

I’m not sure. It’s getting hard to predict the next technology fashion. I have a text editor I’ve been using myself that is so complicated it makes VIM look like Notepad — maybe I’ll release that.

Linus finished his coffee and had to leave. I appreciate his time and for making me think about Linux and Git in new ways.

Comments

Lavneet Sharma, 20 September 2013 at 5:38 pm

great programmers can be spotted just by their talks.. :p Amazing.. and yes. his grandmother won’t come to know about OS until she looks at facebook icon. I don’t understand that? Is facebook icon different for various OS? well whatever, the guy is awesome. 😉 Hats off !!

X, 30 November 2013 at 2:22 am

Never heard of “Linux operating system”. It’s a kernel!

Vlad, 1 December 2013 at 9:13 am

Now this sounds… strange. Linus argues about DHH, Rails, WoW and… ‘I have a text editor I’ve been using myself that is so complicated it makes VIM look like Notepad’.. wtf? uEmacs/PK is well known and already released many years ago. Humm.. sound like a fake interview. A little.

joe, 18 December 2013 at 7:40 am

Its a fake interview folks.

shackra, 21 April 2014 at 9:20 pm

It is horrible how Torvalds takes all the credit for himself and his helpers when he talks about Linux as the entire GNU/Linux Operating System. On the first paragraphs my mind was centered on the kernel, but when he just re-named the thing when doing a comparison between GNU/Linux to Windows and MacOs, I totally felt shame of be born the same day and month that him.

J Harton, 24 March 2015 at 12:14 am

@shackra

What’s /truly/ horrible is how little credit he gets for his work. There’s a bunch of idiots like you out there happy to ride the bandwagon and bash Linus. The fact of the matter is that the kernel isn’t something you can do without, while anyone can write user space tools. The GNU tools might be lovely but aren’t any good on their own without a system to run on/within. Yes the programmers who wrote them deserve some credit, but to say they deserve most or all of it and belittle the contributions of Linus and others is ridiculous. — also everyone except you nuts and band wagoners realizes, at least dimly, that Linux as a term refers to the sum of it’s parts. Also aside from it’s somewhat questionable implications, “GNU/Linux” doesn’t really roll off the tongue nicely.

Andras, 6 April 2015 at 12:50 pm

I’m interested in the editor (used VIM for over a decade). Bring it on, I’ll try it!

Greg Jorgensen, 16 May 2015 at 10:37 am

@shackra I think you missed that this is satire, not a real interview with Linus.

@JHarton I think it’s a stretch to say Linus gets little credit for his work. Most people don’t care about the difference between kernel and GNU userspace tools, the whole thing is an OS to them. Trying to call it GNU/Linux is pedantry, like saying OSX/BSD. As an old-time Unix user I could get pissed off about confusing Linux and Unix, but for all practical purposes they’re the same.

This article is satire and poking some fun, but it’s not poking fun at Linus.

Most pingbacks of all of my articles

Pingback: In the News: 2012-09-26 \| Klaus' Korner (Edit)
Pingback: Linus Torvalds on Git \| Lazycoder (Edit)
Pingback: Quand Linus Torvalds se lâche sur Linux et Git \| Dico Micro (Edit)
Pingback: heh \| Pearltrees (Edit)
Pingback: Linus Torvalds on Linux and Git « Accidents Happen (Edit)
Pingback: No technology can ever be too arcane — The Endeavour (Edit)
Pingback: pinboard September 27, 2012 — arghh.net (Edit)
Pingback: Linus Torvalds goes off on Linux and Git \| Brent Sordyl's blog (Edit)
Pingback: Git (revision control): What did Torvalds mean when he said: "all meaningful operations can be expressed in terms of the rebase command"? - Quora (Edit)
Pingback: Sysadmin Sunday 100 - Server Density Blog (Edit)
Pingback: Weekend WebOps #1 \| Monitive Blog (Edit)
Pingback: Random Sauces 2 \| ikt (Edit)
Pingback: Getting Git Wrong \| daihuws.me.uk (Edit)
Pingback: Linus Torvalds goes off on Linux and Git - jondavidjohn.com (Edit)
Pingback: Linux News Watch \| Typical Programmer – Linus Torvalds goes off on Linux and Git (Edit)
Pingback: Linus Torvalds goes off on Linux and Git \| Enjoying The Moment (Edit)
Pingback: Visto nel Web – 108 \| Ok, panico (Edit)
Pingback: Perché ho detto No all'acquisto del Nexus 5 - Lorenzo Setale (Edit)
Pingback: COMPUTER MAGAZINE – \[PDF\] – Linux Format – SEPTEMBER 2012 (UK) \| My Blog (Edit)
Pingback: On text editors \| Blog of Leonid Mamchenkov (Edit)

In the old days when I started programming, green programmers trying to build their skills and get experience started out doing maintenance programming. Only the old hands got to write new code. The newbies cut their teeth debugging and fixing musty old code that still worked for the business. I’ve done my share of new development, but today I do maintenance programming almost exclusively. After working on way too many failed ground-up “green fields” projects I prefer to debug, fix, and enhance production software. I like working with real users rather than making up “use cases.” I prefer building confidence and making my clients happy with my work rather than arguing with them about features, specs, budget, and schedule. I like having a prioritized list of well-defined tasks that refer to existing code rather than trying to blue-sky specifications. I like charging by the hour instead of by the project, and not having clients balk at my rate.

I mainly work with web applications now, and I specialize in database-backed web sites. I rarely take on new development projects, but I frequently get called to salvage projects that went bad. Clients fire their developers (and vice-versa) leaving production code orphaned. Developers put their energy into new development and don’t respond to bugs or enhancement requests from old clients. I charge by the hour and I get the job because I don’t tell clients they need to throw everything away and start from scratch — something they get from other consultants.

Software often stays in use longer than anyone expected when it was written — until recently I supported a law office billing system that was written in 1986 using OMNIS 3 and wouldn’t run on any Macintosh newer than an SE/30. Clients who depend on legacy systems will pay plenty to keep those systems running, because they can’t risk their business on new software, and they can’t afford to fund new software development and the subsequent data migration and training. There’s a rich vein of maintenance work out there that most programmers turn up their noses at, preferring ground-up development projects that neither they nor their client are well-suited for.

Some of the benefits of maintenance programming:

You’ll learn how to debug

Debugging someone else’s code requires more effort and rigor than debugging your own. You have to track down problems step by step; you can’t assume you know what’s wrong and jump to a quick fix. Reading and debugging foreign code makes you think about how the code works (or doesn’t) rather than what you think it should do.

You’ll learn how to program better

You can debug and extend good code easier than bad code, so you will learn how to tell them apart. You will have to reason about the code and deduce the original programmer’s intent. Skilled programmers recognize what makes good code good, and what makes bad code bad, and working with lots of code will impart that skill. You will develop a knack for finding your way around an unfamiliar code base and recognizing bad code smells.

You’ll learn how to optimize

The original programmers already made the premature optimizations, so you get to identify and fix real efficiency problems and performance bottlenecks. Ground-up development project teams often spend a lot of time and energy worrying about performance issues that may never materialize. Working systems have real, identifiable performance problems. Finding and fixing performance problems can be fun, and it’s a great skill. Clients always notice and appreciate you making their application more efficient.

You’ll learn new (to you) skills and tools

You may not get to work with the language du jour, but you will get to learn languages, libraries, tools, and production environments you probably wouldn’t learn otherwise. You’ll find a lot more work involving common, mainstream business tools. Ground-up development projects often bog down just deciding which language and tools to use. You won’t have to make those decisions, sell them to your client, or get blamed for them. You’ll spend time learning some obscure and proprietary tools and languages, but those can pay off too. Larger companies with their own IT departments do their own maintenance work, but often hire contractors with specialized knowledge and experience.

You’ll learn to value data and databases

Applications still typically follow the input -> process -> output model from old data processing textbooks. You have to understand the data and how it moves through a system before the code makes sense. Almost all applications use a database of some kind, usually a relational database management system (MySQL, PostgreSQL, Oracle, SQL Server, etc.). A skilled programmer knows relational concepts and how to navigate a production database. Not understanding databases cripples programmers and leads them to write lots of bad code to do what the DBMS should handle. I frequently refactor pages of slow, buggy application code away by replacing it with SQL.

Clients will appreciate everything you do

Every bug you fix and enhancement you make improves an application people already use. Real users will notice the changes and give immediate feedback. When you develop new software from the ground up every spec change, delay, and additional expense can cause conflict with your client. When you address actual problems everything you deliver makes your client happy.

You will have better specs and clear constraints

Fixes and enhancements are smaller and better-defined tasks than ground-up development, and the existing code and environment constrains your choices. You have to focus on solving a real problem. You and your client can agree on the tasks with less chance of misunderstanding. With new clients I build the relationship by asking them to list just the top five pains they have with their software, then I offer to fix those, and promise not to charge if I can’t.

Easier to walk away from bad clients

We all get involved with clients we don’t want to work with. It’s a lot harder to get out of the relationship if you have a contract for a big development project, time and money invested, and the big expectations clients always have for new systems. Maintenance work is shorter-term and the tasks are smaller, so you have more opportunities to terminate the relationship without breaking a contract or either side losing money.

Agile by default

Fixing bugs and making important enhancements let you work in agile mode without having to sell a methodology to your client. They will want things fixed right away, so you can make incremental releases, collaborate with real users and get immediate feedback, and work with a few bite-size tasks with a defined finish line.

You’ll learn about business

By definition, production software addresses at least some real business need. Learning how it does, and does not, work for the business, and working with real stakeholders familiar with the software forces you to learn something about the business. Knowing something about business applications will boost your career more than learning a new programming language.

You’ll learn people skills

Programmers in ground-up development mainly work with each other. New systems don’t have real users; instead you get use cases and management stakeholders. Maintenance programmers get to work with real users, often the people on the floor who work directly with the application and experience the bugs and where the software doesn’t match the reality of their job. Learning to listen to and talk to people who don’t have the same assumptions and culture of programmers broadens your perspective and gets you out of the programmer shell.

You’ll get follow-on business and referrals

Who do you think gets more repeat business and referrals: a great car salesperson, or a great mechanic? People need their software fixed and enhanced more often than they need a new system developed. And your satisfied clients will send referrals your way because they have friends and colleagues who have broken software systems too.

Your work won’t be off-shored

But you may end up maintaining the results of your client’s off-shoring experiments.

Comments

WC, 27 February 2011 at 2:37 am

Good post! I chose the ‘bugfix’ position at my previous job because I really liked it. (Nobody else wanted it, though.) There’s a thrill in hunting down a bug and fixing it, and proving it was fixed. And the code is always better for it afterwards.

BTW: The input element for ’email’ is named ‘liame’ so non-programmers can’t submit comments.

joe Lee, 28 February 2011 at 6:58 am

Thanks for the post! It’s refreshing to see a varied view point to the typical attitude towards maintenance programming.

I completely agree with many of the points above, especially the point of “You’ll learn to value data and databases”, I think “backend” developers who say ” I rather not write sql” rather surprising since that’s what most data interactions with a data store boil down to!

I also think you can potentially learn how to program better as you may be expose to a better way of doing things, but at the same time you can be exposed to a bad way and pick up the same bad habit.

Manu, 28 February 2011 at 6:25 pm

I totally agree with all the points you have mentioned. If you are in consulting maintenance must be very lucrative. I would also say that a lot of maintenance work does get outsourced (for example development of the older releases of a product would get outsourced so the in-house team can focus on new releases). Many programmers might not consider maintenance as glamorous but it has several useful advantages mentioned in this post.

It is also worth mentioning that one should try to do a mix of maintenance and new development. Such a profile would be a killer profile. Maintenance will teach you how to write good code but new development will help you put into practice what you learned. Otherwise, that just remains good theoretical knowledge in your mind. I have been doing more maintenance work in a product development company (I sure do not regret it) and started looking for jobs at startups or young companies with small development teams. Such companies prefer that you have done more new development.

I’ve been freelancing for over ten years, sometimes moonlighting when I have a full-time job, sometimes doing just freelance work. I’ve learned a few things about successful freelance progrramming.

Do what you know how to do

Clients pay you to solve their problems. They aren’t interested in how cool Haskell is, and they don’t want to pay you to learn new tools and languages. I can’t count how many projects that I’ve taken on after the original programmer quit or was let go because he got in over his head. Expand your skills to stay competitive and don’t fall in love with languages du jour for their own sake.

Stay with the herd

If you have an exotic in-demand skill by all means use it. But if you decide to learn Lisp because you see Lisp freelancers making top dollar, think hard about how you will compete with them. Sure, there are lots of PHP programmers out there, but most of them are amateurs you can easily compete with, and there’s more than enough PHP work to go around. It’s much harder to compete with skilled and experienced experts for a small number of jobs. You want to do what everyone else is making money at, but better, so you’re playing basketball against a team of sixth graders, not trying to join the NBA.

Don’t try to do everything

Be careful you don’t get buried doing technical support, hardware installation, design, art production, etc. Stick with your competencies. Find other freelancers who specialize in the things you don’t do and send them business, and they will do the same for you.

Don’t take jobs that are too big

If the project is too hard to describe on a sheet of paper, or it has a schedule spanning months or years, be careful. Big projects are usually too much for one person. If you are tempted to take a big project try to break it into smaller projects with shorter-term deliverables so you and the client see progress and accomplishment.

Avoid bad clients

If the client seems angry, bossy, stingy, greedy, abusive, too demanding, too controlling, etc. step away. You will find other clients. If you can’t imagine a professional and rewarding relationship with your client, or if you feel dirty about whatever it is they are doing, say no.

You will get most business from referrals

Do a good job, leave your clients happy, work with other freelancers and develop a network of people who will recommend you and send work your way.

Don’t bother trying to look like a “real” business

Plenty of entrepreneurs get caught up accumulating the trappings of a big company. You don’t need a fancy web site, letterhead, office, etc. You probably don’t even need to form a corporation or LLC. Unless you have employees or manage to bill for 72 hours every day you won’t make enough money to justify anything elaborate or supposedly “professional.” Your clients won’t care that you work from home and don’t have all the trappings.

Use source control and issue tracking

Learn how to use source control and web-based issue (bug) tracking. Keep a wiki for each project documenting the requirements, questions, and decisions. Use the tools yourself and get your clients to report bugs through the system. I use ProjectLocker, there are other cheap hosted solutions out there. Don’t waste your time setting this up on your own servers.

Keep track of your time and bill regularly

The best way to get paid is to keep track of your time accurately and bill your clients regularly. I prefer to work hourly and send invoices every two weeks. You don’t need a full-blown accounting system, there are hosted time tracking and invoicing solutions like BlinkSale that do most of the work for you.

Don’t subcontract

I recommend working with other freelancers, but I don’t like subcontracting work out. Chances are you will spend almost as much time explaining the project and reviewing the work as you would spend doing it yourself. Any programmer as good as you will cost as much anyway; if you think you are getting a bargain with a $12/hour programmer you found on Craigslist, good luck with that. If you take on more work than you can handle try referring it to another freelancer who will return the favor.

Use agile development techniques

Clients like to see progress. Misunderstanding requirements is the most costly mistake you will make. Agile development, with short-term deliverables and frequent reviews will keep your client happy and keep you from going off in the weeds. You don’t need to sell your clients on agile, they don’t care anyway. Just tell them how you do things and what you need from them.

Keep your clients informed

Post all work online so your client can see what you’re doing. Programming is mysterious enough to most people, so keep your work visible so clients feel comfortable with you. I’ve found that an agile, transparent process heads off billing disputes, and once clients get used to your high customer service standards they are less likely to give work to other freelancers.

Comments

Alex, 2 April 2009 at 9:30 am

All nice tips, but there is one missing: How to get the first clients! 🙂

ricky, 20 May 2009 at 6:55 pm

Nice. I couldn’t agree more with the subcontracting bit. I’ve been preaching that for years (my old boss was like, “I want you to get out of the mode of coding/designing and more into subcontracting, but I set him straight by taking his clientele)

DynV, 23 January 2010 at 2:17 am

Very good tips. I found some of those the hard way, getting bad clients or over your head really suck !

In his November 24 article Database Thaw Martin Fowler wrote about the lock-in of relational/SQL databases and some possible alternatives. I was with him until he wrote that “you can’t get \[a\] bigger breach of encapsulation than that,” referring to a central database shared by multiple applications. (The indented quotes below are verbatim from Fowler’s article).

For many organizations today, the primary pattern for integration is Shared Database Integration – where multiple applications are integrated by all using a common database. When you have these Integration Databases, it’s important that all these applications can easily get at this shared data – hence the all important role of SQL. The role of SQL as mostly-standard query language has been central to the dominance of databases.

That’s a fair argument, though I don’t agree that SQL has been central to the dominance of databases, or central to making shared database integration the “primary pattern” for integration. Before SQL-based databases took over in the mid-1980s, integration through shared non-relational databases was already the norm. SQL standardizes how users and application get data in and out of the database, and how database schemas are described, but it’s just a language that more or less replaced many proprietary and unstandardized languages that did the same thing.

The heating of the database space comes from the presence of alternatives to integration – in particular the rise of web services. Under various banners there’s a growing movement for applications to talk to each other by passing text (mostly XML) documents over HTTP. The web, both in internet and intranet forms, has made this integration mode even more prevalent than SQL. This is a good thing, I’ve never liked the approach of multiple applications tightly coupled through a common database – you can’t get bigger breach of encapsulation than that.

I’ll go along with the first part, about applications talking with XML over HTTP. I hesitate to call that talking “integration,” though; it’s not comparable to the kind of integration trusted to databases. I’ll talk more about that point later. What jumped out at me was the statement “you can’t get a bigger breach of encapsulation” than a shared database. That struck me as a misunderstanding of what a shared database is for, and particularly a misunderstanding of how a relational/SQL database should be used.

All modern relational databases engines, with the arguable exception of MySQL, are intended to be more than dumb backing store. Database engines are designed to encapsulate data and the valid operations on the data, and to guarantee data integrity to multiple instances of multiple applications accessing the data simultaneously. Large enterprise databases are usually implemented like that. Using those databases, programmers don’t send raw SQL SELECT and UPDATE statements to the database server, and they don’t serialize objects into dumb tables for convenience. The API offered by the database to client applications can and should be the same kind of API you’d expect in a good object-oriented library: well-defined operations on abstract objects, with the underlying data and its representation invisible. If your code is doing this:

INSERT INTO invoices (customer, date, amount) ...;
UPDATE customers SET balance = balance + amount;

and not something like this:

NEW\_INVOICE(customer, date, amount);

you are doing it wrong. A shared database should not expose the invoices and customers tables. Instead there should be an API implemented with stored procedures in the database, or a common data access layer written in a non-SQL procedural language to hide the underlying data representation.

I’ll restate Fowler’s concern like this: You can’t get a bigger breach of encapsulation than spreading knowledge of the data and operations on it across multiple applications.

It isn’t the database engine, or the concept of relational databases, or SQL that is the breach of encapsulation. It’s that far too many programmers are either unaware of how to use a database to promote encapsulation, or they don’t see the problem. My own experience is that a lot of programmers learned relational databases from MySQL, which until recently didn’t support things like transactions, stored procedures, views, and triggers, which are required to make a database into something more than a convenient backing store.

The view of databases as something that promotes bad software implementation is common. Ignorance of how relational databases work and how they should be used to make application integration easier and more robust is widespread and growing, evidenced by the ongoing “alternative database” movement that trumpets every reinvention of old ideas (key/value pairs, network databases, hierarchical databases, etc.) as something new and revolutionary. Fowler continues:

If you switch your integration protocol from SQL to HTTP, it now means you can change databases from being Integration Databases to Application Databases. This change is profound. In the first step it supports a much simpler approach to object-relational mapping – such as the approach taken by Ruby on Rails. But furthermore it breaks the vice-like grip of the relational data model. If you integrate through HTTP it no longer matters how an application stores its own data, which in turn means an application can choose a data model that makes sense for its own needs.

I can’t imagine a worse breach of encapsulation than integrating multiple “application databases” over HTTP. And I can’t imagine a worse threat to data integrity. If each application can have its own copy of data, and its own rules for operating on the data, either the applications are going to have a lot of duplicated code and data (a violation of the DRY principle to say the least), or each application’s copy of the data will be possibly wrong and untrustworthy.

SQL is not comparable to XML over HTTP, for reasons that should be obvious, but apparently aren’t. SQL is a declarative language for describing legal operations on a database. XML over HTTP is a data transport mechanism. XML+HTTP can’t address the problems already solved by relational database engines: integrity, consistency, reliability, transactions, locking, and so on. All of those things that are important to real applications will either be ignored or implemented over and over again, many times incorrectly.

Fowler concludes:

I don’t think this means that relational databases will disappear – after all they are the right choice for many situations. But it does mean that now application developers should think about what the right option is for their needs. As non-relational projects grow in popularity and maturity, more and more will go for other options.

I’ll agree that relational databases are not the right choice for everything, and that there are applications that don’t need to care about data integrity or encapsulation or sharing across applications. I hope that when developers look at those “other options” they carefully consider what they might be giving up going down the non-relational path. It will help if developers making those decisions have a serious understanding of relational databases and the powerful features of modern RDBMSs, and don’t just dismiss a successful and mature set of tools as an obsolete holdover from the computing stone age.

I’m always interested in what might be happening in the world of databases, which is why I read Martin Fowler’s article in the first place, but if this is what is happening — a giant step backward in data integrity and encapsulation — I am going to keep my relational database and SQL skills sharp for the coming Rails/MySQL, XML+HTTP, “application database” apocalypse.

Comments

Mike, 5 December 2008 at 3:16 pm

Hi Greg,

I agree with most of what you have to say here. Databases should encapsulate your business logic across multiple application languages, as that is what they are there for: to define and protect the data model. Unfortunately, as Rob said, the languages (SQL and its extensions) used by the various database vendors is too limited to accurately reflect this business logic in any meaningful, discoverable way.

One possible solution to this is CodaServer, an open-source tool I am developing. You can get it at http://codaserver.com.

It uses a SQL-like language that provides much richer application metadata and includes the ability to define workflow at the schema level, so application languages can discover it easily and use it to build smart user interfaces. (Be warned: I am uploading a new release tonight that fixes a pretty serious bug, so it may be best to try downloading tomorrow. 🙂

I do support Martin’s idea of an application orientation for data, but think that this integration needs to happen over (as you said) well-defined interfaces.

-Mike

Greg Jorgensen, 5 December 2008 at 4:13 pm

Mike: I agree that PL/SQL and T-SQL are not the greatest languages for implementing business logic. But Oracle and SQL Server support other languages for writing stored procedures, and I believe DB/2 and PostgreSQL do as well.

In my article I wrote “… there should be an API implemented with stored procedures in the database, or a common data access layer written in a non-SQL procedural language to hide the underlying data representation.” I don’t think it matters if the API to the shared data is implemented inside the database engine or in a layer between the database engine and the database clients; what matters is that there is a single API to enforce integrity and consistency, not different implementations in every client application.

Mike, 5 December 2008 at 4:49 pm

Hi Greg,

I think we agree here for the most part.

Respectfully, the fact that database engines allow different languages to be used to code procedures inside their environment is sort of beside the point; familiar syntax is great, but the issue with SQL databases is that they don’t have a rich enough set of metadata available for their most common usage scenarios.

For instance, literally every application I have worked on implements its own user management and permissions system outside of that provided by SQL, because SQL permissions are inadequate to describe how users actually interact with various tables. Everything is an UPDATE, basically. User management and permissions are a critical part of business logic, forced to live in application code somewhere.

I do generally agree with the idea of having stored procedures (or something like them) surrounding complicated operations, but I think they only address half of the problem. What does our workflow look like? What are the relationships between our tables? How can we make our user interfaces more responsive to the database schema? Stored procedures certainly help make sure things can’t be updated incorrectly, but SQL’s lack of easily discoverable metadata still makes it difficult to know the meaning of the schema across multiple application languages.

-Mike

Dan, 7 December 2008 at 6:17 pm

Greg,

I’m not sure that you understand what Fowler means when he says ApplicationDatabases (http://martinfowler.com/bliki/ApplicationDatabase.html). He does not mean that each “application can have its own copy of data”, there is no copying of data; each application encapsulates some of a global schema, the only way for any other application to get the data is by talking the encapsulating application (via xml over http for example, but it could be via any interface). Instead of one large databases, there are lots of small database. None of the databases duplicate data from another, there is no violation of the DRY principles.

The technology stack for each application can be upgraded and migrated independently. Apps can be deployed and scaled independently. The critical part is how the global schema is broken up into the small databases. If it is done well you will have a high performing maintainable system. If this is done poorly then you will have a slow mess.

In a little-noticed deal that closed yesterday the Silicon Valley startup apostrophree secured a $25 million first round with Bolus Venture Capital of Palo Alto. apostrophree received seed money from Paul Graham’s Y Combinator earlier this year.

apostrophree is currently conducting a private beta test of their product, a proxy service that corrects common errors of spelling, punctuation, grammar, and usage in blogs and especially comments and discussion forums. I spoke with John Scogan, founder of apostrophree, about his product and company.

Typical Programmer: I have no idea what apostrophree does.

John Scogan: Every day hundreds of thousands of blog articles are posted, and those articles generate millions of comments. Aggregator sites like Digg and reddit also allow comment and discussion, so comments to blog entries are posted all over the web. We studied comments posted on popular blogs and aggregator sites and found that 45% to 50% of the comments contain at least one serious spelling, punctuation, or usage error. Almost 18% contain (sp?) or some other indication that the author knew their writing was questionable.

So apostrophree corrects the errors?

Yes. Apostrophree corrects many common errors on the fly, before readers see them. For example, the homonymns there, their, and they’re are frequently confused. Words like affect and effect or imply and infer are often mixed up. And missing or misplaced apostrophes abound — that’s where our name comes from.

Misplaced apostrophes?

Usually I don’t get through three articles in my RSS feed before I encounter it’s when the author meant its, or FAQ’s, which is almost as common on the web as FAQs, and more common than the correct FAQ, since the Q stands for “questions” and is therefore already a plural.

How does apostrophree work?

It’s a proxy service, installed on the customer’s LAN. Apostrophree looks at all incoming HTML content and applies proprietary language processing algorithms to find and correct the most egregious errors. It also looks at articles that contain flags like (sp?), which is the author telling us that he or she failed high school English. Apostrophree won’t catch everything but it will fix enough to pay for itself within a few months.

Why would an organization buy apostrophree? How does it save money?

In a typical company a small but significant number of employees read blogs and comment threads every day, sometimes several times in a work day. I say a significant number because the people who read comment threads and post comments of their own are very often among the most highly-paid people in the organization. The time apostrophree saves goes right to the bottom line.

But how does apostrophree save time? Don’t people just read through bad spelling and ignore missing apostrophes?

Most people either don’t recognize or don’t care when they encounter a misspelled word or incorrectly-formed plural. But some people do notice, and there’s a personality type that will spend a lot of time demonstrating their superior English skills online. We’ve studied this for over a year, in many settings, and over and over we find the same thing: the most expensive employees, especially technical people such as programmers, can be provoked by the smallest error to post a comment of their own correcting the error and chastising the original poster. Observing technical staff in one organization we found that just two common errors — it’s instead of its and there instead of their — accounted for six hours of essentially wasted time per month per employee.

We’ve also found that authors who flag their lack of writing ability with (sp?) actually attract more corrections, because on top of the corrections to the original error the laziness implied by (sp?) works some readers into a froth. Correcting such self-identifying comments, or just removing them altogether if there is no chance of converting the gibberish into proper English, stops another source of wasted time.

I can’t believe that posting corrections to comments takes that much time away from real work.

In a double-blind experiment at OSCON last month we scheduled a presentation that began by showing the URL of the blog the presentation was posted on. We invited the full room of 163 attendees to comment on the presentation in real time. We had introduced some errors into the presentation, some obvious like it’s and its, and some more subtle. We measured the time attendees spent composing and posting corrections. Over half of the attendees posted at least one correction, and 57 posted lengthy corrections with links to online references when we used elude instead of allude and imply instead of infer.

Programmers seem to divide into two groups when it comes to writing: nearly illiterate on the one hand, and pedantic on the other.

It’s worse than that — tempers flare and comment threads feed on themselves, corrections made to corrections. Sometimes a thread that starts out correcting something relatively benign like irregardless turns into an all-out attack on the author’s intelligence and competence, with the kind of name-calling and closely-reasoned arguments and links to Wikipedia that you find all over Digg and reddit.

So apostrophree corrects these kinds of errors before people see them, preventing employees from spending time posting corrections and engaging in online flame wars about English usage?

That’s our business model. We’ve obtained several patents on the technology behind apostrophree and we have some big organizations in our beta group. One of our beta testers, a large software company, is already reporting that their technical staff is spending less than half the time posting comments over the same period before installing apostrophree. They are also reporting, anecdotally, that some of their more volatile engineers seem more productive and less anxious than before, but that isn’t something we measure.

Do the people reading the corrected content object to this kind of editing?

They don’t know it’s happening. None of the employees involved in the beta test have asked why the literacy levels on their favorite sites have gone up from fourth grade to college junior, and that’s pretty close to the improvement you get with apostrophree.

Does apostrophree save time for all employees, or is it mainly for technical staff?

Everyone will benefit from fewer errors, but we’ve found that technical staff benefit the most. Executives and managers seem unaware of most errors, and even when errors are pointed out to them they are not usually inclined to post a correction. Technical staff are both likely to notice and get upset about errors in things they read, and they are far more likely to read articles online and post comments than managers or hourly line workers. We observed that non-technical staff with Internet access are more likely to use their computers to view and collect pornography than to post spelling corrections.

How do you plan to grow apostrophree so you can sell upgrades and keep money coming in?

We’re working on some things now, like cliché removal, that look promising. We have a team in the U.K. working on changing passive voice to active. Even something as simple as correcting capitalization of technical words and acronyms can pay off. If one of your expensive programmers comes across PERL instead of Perl he can spend thirty to forty-five minutes posting a correction, including extracts from two or three Wikipedia articles and Usenet archives. That’s ten to fifteen minutes per uncapitalized letter. And your programmer will compose and post a new version of the correction every time PERL is encountered online. That’s more than five times as long as is typically spent correcting presently when currently was meant.

Any plans for spending your venture capital?

Part of the deal we have with Bolus is the acquisition of a Russian company that is working on what they call a “clue gate.” The idea is to identify and filter out postings from newbies, particularly on technical forums, so employees are not tempted to insult someone who can’t install Python, for example, or to spend an hour explaining why a real programmer has to know C and not just Java.

Sounds like an interesting product. Good luck to you and the team at apostrophree.

Thank you.

Comments

noyuo, 12 August 2008 at 9:48 am

lol i see what yuo did there

bruce manly, 12 August 2008 at 10:53 am

I think that for all intensive purposes the web might become a more duller place. Irregardless though, I’d use it! 🙂

Josh, 12 August 2008 at 10:57 am

Hmm… 25 million for a glorified spellchecker? If it was at least a service like Askimet, I could at least almost see a reason to invest in this.

So basically you’re screwed if IE8 comes with a spellchecker. Thanks to Firefox it’s pretty much guaranteed that the major browsers will have spell checking in the near future and grammar checking soon after that.

…that or I just don’t get it.

Brian Culler, 12 August 2008 at 10:59 am

This has to be a joke.

Josh, 12 August 2008 at 11:10 am

…and I just didn’t get it.

I mean didnt.

Milan, 12 August 2008 at 12:50 pm

At least once in a while, the system must produce false positives: correcting ‘errors’ that do not exist, or correcting things in an incorrect way.

At the very least, websites using the technology should advertise the fact.

Mayson Lancaster, 12 August 2008 at 12:52 pm

The question is, how confusing will it be, when reading a site, to see all the frigging corrections to posts which your proxy server has auto-corrected? Not to speak of the flame wars that will occur when you get it wrong. And believe me, unless you’re a lot better than any spelling checker or grammar checker I’ve ever seen, you wll get it wrong (at least occasionally).

Best wishes, Mayson

CANT PATENT, 12 August 2008 at 1:18 pm

why would anyone pay for it? It can implemented very easily for free and once that is done the company is sunk.

SoWhatI’mAlive, 12 August 2008 at 1:28 pm

These people have no regard for what language actually is — a historical (an a-historical!) document of intention and meaning that once conveyed begins to degrade till thought of as nonsense; as the Minoans to us, so us to the Neominoans.

Jessica Livingston, 12 August 2008 at 1:31 pm

Greg,

Y Combinator did not fund apostrophee. Would you mind correcting your post? Thanks.

Larry, 12 August 2008 at 2:06 pm

“For example, the homonymns there, their, and they’re…”

homonymns.

I’m just sayin’.

Andrew, 12 August 2008 at 2:10 pm

Jessica —

I believe you meant to say “Y-Combinator.”

Khang Toh, 12 August 2008 at 6:16 pm

sign… $25M wasted like that … wrongs spelllings are the breadz and butterz of blogging and web 2.0 startpus like Flickr

Evan, 13 August 2008 at 1:55 am

great, prescriptivist software. Just what we need

http://languagelog.ldc.upenn.edu/nll/?p=471

This is the first interview posted on Typical Programmer. I plan to do more interviews with programmers who are working on interesting projects and pushing new ideas and technologies.

I had a chance to interview Boyd Hakluyt at O’Reilly’s OSCON 2008 in Portland last week. Boyd is working on a new web application framework called Miasma. Boyd gave a demo at OSCON, and afterward we had some of the local microbrew Portland is famous for and talked about web application frameworks.

Typical Programmer: I hadn’t heard of Miasma until OSCON, and even then I only saw the demo because the room numbers were mixed up on the program.

Boyd Hakluyt: Yeah I got on the program at the last minute. Some of the people who came to the demo were expecting a “birds of a feather” meeting to talk about “life extension for geeks,” which was actually on the OSCON program this year.

You’re flying under the radar with Miasma. When do you expect to have something to release?

I was hoping to have a beta version ready in time for OSCON but we lost a key developer last month so the code isn’t as ready or tested as I would like. I also need to get the web site and documentation finished. The framework was renamed in June so the domain name I had picked out doesn’t work anymore.

What was it called before you renamed it Miasma?

Originally it was called Darlene. One of our lead developers, who wrote the core URL routing code and the template parser named the first version after his girlfriend. When they broke up we talked about renaming it couldn’t settle on a good name that wasn’t taken. When he left the project Darlene didn’t seem to fit anymore. I think Miasma is cool name and no other programming project is using that name. I have a friend working on a logo, too.

How many people are working on Miasma?

Right now it’s just me, but I’m working on it almost every day.

There are lots of web frameworks to choose from. What originally inspired you to write a new framework and what makes it different?

I’ve looked at most of the available frameworks and they all have problems. None of them do exactly what I need to do. I was working on a side project developing an appointment and contact management system for my optometrist and it occurred to me that most of the code I had written could be used to build other web-based applications. My client agreed to let me retain ownership of the code as long as I finished their application for free.

What’s different about Miasma is the philosophy behind it. It’s not just another big collection of classes and objects. From the beginning my plan was to make a framework that the average programmer could use to build the kinds of web sites I work on. You don’t have to be a language geek or understand things like methods or closures. Most of the code you write with Miasma is the HTML and JavaScript you are used to writing.

What language is Miasma written in?

We started with Ruby but ran into too many limitations and performance problems with it. Then we ported the code to Haskell and got it partially working on our laptops, but it was too hard to deploy on our client’s hosting account. One of the most intense ideas we had was a five day long programming contest to write the framework core in as many languages as we could think of. We invited all of our friends who know how to code over, got a bunch of frozen pizzas, snacks, beer, soda, and two XBox 360s hooked up to LCD TVs. We had people writing the framework in Lisp, Scheme, Erlang, C++, Lua, Python, Cat, Forth — everything but COBOL. That was a very cool way to play with different languages.

None of the code from the contest was really beta-quality, though, and we still had problems getting any of it to run on my GoDaddy hosting account, so we decided to go with PHP.

There are so many PHP frameworks already. Why would I use Miasma instead of, say, Zend or Cake?

Cake has a huge manual and is just too big to easily understand. I haven’t used Zend or looked at it but a friend of mine has played with it and told me it’s complicated and doesn’t really hide database SQL or make AJAX easier to do.

My philosophy is to make things simple. If you look at Cake or Ruby on Rails or most of the other frameworks you notice they have hundreds of classes and objects. But a lot of programmers don’t understand object orientated \[sic\] programming and don’t want to learn it just to make a simple web site. What they want is something that does database operations without forcing them to learn SQL — what the Rails people call CRUD.

With Miasma I’ve concentrated on things that web developers do over and over. When I’m writing a web site most of my time is spent getting variables from the server displayed in the right place in the HTML and debugging JavaScript. It’s not spent making a bunch of classes and objects that will most likely never be reused.

Does Miasma support the Model-View-Controller (MVC) pattern?

Yes. The Wikipedia article about MVC has been a big help figuring out how to organize the code. When you start a new Miasma project you run a script that creates a project folder and three folders inside of that named models, views, and controllers. By starting with that structure programmers are encouraged to follow MVC best practices when deciding which folder to save their scripts in.

A new Miasma project with folder layout showing MVC structure:

Does Miasma use other software patterns?

When people ask about patterns I like to tell them Miasma is based on the simpleton pattern, because it’s very simple to use. I know in real OO programming simpletons are usually a bad idea but I mean it ironically.

Most of the successful frameworks have large developer communities. Ruby on Rails is probably the most obvious example. How will you get that kind of excitement and interest behind Miasma?

I’ve studied how David Heinemeier Hansson \[author of Ruby on Rails\] has popularized Rails. Many of his talks are available on YouTube. Rails was extracted from a bigger project, and so was Miasma. I plan to start giving talks at conferences like David did when Rails came out. I guess you can think of Miasma as having the benefits of Rails without a lot of hard to understand Ruby OO code, and without the profanity.

Does Miasma have a database abstraction layer? Did you write your own or use an existing database library?

I wrote my own. In the site configuration file, which is XML by the way, there’s a setting for the database you’re using. There are functions for doing database operations like SELECT and INSERT and UPDATE and they choose which PHP database functions to call based on the configuration. The programmer passes arguments that are converted to WHERE conditions and ORDER BY and things like that. It’s much easier than writing SQL. I don’t know SQL very well and that was my incentive for writing the database functions.

An example of Miasma’s database model functions compared to plain PHP. Notice no SQL knowledge required with the Miasma database abstraction functions:

/\* plain PHP with MySQL \*/
$appointments = array();
$apptnum = 0;
$result = mysql\_query("
 select \* from appontments\_table where office=0", $dbconn);
while ($row = mysql\_fetch\_assoc($result)) {
 if ($row\["appt\_doctor"\] == 1)
 $appointments\[$apptnum++\] = $row;
}

/\* miasma ORM functions */
$appointments = array();
$apptnum = 0;
$wherecond = " where office=0 ";
$temp = mia_model_select("appontments_table", "*", $wherecond);
$appointments = mia_model_filter($temp, "appt_doctor", 1);

You’ll notice there’s no SQL in the Miasma example. One thing I don’t have working yet is making plurals for the table names, like Rails does, so you have to use the actual database table name. I’ll have that working for the beta.

What about templates?

Miasma has a templating engine just like the other frameworks. I looked at probably twenty different template systems but they all had issues that didn’t work for me. Basically you create a template that has placeholders for variables that come from the model or controller. The placeholders are marked in the template by surrounding them with ++var++. When the controller processes the template it reads in the file, uses regular expressions to find all of the placeholders, then replaces those with values from an array of substitutions.

A snippet from a Miasma template:

<div class="appointment">
 <b>Date</b> ++var++DATE++var++ <br>
 <b>Name</b> ++var++FIRST\_NAME++var++
 ++var++LAST\_NAME++var++ <br>
 ...
</div>
<!-- var placeholders can be used in html tags -->
<div class="++var++TYPE\_OF\_APPT\_CLASS++var++">

The template engine is the biggest piece of code and the one that I’ve tweaked the most, because a big template with lots of placeholders can be slow. My goal is to add things like conditionals (++if++) and loops (++for++) so it can do the same kinds of things developers normally do with raw PHP.

I don’t mean to ask a stupid question, but why not just use PHP instead of making a new template language?

The main problems with PHP are no namespaces, and it has way too many library functions that are just too hard to figure out and remember. By giving the developer a smaller and simpler set of library functions they don’t have to worry about all of the junk in PHP. And all of the Miasma functions are prefixed with mia\_, so they have a unique namespace, unlike most of the PHP library.

I’m looking forward to a beta release I can try out.

I think a lot of developers are tired of how complicated web applications are, and how huge the frameworks have become. When I see the unreadable mess of JavaScript code on any of Google’s pages, for example, or the contortions programmers go through to get PHP to send a valid SQL query, I know there’s a long way to go with web frameworks. I think Miasma is a pretty big step in the right direction.

Thanks for the beer, and good luck.

Thank you.

Comments

ummmm, 29 July 2008 at 5:48 am

is this article is a joke?

forsooth, 29 July 2008 at 8:22 am

Thanks for the article. I like Boyd Hakluyt’s philosophy. (BTW, his first name is misspelled in the first statement.)

Wolf, 30 July 2008 at 3:06 am

You say “We started with Ruby but ran into too many limitations and performance problems with it.” and choose PHP! What kind of limitations you get with Ruby and haven’t with PHP? I think this article is bullshit.

espressoexcess, 12 August 2008 at 1:35 pm

I have the same problem that Boyd has with overly complicated app frameworks. Who has time to read all that documentation when there’s a client breathing down your neck, and a burgeoning WOW guild to maintain?

I myself have created a web application framework like Miasma, that’s easy for non-programmers to use. Mine’s just called Asma, because it doesn’t have to be my Asma, it can be your Asma too.

Asma does everything with just a single CAsma class, (pronounce it “hhhas-ma” with a Middle Eastern twist). CAsma inherits privately from a Simpleton base class. The class has only one function, doh(), which is really easy to call, as long as you understand that it takes no arguments and returns no result. And the CAsma class is declared final to prevent people from destroying my framework’s simplicity by subclassing.

Lots of successful web sites have been built on the Asma framework (virtual pet rock ecommerce site, my D&D club site, etc.) and I have to say they all work flawlessly.

Or at least the parts of them that use the Asma framework do. I can’t vouch for the quality of all that other HTML, JavaScript and PHP crud that people like to wrap my nice Asma framework with.

Mark, 13 August 2008 at 9:53 am

I know in real OO programming simpletons are usually a bad idea but I mean it ironically.

Do you think maybe he is thinking of singletons, but he forgot what they are called?

I was looking for the April 1st date on this article.