How Wal-Mart enables ‘innersourcing’ with Github

store front night 129860898494412686

Talk to Wal-Mart software engineers and you’ll hear them speaking candidly about how open source technology influences virtually every project they undertake. What you may not know is that the retail giant’s programmers have also embraced the open source ethos of sharing code to run ecommerce, search, mobile payments and other digital functions. Known as enterprise source, internal open source and innersource, the twist on traditional open source has caught on at other organizations, including Autodesk, Capital One, Paypal and Bloomberg.

“The more we can do this internal open source, the more flexible we can be and the faster we can release code to customers,” says Jeremy King, CTO of Wal-Mart Global eCommerce and head of @WalmartLabs. “We want you to consume OS because it’s faster, cheaper and better but we want you to contribute because it’s the right thing to do for the community,” King says.

Open source has become more the rule than the exception in recent years thanks to wide-scale code sharing by Google, Facebook and several other Silicon Valley companies. Walmart has itself has embraced the model. The group earlier this year launched OneOps, a platform that enables programmers to test and switch between different cloud providers, a crucial benefit as companies embrace hybrid cloud models.

Innersource helps big companies reduce bureaucracy

Innersource is essentially open source software developed behind the firewall and shared among engineering groups. The approach is particularly valuable at large organizations that employ thousands of employees, and where even the hint of bureaucracy is difficult to snuff out.

Wal-Mart is large – the Labs pushes out 30,000 code deployments each month – but King says his unit operates as the “world’s largest startup.” Some companies may have 1,000 people working on single feature set, but Wal-Mart Labs flips that model on its ear.

Its more than 1,000 engineers, strategically deployed in 100 teams with between 10 and 20 programmers each, build, test and ship code in the DevOps model. Yet even though developers pump out code in the continuous integration/continuous deployment fashion, bottlenecks can crop up when the teams have a lot of projects in their queues. Giving engineers the autonomy to write and contribute code for other groups reduces these single points of failure, King says.

Innersource works like this at Wal-Mart, says King: Suppose an engineer needs a new connection to the payment gateway but the team that supports that project has five projects ahead of it. The engineer can write his or her own connection and have it reviewed and accepted (or rejected if it’s not up to par). Or maybe a member of the Labs payment team who has an idea for new shipping path for the shopping cart/checkout team. The payment team representative can simply build the feature and contribute it directly to the cart/checkout code through GitHub, the code repository Labs relies heavily on to test and review code. “Ideally all code will be like that,” King says.

But as King found innersource isn’t the perfect approach for every Labs project. He says it’s difficult for members outside his search team to contribute to the search algorithms because the platform is so complex, with Ph.D-wielding data scientists and other wonks refining the software. “We want more people to contribute but it’s extremely complicated,” King says. “It’s not something you can just walk into on GitHub and contribute.”

Programming prospects prefer open source

Wal-Mart runs innersource through GitHub Enterprise, which is similar to the public code review repository with one key difference: Companies host it behind their own firewalls. King says this provides piece of mind for his Labs, particularly for highly proprietary search algorithms. GitHub Enterprise is distinct from the public version of GitHub, which the Labs also used to launch OneOps. King says his unit is preparing to launch on GitHub proper Electrode, a tool to enable programmers working with the React OS component framework to enable developers to more quickly build web pages for the mobile web and mobile apps.

Wal-Mart’s love of open source accelerated after King’s arrival from eBay in 2011. In an effort to make Labs run more nimbly, King says he replaced a lot of decade-plus-old legacy software with NoSQL databases such as Cassandra technology, Kafka queues for its messaging bus and various other open source programming tools, including Node.js, the server-side JavaScript environment. In a highly-publicized move, the company rebuilt its ecommerce platform using OpenStack, becoming one of the largest deployments of the nascent cloud platform.

King says that Wal-Mart’s adoption of open source – both taking and giving back — has helped it attract top-tier technical talent. King says that while half of the prospects he targets have in hand offers from Apple, Google, Facebook, LinkedIn and other Silicon Valley players, Labs wins such talent contest 70 percent of the time. He chalked up the victories to brand, the promise of challenging work on small dedicated teams, and, of course, open source. “[Prospects] want the reach that we’ve got but work like they’re in a startup,” King says.