About Ryan Tomayko

2ndscale.com is my place on the web to talk about software development, systems design, and technology in general.

I've been so extremely fortunate to have worked with some of the best software engineers, designers, and product folks in the industry to build and ship software products used by millions of people. Below is what I'd consider some of my best work and favorite projects, none of which would have been possible without a huge list of talented people.

Most of my background is in software development tools, distributed systems design, and large-scale software infrastructure but I also ❤️️ product design and development.

GitHub / 2009 - 2016

I started at GitHub in 2009 to help scale a small but fast growing site to every software developer in the world. It's been a hell of a ride and we're just getting started.

Scaleable Git Repository Storage

Today, GitHub.com's storage infrastructure hosts hundreds of terrabytes of Git data spanning 20 million repositories. Those numbers were much smaller when I started in 2009 to focus on performance, reliability, and scalability but the rate of growth has been consistent. I've had a hand in building everything from our custom Git RPC system, shared object storage ("net-shard"), to visualization and monitoring tools.

GitHub's "fs-util" (filesystem utilization) visualization. On display and updated in real time on big TVs at GitHub HQ.

The Pull Request

Early on, development at GitHub needed to move at a fast clip so we were especially interested in product features that brought more of the propose, discuss, iterate, accept workflow—unavoidable any time more than one person is working on a codebase—directly into the site.

When we discussed each other's local workflows, we found that we'd each written a separate set of crappy scripts to automate away some of the tedium involved with getting a simple commit list and diff for reviewing a branch's proposed changes. We thought it might be interesting to combine these powerful Git history / diff tools with some of the discussion features already present in GitHub Issues.

Five years and over 30 million Pull Requests later, I think we can call this little experiment a success. Tip of the hat to @kneath, who did all the original design work, and @schacon who spiked out the first bits of the backend and Git integration.

The first Pull Request ever submitted. I don't have a screencap with the original visual style, unfortunately, but the ship blog post does if you're into nostalgia.

Hubot + CHATOPS

As a teenager, one of my earliest programming experiences was running an EggDrop chatbot on IRC, trying to ride net splits and take over channels like #teen on EFnet. I got pretty good at Tcl (used for scripting EggDrop) and experienced first-hand how useful and fun a chatbot can be for getting important work done.

GitHub's first office was a chat room so I guess it was only natural that I would build and demo the initial version of Hubot at the first ever GitHub Summit in 2010. The chatbot had only a few simple tricks at that point, like showing the system load average on all production hosts when asked, but would grow from there to become the backbone of GitHub's operations tooling, not to mention GIF culture.

Today, GitHub's internal version of Hubot has been programmed with nearly a thousand commands contributed by more than 100 contributors. CHATOPS has become something of an industry trend .

The Hubot website, lovingly crafted by @bryanveloso. That's my sig on the, uh, patent.

There are too many people crucial to Hubot's success to list here but I want to send a special shout out to @atmos, for getting the thing to the point where it could be released as open source, and also to @tombell and @technicalpickles who have been maintaining Hubot for some time now. Massive chunks of GitHub's operational Hubot scripts were written by @jnewland and he also gave a landmark talk on Hubot and CHATOPs at GitHub.

Heroku / 2008

I worked at Heroku for a year right after they decided to pivot from an in-browser IDE to focusing primarily on "PaaS", a whole concept area and industry segment that didn't exist before their product hit big with developers.

I was hired primarily for my background and experience in HTTP servers and caches, since they wanted to build this crazy routing mesh thing (spoiler alert: they did), but ended up spending most of my time working to scale and improve the Git and application server infrastructure.

I wrote the original version of heroku ps , including the CLI, state machine logic for scaling dynos up and down on demand, and basic process life-cycle management + reporting across a large and elastic cluster of machines. Also, the first streaming version of the heroku logs, which I remember fondly because it was the first time I'd found a practical use for mkfifo(1) .

Early high-level Heroku concept diagram by James Lindenbaum. I worked on some of this stuff.

Fun times. Heroku was the first job I had a really hard time leaving when the time came and remains one of the best experiences of my career.

Songbird / 2007

Cool little Mozilla/XUL based music player by the same people that did Winamp originally. Apple put an enormous amount of effort into their iPod, iPhone, iTunes Store dynasty during this time, which unfortunately cast a large shadow over the great work going on at this place. I worked on the Songbird Addons site, pretty straightforward Ruby on Rails development with some design here and there.

Sterling Commerce / 2000

My first real gig out of college. I gained a lot of experience in early web tech, ASP 3.0, Java, and even a bit of C.

Bending Light Productions / 1990s

When I was in high school I opened a DBA account at the bank and started getting checks in the mail for Windows shareware. I built an HTML editor called SiteBuilder X (not joking) and an HTTP/FTP download manager called Power Download that supported resuming downloads, which was a big deal because getting knocked off of a 14.4K modem-line was the state of the art in internet connectivity. For something I considered a hobby at the time, it still surprises me how much the types of tech I was gaining experience in here would grow to be so important to my career and the world today.