Bonanza

Back to Jobs

Devops Engineer

Programmer

Five million users making 40m monthly pageviews across 30 servers with 99.99% uptime? That's what you walk into on day one. The right candidate for this job knows that when you're proactive about managing and monitoring servers, DevOps is less about firefighting and more about empowering millions of users via a well-crafted infrastructure.

Job Description

Bonanza is a fast-growing online marketplace that has proven Rails does scale, even when 90% of our content is dynamically rendered. Our infrastructure consists of more than 30 server instances, split between app, async, test, database, search, utility, and memcached-focused self-hosted Linux installs on around 25 physical servers. We make heavy use of ansible to provision servers and ensure consistency across the various server roles. 

The principles we've used to design our infrastructure include:

  • 99.99% uptime target. Our total downtime has averaged less 5 hours per year for 8 years by being tireless about root cause analysis, and working doggedly to eliminate single points of failure. 
  • No unsolved mysteries. This is partly possible because of our consistent use of root cause analysis when problems occur. The other part is ensuring that no server setup is a one-off endeavor. All server configuration is handled via ansible, which means we can store and track server history through source control.
  • Minimize proliferation of architecture components. Developers always want to employ the latest hotness, but at Bonanza they must clear a high barrier of addressing "why?" when they seek to introduce a new dependency or failure point to the existing tech stack. That said, Bonanza and Background Burner are many-faceted organisms that inevitably require understanding a breadth of systems.

The sum of these and other measures have created an environment that has proven highly stable in spite of its many components, as listed in the "Requirements" section.

Our presumption is that our pure Devops-related needs will occupy 50-80% of a full time role; with whatever remaining time our Devops engineer may find, they will make contributions to our Ruby on Rails codebase.

Job Requirements

Our Devops Engineer will be responsible for overseeing and creating failover mechanisms across the Bonanza technology stack. At present, this stack includes (in rough order of usage):

  • Ruby on Rails
  • Phusion Passenger on Apache
  • MySQL
  • Resque
  • Ubuntu
  • Redis
  • pfSense
  • HAProxy
  • Sphinx search
  • Ansible
  • Jenkins (CI server)
  • Memcached
  • Phusion Passenger on nginx (for websockets)
  • Pusher (for websockets)
  • ... and a few others

For monitoring, we're currently using:

  • New Relic
  • Datadog
  • VividCortex
  • Papertrail
  • Rollbar
  • Monit
  • Nagios

We don't expect anybody will step into this role having experience across all of these systems, but the more that one is familiar with, the quicker they can begin to make an impact. 

We prefer candidates with at least 3-5 years of relevant experience in a Devops-related role. We need someone who has proven capable of maintaining a high uptime technology stack at scale (if not necessarily Bonanza-scale). We also need someone familiar with security best practices, who can help us identify potential weak points in our network and periodically audit our system-wide security architecture.

Job Benefits

Bonanza is a product-first company run by a CEO who has spent most of his adult life as a Programmer. The implications of this are that we've obsessed for literally years about how to foster an environment optimized for Developer happiness and throughput. It starts with our salaries, which, at the high end, range to levels beyond those offered by the tech titans. More importantly, we provide a working environment where you will virtually never be interrupted by meetings, and where you will be given access to the best hardware available. Because your productivity should never constrained for lack of resources. We also offer a thoughtfully constructed collection of perks that stands apart from our competitors:

  • Fast growing and already-profitable company. It's fun to work for a winner.
  • "Work from home Wednesday" - every Wednesday we get to work from home, a coffee shop, or even the office, if that's your bag.
  • "Freedom February" - every February we retreat from Seattle and spend a month annoying our friends with selfies taken in Mexico, Belize, Europe, etc.
  • Five bonus (not counted as vacation) days off in February to enjoy your vacation or staycation
  • Twice-weekly catered lunches
  • Monthly on-site full body massage
  • Work from a waterfront office in downtown Seattle (food trucks aplenty!) convenient to many bus lines
  • Full arcade room in the office
  • Pong table on site
  • The boring stuff: fully paid health insurance coverage, vacation benefits, holidays, stock options, and more.

Beyond this generous set of perks, our commitment to our employees is that if ever they have ideas on how we can improve our benefits package to better serve our team, we will consider those ideas and try to make them happen when economically possible. That's probably why we keep winning awards like these:

Entrepreneur-360

Entrepreneur

2016 Best Places to Work

2015-best-in-business-award

Best in Biz Awards

2015 & 2016 Silver Winner - Small Company of the Year

2016_sellers_choice

EcommerceBytes

2016 Most Recommended Marketplace

Apply for this job

Got a resume or cover letter that's not well suited for the format above? Drop us a line at jobs@bonanza.com.

Back to the mothership