Sedimental

Cruising through complex data

2023-01-19T14:00:00Z

This post is a showcase of data wrangling techniques in Python, using glom. If you haven't heard of glom, it's a data transformation library and CLI designed for Python. Think HTML templating, but for objects, dicts, and other data structures.

It's been almost five years since the first release of glom. That version now looks quaint in comparison to the just-released glom 23. Out of all the new functionality, we're going to take a look at six techniques that'll level up your complex data handling.

Contents

Star path selectors
Deep assignment and deletion
The Data Trace
Pattern matching
Streaming
Flattening and Merging
Other core updates

NB: Throughout the post, you'll note examples linking to a site called glompad. Like so many regex and JS playgrounds, glompad is glom in the browser. Very much an alpha, I'll save the details for another post. In the meantime, try it out and let me know how it goes!

Star path selectors

Years in the making, glom's newest feature is one of the longest anticipated. Since its first release, glom's deep get has excelled at fetching single values:

target = {'a': {'b': {'c': 'd'}}}
glom(target, 'a.b.c')
# 'd'

As of the latest release, glom now does glob-style * and ** as path segments, aka wildcard expansion:

glom({'a': [{'k': 'v1'}, {'k': 'v2'}]}, 'a.*.k')  # * is single-level
# ['v1', 'v2']

glom({'a': [{'k': 'v3'}, {'k': 'v4'}]}, '**.k')  # ** is recursive
# ['v3', 'v4']

Notably, this is one of the only breaking features in glom's history. Star selectors were added as an option in glom 22, and baked for a year (with warnings for any users with stars in their paths) before becoming the default in glom 23.

Deep assignment and deletion

By default, glom makes and returns new data structures. But glom's default immutable approach isn't always a perfect fit for the messy, deeply-nested structures one gets from scraped DOMs, ancient XML, or idiosyncratic API wrappers.

So one of glom's earliest additions, way back in 2018, enabled declarative deep assignments that would work across virtually all mutable Python objects. First with Assign() and the assign() convenience function (example, docs):

target = {'a': [{'b': 'c'}, {'d': None}]}
assign(target, 'a.1.d', 'e')  # let's give 'd' a value of 'e'
# {'a': [{'b': 'c'}, {'d': 'e'}]}

Assign also unlocked a super useful pattern of automatically creating nested objects without the need for defaultdict and friends (example):

target = {}
assign(target, 'a.b.c', 'hi', missing=dict)
# {'a': {'b': {'c': 'hi'}}}

And for something more destructive, there's Delete() and delete() (example, docs):

target = {'a': [{'b': 'c'}, {'d': None}]}
delete(target, 'a.0.b')
# {'a': [{}, {'d': None}]}

Assign() and Delete() both shine when manipulating ElementTree-style documents from etree, lxml, html5lib, and the like.

Like glom's other path-based functionality, the nuances of assigning Python dict keys, object attributes, and sequence indices are handled for you. There's also an extension system for adding support especially unique types.

The Data Trace

The main appeal of glom has always been succinct and robust data access and transformation. No single glom feature showcases this quite as much as the data trace.

Data traces make glom's errors far more debuggable than Python's default exceptions. You don't see internal glom or Python stack frames; just you, your code, and your data:

>>> target = {'planets': [{'name': 'earth', 'moons': 1}]}
>>> spec = ('planets', ['rings'])  # a spec we expect to fail
>>> glom(target, spec)
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/home/mahmoud/projects/glom/glom/core.py", line 1787, in glom
     raise err
 glom.core.PathAccessError: error raised while processing, details below.
  Target-spec trace (most recent last):
  - Target: {'planets': [{'name': 'earth', 'moons': 1}]}
  - Spec: ('planets', ['rings'])
  - Spec: 'planets'
  - Target: [{'name': 'earth', 'moons': 1}]
  - Spec: ['rings']
  - Target: {'name': 'earth', 'moons': 1}
  - Spec: 'rings'
 glom.core.PathAccessError: could not access 'rings', part 0 of Path('rings'), got error: KeyError('rings')

Failures before and after the data trace. Full text here.

One day I'll write a post about how tracebacks are an oft-neglected part of a library's interface. The right traceback can turn an all-night debugging session into a quick fix anyone can push.

For now, see the doc with examples and more explanation here.

Pattern matching

While glom started as a data transformer, you often need to validate data before transforming it. Data validation fits nicely into spec format, and so glom's Match specifier was born:

# load some data
target = [{'id': 1, 'email': 'alice@example.com'}, 
          {'id': 2, 'email': 'bob@example.com'}]

# let's validate that the data has the types we expect
spec = Match([{'id': int, 'email': str}])

result = glom(target, spec)
# result here is equal to the data itself

Glom's pattern matching now features its own shorthand M spec, which is great for quick guards, and a Regex helper, too:

# using the example data above, we can also validate the contents of the data
spec = Match([{'id': And(M > 0, int), 'email': Regex('[^@]+@[^@]+')}])

result = glom(target, spec)
# result here is again equal to the target data

Even a simple pattern matching example shows the power of the glom data trace. Check out the error message when some bad data gets added:

>>> target.append({'id': '3', 'email': 'charlie@example.com'})
>>> result = glom(target, spec)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "../glom/core.py", line 2294, in glom
    raise err
glom.matching.TypeMatchError: error raised while processing, details below.
 Target-spec trace (most recent last):
 - Target: [{'email': 'alice@example.com', 'id': 1}, {'email': 'bob@example.com', 'id': 2}, {'ema... (len=3)
 - Spec: Match([{'email': str, 'id': int}])
 - Spec: [{'email': str, 'id': int}]
 - Target: {'email': 'charlie@example.com', 'id': '3'}
 - Spec: {'email': str, 'id': int}
 - Target: 'id'
 - Spec: 'id'
 - Target: '3'
 - Spec: int
glom.matching.TypeMatchError: expected type int, not str

The data trace gets even sweeter when we introduce flow control with Switch. See the data trace in action in this example. Users of shape-based typecheckers like Flow will especially appreciate the specificity of glom's error messages in these validation cases.

Streaming

For datasets too large to fit in memory, glom grew an Iter() specifier in 2019 (example, docs). Iter() offers a readable chaining API that lazily creates nesting generators.

target = [1, 2, None, None, 3, None, 3, None, 2, 4]

spec = Iter().filter().unique()  # this gives a streaming generator when evaluated
glom(target, spec.all())  # .all() converts the generator to a list
# [1, 2, 3, 4]

Iter()'s built-in methods also include .split(), .flatten(), .chunked(), .slice(), .limit() among others. In short, endless possibilities for endless data.

Flattening and Merging

So much data revolves around iterables that in 2019 glom introduced the ability to "reduce" those iterables to flatter values, with the introduction of Flatten (example, docs):

list_of_iterables = [{0}, [1, 2, 3], (4, 5)]
flatten(list_of_iterables)
# [0, 1, 2, 3, 4, 5]

Even a mix of iterables (iterators, lists, tuples) combines nicely.

With Flatten came the numeric Sum, not unlike the builtin:

glom(range(5), Sum())
# 15

And the generic Fold, useful for some rare cases:

target = [set([1, 2]), set([3]), set([2, 4])]
result = glom(target, Fold(T, init=frozenset, op=frozenset.union))
# frozenset([1, 2, 3, 4])

A later release brought flattening to mappings, via Merge (example, docs):

target = [{'a': 'alpha'}, {'b': 'B'}, {'a': 'A'}]
merge(target)
# {'a': 'A', 'b': 'B'}

Merge() is great for deduping documents with a simple last-value-wins strategy.

Other core updates

The features above, and myriad others from the changelog, required multiple evolutions of the glom core. Underneath glom's hood is a loop that interprets the spec against the target. A simple, early version is preserved here in the docs.

However, the inner workings of the core were not part of glom's API, which limited extensibility. A lot of progress has been made in opening up glom internals for those use cases we couldn't predict.

Scope

Most transformations only requires a target and spec. Most... but not all.

For cases that needed additional state, like aggregation and multi-target glomming, we added the glom Scope (example, docs):

# Make a spec that uses the T singleton to call 
# the target's count method using the search value in the scope (S)
count_spec = T.count(S.search) 
scope = {'search': 'a'}  # additional context we'll pass in
glom(['a', 'c', 'a', 'b'], count_spec, scope=scope)
# 2

Here, the scope is used to pass in a search parameter which will be used against the target (T). Usage can get quite advanced, including specs that write to the scope (example):

target = {'data': {'val': 9}}

spec = (S(value=T['data']['val']), 
        {'result': S['value']})

glom(target, spec)
# {'result': 9}

Here we grab 'val', save it to the scope as 'value', then use it to build our new result.

Modes

As discussed in pattern matching above,
some applications outgrew glom's initial data transformation behavior. To handle these diverging behaviors, glom introduced the concept of modes.

Glom specs stay succinct by using Python literals, and modes allow changing the interpretation of those objects. Glom comes with two documented modes, the default Auto() and Match() (example), which can be interleaved as necessary:

spec = Auto([Match(int, default=SKIP)])
target = [1, 'a', 2, 'c', 'a', 'b']
glom(target, spec)
# [1, 2]

We're working on adding more. You can easily add your own, too.

Extensions

We strive to make glom as widely applicable as possible, but data takes too many forms to count. We solve this by making glom extensible in several ways:

Registering new target types and new operations on the target
Creating new Spec types
Adding new modes

By understanding glom's scope and its internals, it becomes clear that most built-in glom functionality is implemented through these public interfaces. So while glom can feel magical at times, now you can extend glom without touching the core, and be a part of the magic, too. ☄️

Not bad for five years, and we haven't even scratched all the surfaces, yet. Hopefully the next showcase won't be quite so far out.

Intentional Creation

2023-01-04T08:00:00Z

Reliably tap into your creativity with the 4 Cs: Consume, critique, curate, create.

This is one of my oldest ideas, finally published on the GitHub ReadME Project blog, along with a profile, in June 2022. For more like this, follow me on Twitter or Mastodon. (You can also read it in 中文 here. Thanks Dominic Huang!)

We all have creative potential. Whether it gets you up in the morning or keeps you up at night, you've felt its gnaw.

Turning that potential into productivity can prove challenging in an internet-connected environment that offers a constant stream of consumables. How do we pick a direction?

In this guide, you'll see how to distill the elements of creativity into four deliberate stages, and how to put the process to use:

Contents

Consume
Critique
Curate
Create
- Debugging the process
Putting it into practice

These 4 Cs comprise a straightforward, adaptable approach that works well in both group and solo settings. You've already started on step 1. Read on to find out what to do next.

Consume

Step 1: Turn passive consumption into active research.

From the moment you open your eyes in the morning, you're accosted with calls to consume. Articles, videos, podcasts, the newest Wordle variant. When consumption is the default mode of our modern computing environment, how is a builder supposed to build?

To create, we must first recognize its inverse: consumption. Consumption is a useful stage, but can be dangerous if it’s terminal. An infinite loop in this stage kills any chance of creation.

Little is created in a vacuum. Creation still starts with consumption, albeit consumption disarmed with an intention. Turn pure consumption into active research, punctuated with critique.

Critique

Step 2: Capture your reactions in critiques, and research no faster than you can react.

As with so many software problems of our day, the answer is simple: React. No, not the JavaScript framework, but the human act of reaction. Intentional creation starts with giving yourself pause on new inputs. Seek a reaction from yourself. A semi-structured reaction, or critique, is a time-honored practice in creative fields, like architecture.

Infinite scroll may prove challenging to overcome, but before turning your consideration to the next item in your feed, activate your critical senses. Draw some conclusions. Even unvetted, they're yours.

If you're finding it hard to summon a critique, this is a clear sign you're consuming faster than you can reflect. If you're not reflecting, you're not learning. You may need to go deeper on individual items, or just take a break.

Your critiques have never been easier to capture, whether typed in markdown, dictated to automatic transcription, or written down in a notepad on your desk. Try opening your editor or critique tool before opening any new resources. Feel free to open one now.

If you're not reflecting, you're not learning.

Curate

Step 3: Curate critiques into collections that act as reservoirs of creative reference.

Critiques are only proto-creative output. Writing anything helps prime the creative pump, but criticism is raw reaction. You want a refined synthesis. Once you've got enough critiques under your belt, curate the positive examples into a collection.

From interior designers to lab researchers to club DJs, creators recognize the value of a structured, referenceable collection. Sometimes a situation calls for urgency or direction, and an organized, well-researched collection can offer an existing solution. Sometimes, in the context of a comprehensive collection, the lack of a referenceable solution is itself a signal that it's time to invent.

Curated collections become artifacts unto themselves. I've helped create a few, including 0ver.org, seealso.org, and the Awesome Python Applications list. There’s more awesome out there beyond Awesome Lists, like explorabl.es, the Cooperpress newsletters, or the “swipe file” phenomenon used among designers and content creators. There's respectable work in curation. Still, curation is more important as a stepping stone to our original higher calling. Less is more.

Create

Step 4: Return to your curations regularly to discover your creative path forward.

Collections of a certain size tend to produce interesting findings. Patterns and gaps emerge that inspire creative next steps. As an example, while researching approaches to Python packaging, a pattern emerged that led to one of my most popular concepts/blog posts/talks, The Packaging Gradient.

Whole projects can be born out of connections made with collections. My framework Clastic, which was eventually used by teams at PayPal and Wiki Loves Monuments, came out of the curated combination of pytest dependency-injection semantics with werkzeug primitives.

Realistically, the majority of creation happens below the threshold of standalone artifacts. For instance, when adding a feature to an existing system, a parallel approach in a different project serves as a useful guide. I've lost track of the number of times I've swiped techniques from Awesome Python Applications, including ones used to port my dayjob's 300k SLOC codebase from Python 2 to 3.

Most creative outputs have a similar lineage. Only now we have an explicit process.

Debugging the process

It's easy to see creations we appreciate as towering achievements that sprung fully-formed from their creators' genius. But creation comes in fits and starts. If creation comes slowly, here are a few strategies to consider:

Search for a natural split in an existing collection that's getting too big, and explore what makes it interesting.
Revisit an old, contentious critique and re-react. What did you get right/wrong?
Pick a particular exemplar and turn it into a case study. One beautiful aspect of FOSS projects is that going deep can mean getting involved. There's nothing like proximity to a problem to inspire creative thinking.

More generally, be wary of one-size-fits-all solutions; while prescriptive techniques such as the Zettelkasten Method may work for some, creation is idiosyncratic. Embrace your own process.

Putting it into practice

When inspiration hits, connections can form so quickly that we take for granted what goes on. When inspiration proves less willing to strike, we can keep ourselves primed for creativity by ensuring all four activities continue in balance.

There are a few notable benefits of intentional creation:

When you've built something, the influences are well-documented. It can be easier to involve others when there's a clear creative thread to pull on.
Sharing your critiques and curations invites collaboration with other creators and curators.
Self-awareness. If you're not finding your critiques crystallizing into new thoughts and ideas for projects, that's a sign you're looking at the wrong stuff. Are you following your interests or passively consuming trending content?

Practically, intentional creation means consciously spending less time on consumer sites, from Twitter to Hacker News, and more time taking notes, tagging bookmarks, and creating your own knowledge base. Attempt activities that are less entertainment and more you, ultimately closing the gap between you and your creative goals.

If it sounds too simple, that's because it is. You're still accountable to you, that's the hard part. But hopefully you'll find some value in this simple hierarchy that lets you check in on your own activities and make adjustments toward a more creative end. Spend less time consuming, and more time on the other three Cs. Consume only enough to allow yourself to critique, curate, and create.

If you made it this far, then start now. Step 2. Use any tool or service you like, from spreadsheets to YAML, and answer this: What's your critique?

Changing the Tires on a Moving Codebase

2021-03-10T16:30:00Z

2020 was a year of reckonings. And for all that was beyond one’s control, as the year went on, I found myself pouring more and more into the one thing that felt within reach: futureproofing of the large enterprise web application I helped build, SimpleLegal.

Now complete, this replatforming easily ranks in my most complex projects, and right now, holds the top spot for the happiest ending. That happiness comes at a cost, but with some the right approach that cost may not be as high as you think.

Contents

The Bottom Line
The Setup
The Outset
The Traction Issues
The Sentry Pivot
The New Road
The Rollout
The Aftermath

The Bottom Line

We took SimpleLegal’s primary product, a 300,000 line Django-1.11-Python 2.7-Redis-Postgres-10 codebase, to a Django 2.2-Python 3.8-Postgres-12 stack, on-schedule and without major site incidents. And it feels amazing.

Speaking as tech lead on the project, what did it look like? For me, something like this:

But as Director of Engineering, what did it cost? 3.5 dev years and just about $2 per line of code.

And I'm especially proud of that result, because along the way, we also substantially improved the speed and reliability of both the site and development process itself. The product now has a bright future ahead, ready to shine in sales RFPs and compliance questionnaires. Most importantly, there’ll be no worrying about when to delicately break it to a candidate that they’ll be working with unsupported technology.

In short, a large, solid investment that’s already paying for itself. If you just came here for the estimate we wish we had, you've got it. This post is all about how your team can achieve the same result, if not better.

The Setup

The story begins in 2013, when a freshly YC-incubated SimpleLegal made all the right decisions for a new SaaS LegalTech company: Python, Django, Postgres, Redis. In classic startup fashion, features came first, unless technology was a blocker. Packages were only upgraded incidentally.

By 2019, the end of this technical runway had drawn near. While Python 2 may be getting extended support from various vendors, there were precious few volunteers in sight to do Django 1 CVE patches in 2021. A web framework’s a riskier attack surface, so we finally had our compliance forcing function, and it was time to pay off our tech debt.

The Outset

So began our Tech Refresh replatforming initiative, in Q4 2019. The goal: Upgrade the stack while still shipping features, like changing the tires of a moving car. We wanted to do it carefully, and that would take time. Here are some helpful ground rules for long-running projects:

Any project that gets worked on 10+ hours per week deserves a 30-minute weekly sync.
Every recurring meeting deserves a log. Put it in the invite. Use that Project Log to record progress, blockers, and decisions.
It’s a marathon, not a sprint. Avoid relying on working nights, weekends, and holidays.

We started with a sketch of a plan that, generously interpreted, ended up being about halfway correct. Some early guesses that turned into successes:

Move to pip-tools and unpin dependencies based on extensive changelog analysis. Identify packages without py23 compatible versions. (Though we’ve since moved to poetry.)
Add line coverage reporting to CI
Revamp internal testing framework to allow devs to quickly write tests

More on these below. Other plans weren’t so realistic:

Take our CI from ~60% to 95% line coverage in 6 months
Parallelized conversion of app packages over the course of 3 months
Use low traffic times around USA holidays (Thanksgiving, Christmas, New Years) to gradually roll onto the new app before 2021.

We were young! As naïve as we were, at least we knew it would be a lot of work. To help shoulder the burden, we scouted, hired, and trained three dedicated off-shore developers.

The Traction Issues

Even with added developers, by mid-2020 it was becoming obvious we were dreaming about 95% coverage, let alone 100%. Total coverage may be best practice, but 3.5 developers couldn’t cover enough ground. We were getting valuable tests, and even finding old bugs, but if we stuck with the letter of the plan, Django 2 would end up being a 2022 project. At 70%, we decided it was time to pivot.

We realized that CI is more sensitive than most users for most of the site. So we focused in on testing the highest impact code. What’s high-impact? 1) the code that fails most visibly and 2) the code that’s hardest to retry. You can build an inventory of high-impact code in under a week by looking at traffic stats, batch job schedules, and asking your support staff.

Around 80% of the codebase falls outside that high-traffic/high-impact list. What to do about that 80%? Lean in on error detection and fast time-to-fix.

The Sentry Pivot

One nice thing about startup life is that it’s easy to try new tools. One practice we’ve embraced at SimpleLegal is to reserve every 5th week for developers to work on the development process itself, like a coordinated 20% time. Even the best chef can’t cook five-star food in a messy kitchen. This was our way of cleaning up the shop and ultimately speeding up the ship.

During one such period, someone had the genius idea to add dedicated error reporting to the system, using Sentry. Within a day or two, we had a site you could visit and get stack traces. It was pretty magical, and it wasn’t until Tech Refresh that we realized that while integration takes one dev-day, full adoption can take a team months.

You see, adding Sentry to a mature-but-fast-moving system means one thing: noise. Our live site was erroring all the time. Most errors weren’t visible or didn’t block users, who in some cases had quietly learned to work around longstanding site quirks. Pretty quickly, our developers learned to treat Sentry as a repository of debugging information. A Sentry event on its own wasn’t something to be taken seriously in 2019. That changed in 2020, with the team responsible for delivering a seamless replatform needing Sentry to be something else: a responsive site quality tool.

How did we get there? First step, enhance the data flowing into Sentry by following these best practices:

Split up your products into separate Sentry projects. This includes your frontend and backend.
Tag your releases. Don’t tag dev env deployments with the branch, it clutters up the Releases UI. Add a separate branch tag for searches.
Split up your environments. This is critical for directing alerts. Our Sentry client environment is configured by domain conventions and Django’s sites framework. If it helps, here's a baseline, we use these environments:
- Production: Current official release. DevOps monitored.
- Sandbox: Current official release (some companies do next release). Used by customers to test changes. DevOps monitored.
- Demo/Sales: Previous official release. Mostly internal traffic, but external visibility at prospect demo time. DevOps monitored.
- Canary: Next official release. Otherwise known as staging. Internal traffic. Dev monitored.
- ProdQA: Current official release. Used internally to reproduce support issues. Dev monitored.
- QA: Dev branches, dev release, internal traffic. Unmonitored debugging data.
- Local test/CI: Not published to Sentry by default.

With issues finally properly tagged and searchable, we used Sentry’s new Discover tool to export issues weekly, and prioritize legacy errors. To start, we focused on high-visibility production errors with non-internal human users. Our specific query: has:user !transaction:/api/* event.type:error !user.username:*@simplelegal.*

We triaged into 4 categories: Quick fix (minor bug), Quick error (turn an opaque 500 error into a actionable 400 of some form), Spike (larger bug, requires research), and Silence (using Sentry’s ignore feature). Over 6 weeks we went from over 2500 weekly events down to less than 500.

Further efforts have gotten us under 100 events per week, spread across a handful of issues, which is more than manageable for even a lean team. While "Sentry Zero" remains the ideal, we achieved and maintained the real goal of a responsive flow, in large part thanks to the Slack integration. Our team no longer hears about server errors from our Support team. In fact, these days, we let them know when a client is having trouble and we’ve got a ticket underway.

And it really is important to develop close ties with your support team. Embedded in our strategy above was that CI is much more sensitive than a real user. While perfection is tempting, it’s not unrealistic to ask a bit of patience from an enterprise user, provided your support team is prepared. Sync with them weekly so surprise is minimized. If they’re feeling ambitious, you can teach them some Sentry basics, too.

The New Road

With noise virtually eliminated, we were ready to move fast. While the lean-in on fast-fixing Sentry issues was necessary, a strong reactive game is only useful if there are proactive changes being pushed. Here are some highlights we learned when making those changes:

Committing to transactions

Used properly, rollbacks can make it like errors never happened, the perfect complement to a fast-fix strategy.

The truly atomic request

Get as much as possible into the transactions. Turn on ATOMIC_REQUESTS, if you haven’t already. Some requests do more than change the database, though, like sending notifications and enqueuing background tasks.

At SimpleLegal, we rearchitected to defer all side effects (except logging) until a successful response was being returned. Middleware can help, but mainly we achieved this by getting rid of our Redis queue, and switching to a PostgreSQL-backed task queue/broker. This arrangement ensures that if an error occurs, the transaction is rolled back, no tasks are enqueued, and the user gets a clean failure. We spot the breakage in Sentry, toggle over to the old site to unblock, and their next retry succeeds.

Transactional test setup

Transactionality also proved key to our testing strategy. SimpleLegal had long outgrown Django’s primitive fixture system. Most tests required complex Python to set up, making tests slow to write and slow to run. To speed up both writing and running, we wrapped the whole test session in a transaction, then, before any test cases run, we set up exemplary base states. Test cases used these base states as fixtures, and rolled back to the base state after every test case. See this conftest.py excerpt for details.

Better than best practices

Software scenarios vary so widely, there’s an art to knowing which advice isn’t for you. Here’s an assortment of cul de sacs we learned about firsthand.

The utility of namespaces

Given how code is divided into modules, packages, Django apps, etc., it may be tempting to treat those as units of work. Don’t start there. Code divisions can be pretty arbitrary, and it’s hard to know when you’ve pulled on a risky thread.

Assuming there are automated refactorings, as in a 2to3 conversion, start by porting by type of transformation. That way, one need only review a command and a list of paths affected. Plus, automated fixes necessarily follow a pattern, meaning more people can fix bugs arising from the refactor.

Coverage tools

Coverage was a mixed bag for us. Obviously our coverage-first strategy wasn’t tenable, but it was still useful for prioritization and status checks. On a per-change basis, we found coverage tools to be somewhat unreliable. We never got to the bottom of why coverage acted nondeterministically, and we left the conclusion at, “off-the-shelf tools like codecov are probably not targeted at monorepos of our scale.”

In running into coverage walls, we ended up exploring many other interpretations of coverage. For us, much higher-priority than line coverage were “route coverage” (i.e., every URL has at least one integration test) and “model repr coverage” (i.e., every model object had a useful text representation, useful for debugging in Sentry). With more time, we would have liked to build tools around those, and even around online-profiling based coverage statistics, to prioritize the highest traffic lines, not just the highest traffic routes. If you’ve heard of approaches to these ends, we’d love to discuss them with you.

Flattening database migrations

On the surface, reducing the number of files we needed to upgrade seems logical. Turns out, flattening migrations is a low-payoff strategy to get rid of files. Changing historical migration file structure complicated our rollout, while upgrading migrations we didn’t flatten was straightforward. Not to mention, if you just wanted the CI speedup, you can take the same page from the Open EdX Platform that we did: build a base DB cache that you check in every couple months.

Turns out, you can learn a lot from open-source applications.

Easing onto the stack

If you have more than one application, use the smaller, simpler application to pilot changes. We were lucky enough to have a separate app whose tests ran faster, making for a tighter development loop we coul learn from. Likewise, if you have more than one production environment, start rollouts with the one with the least impact.

Clone your CI jobs for the new stack, too. They’ll all fail, but resist the urge to mark them as optional. Instead, build a single-file inventory of all tests and their current testing state. We built a small extension for our test runner, pytest, which bulk skipped tests based on a status inventory file. Then, ratchet: unskip and fix a test, update the file, check that tests pass, and repeat. Much more convenient and scannable than pytest mark decorators spread throughout the codebase. See this conftest.py excerpt for details.

The Rollout

In Q4 2020, we doubled up on infrastructure to run the old and new sites in parallel, backed by the same database. We got into a loop of enabling traffic to the new stack, building a queue of Sentry issues to fix, and switching it back off, while tracking the time. After around 120 hours of new stack, strategically spread around the clock and week, enough organizational confidence had been built that we could leave the site on during our most critical hours: Mondays and Tuesdays at the beginning of the month.

The sole hiccup was an AWS outage Thanksgiving week. At this point we were ahead of schedule, and enough confidence had been built in our fast-fix workflow that we didn’t need our original holiday testing windows. And for that, many thanks were given.

We kept at the fast-fix crank until we were done. Done isn't when the new system has no errors, it's when traffic on the new system has fewer events than the old system. Then, fix forward, and start scheduling time to delete the scaffolding.

The Aftermath

So, once you’re on current LTS versions of Django, Python, Linux, and Postgres, job complete, right?

Thankfully, tech debt never quite hits 0. While updating and replacing core technologies on a schedule is no small feat, replacing a rusty part with a shiny one doesn’t change a design. Architectural tech debt -- mistakes in abstractions, including the lack thereof -- can present an even greater challenge. Solutions to those problems don’t generalize between projects as cleanly, but they do benefit from up-to-date and error-free foundations.

For all the projects looking to add tread to their technical tires, we hope this retrospective helps you confidently and pragmatically retrofit your stack for years to come.

Finally, big thanks to Uvik for the talent connection, and the talent: Yaroslav, Serhii, and Oleh. Shoutouts to Kurt, Justin, and Chris, my fellow leads. And the cheers to business leadership at SimpleLegal and everywhere, for seeing the value in maintainability.

Thanks, 201X!

2019-12-02T18:00:00Z

Thought I'd take a Sunday afternoon to reflect on, oh I don't know, a decade.

Been a long ten years, but it's flown past. This particular decade happens to coincide with my first years of full-time professional software engineering.

The Quantity

I can't possibly summarize it all, and if I tried, it'd still be colored by what's on my mind right now. But I can point to the artifacts I tried to leave along the way:

Twitter FWIW¹ (2008+)
~20 Open-Source Projects (2012+)
~15 Hatnote Projects (2013+, follow us)
~25 entries on this blog (2015+)
+7 here (2014-2016)
Not including pythondoeswhat.com or blog.hatnote.com
(or other posts on the blogs only real heads know)
~10 Talks (2016+)
Lest I forget: O'Reilly's Enterprise Software with Python (2016)
And several podcast/media appearances
calver.org (2016) and 0ver.org (2018) (Versioning is a fun pastime)
Pyninsula (2017+) - YouTube, Meetup, Email Announce

Taking a chronological look at each of the above, I'm relieved to see obvious growth.

If I were to highlight one resource, it would probably be the talks. Despite the stress of preparation and delivery, I'm least concerned with having a massive miscommunication when we're all in the room and I can see the points hitting home. It's impossible to pick a favorite, but Ask the Ecosystem (2019), the Restructuring Data lightning talk (2018), and The Packaging Gradient (2017) seem like audience faves from where I'm sitting.

The Quality

Each project, post, and talk had its own reward, but I guess I've got more than just those to show for the decade.

On the more profit-driven side, I built tools and teams at PayPal, but once I could manage the risk, I got to dip into startups for the last few years. Lucky for me, it wasn't a total bust, and the wife and I bought a place in my favorite neighborhood (in the USA). Not a millionaire, but I'm hoping and working for a world where no one has to be.

More recently, the Python Software Foundation made me a Fellow. This isn't something I can be nonchalant about, and I'm not going to understate how much this means, to me, working in a field like software, where concrete symbols of progress are alternatingly elusive and vanishing. Plus it's Python, and reciprocated love is nice. I have hundreds of people to thank for helping me reach this point, and I have to thank the PSF for dedicating the time to ramping up these awards. They've convinced me more than ever that we need more institutions to build this sort of advancement.

To all of you, thank you.

The Struggle

I like to think I managed to do all of the above while staying away from industry hype, on the principle that massive speculative capital influx isn't where real value is added to society, and doesn't generate the kind of innovation that excites me.

I may have been naïve, but I came to Silicon Valley with an idea about the transformative power of software. Changing times may illustrate a grittier interpretation than the one I had and have, but I continue to hold dear software's potential for positive impact. If you've felt that vision waver, let me tell you, you're not alone.

In the past decade, I've seen too many engineers sucked in by new technologies and ventures, only to find themselves alienated from their work. Episodes ranging from an afternoon lost to debugging Docker/k8s clusters, to years of work disappearing at the end of a VC runway. Nothing has been harder to watch than those bedraggled-but-persistent idealists regroup, each time a bit more cynical than the last.

Even if its seeming intractibility has taken it from the center stage, the burnout conversation continues to smolder, because there's no issue realer. I know; I released more ceramics than software back in 2014.

Some problems can be solved by paying the maintainers, but I think the vastly bigger issue is around losing the human connection between the real effort software takes and the real benefits it brings, combined with FOSS's dearth of collaborators in supporting roles (QA, product/project/release management).

That's why I'm incredibly thankful for the Wikimedia community for always being there, patient with schedules and issues, as long as the software got the job done. It can be a challenge to juggle projects, but I tell every budding engineer: find that direct connection to people who will appreciate your work, and avoid cynicism at all costs.

There are some interesting prospects in the works, but I'm keeping this post retro. Besides, if 2029 rolls around and all I did was break even with 2009-19, I don't see how I can be disappointed.

Thanks again for everything in 201X, and for sticking with me in 202X.

Despite using Twitter for over a decade, the process of tweeting feels so perfunctory, and the service itself so tenuous, that I still can't bring myself to invest the time. I mostly use it to crosspost my blog posts or help friends promote their posts/projects.

But until I start an email newsletter, or really get on top of yak.party, it's still the best I got for announcing where I'm speaking next. ↩

Awesome Python Applications

2018-12-20T19:20:00Z

What we can learn from 180+ case studies on successfully shipping Python software.

If you're reading this (or hearing this), you read and write code, probably Python. And for all the code you've shipped, you've probably had your share of missed requirements. Somehere in the excitement of software abstraction, we can lose sight of what really matters, what makes our well-factored modules and packages and frameworks turn into real-world applications.

That's why I'm announcing Awesome Python Applications, a hand-curated list of 180+ projects, all of which are:

Free software with an online source repository.
Using Python for a considerable part of their functionality.
Well-known, or at least prominently used in an identifiable niche.
Maintained or otherwise demonstrably still functional on relevant platforms.
Shipped applications, not libraries or frameworks.

The result is a list of predominantly focused on software that installs without pip or PyPI, and whose audience is mostly not developers. There's still plenty of that in there, too, with other exceptions, but the breadth of the list speaks for itself.

So why spend weeks cataloguing open-source Python applications?

Aside from holiday cheer, three big reasons.

Contents

Goal #1: A Better Development Cycle
Goal #2: A Complete Python Production Loop
Goal #3: Grounding for the Python Ecosystem
Next steps

Goal #1: A Better Development Cycle

Ever since I started talking about Python packaging, people have been asking me questions about which packaging technique is best for their software. I was struck, over and over again, how far people can get in developing an application before reaching the fundamental question of delivery. Exploring this, I landed on a more basic question:

Why are so many people building applications from first principles (blog posts and Stack Overflow)?

Isn't Python one of the biggest names in the software world? Aren't there dozens of successful, real-world applications written in Python? What are the chances your application is totally unique?

Awesome Python Applications attempts to open up a new flow for answering the toughest development questions.

When building an application, scan the list to find projects which most closely match your project's requirements. Then, use that application as a guide for answering your own questions. This works especially well for abstract questions surrounding architecture, deployment, and testing.

Back in school, I learned more about architecture and software development from the MediaWiki source code than I did from any class. It continues to inspire me to this day. APA is the next step in enabling the holistic education of a working application with real users.

In short, while we may lack the time to write them, each production application is worth a thousand blog posts.

Goal #2: A Complete Python Production Loop

We Python programmers are also software users. But unlike other software users, we know how to file issues and may even make significant contributions back to our applications of choice.

By choosing Python software when possible, we take one step closer to pitching in. What better way for a future application developer to get started?

I would love to see more developers connect with software they didn't realize was Python. My (minor) contributions to the Twisted were greatly energized by the knowledge that one of my favorite applications, Deluge, heavily used the library. Using free software leads to creating more free software.

Goal #3: Grounding for the Python Ecosystem

With the pace and cerebrality of technology, it can be easy to get ahead of ourselves and our end users. Infrastructure devs get disconnected from application devs, and that makes for worse software over time. This problem is compounded when applications get less developer attention. Most APA entries have three- and even two-digit starcounts, unless users are highly technical. Few major Python applications are distributed with PyPI, so download statistics can't help us either. Even if they did, lower-level libraries have way more fanout. And of course free software projects can't lay down big donations or conference sponsorships, so representation tends to be pretty sparse all around.

These applications represent the best of the free and living portion of Python. Not only are they a source of utility and pride, but they need our support, in spirit and in practice. It is my sincere hope that the APA will help to anchor the Python community in its real-world applications.

What does this mean, concretely? A keen eye will notice how the list is structured. This isn't just for consistent rendering, but an attempt at an API for the dataset. We must explore our ecosystem with the relzationship between libraries and applications in mind.

I know I'm going out on a limb here, and metrics aren't everything, but it would be very interesting to see the Python FOSS ecosystem explored as an analogue of the scientific publishing framework. Can we get some sort of developer h-index by treating libraries as "articles" and applications as "journals"? Adding in some application userbase approximations (via social altmetrics and other means) can give us much deeper insight into real-world impact.

Next steps

If this essay seems shorter than my usual, that's because it's really an introduction to the list itself. I got caught up in several projects' codebases while doing the research, and you will, too.

If we've missed a project, please open an issue or PR. If you're as excited about this as I am, consider helping with some of the open issues. There are still a lot of application features to survey: licenses, Python versions, frameworks, and more. And as always, watch this space (and the repo) for updates as we make more discoveries!

Announcing glom: Restructured Data for Python

2018-05-09T17:00:00Z

This post introduces glom, Python's missing operator for nested objects and data.

If you're an easy sell, full API docs and tutorial are already available at glom.readthedocs.io.
Harder sells, this 5-minute post is for you.
Really hard sells met me at PyCon,
where I gave this 5-minute talk.

The Spectre of Structure

In the Python world, there's a saying: "Flat is better than nested."

Maybe times have changed or maybe that adage just applies more to code than data. In spite of the warning, nested data continues to grow, from document stores to RPC systems to structured logs to plain ol' JSON web services.

After all, if "flat" was the be-all-end-all, why would namespaces be one honking great idea? Nobody likes artificial flatness, nobody wants to call a function with 40 arguments.

Nested data is tricky though. Reaching into deeply structured data can get you some ugly errors. Consider this simple line:

value = target.a['b']['c']

That single line can result in at least four different exceptions, each less helpful than the last:

AttributeError: 'TargetType' object has no attribute 'a'
KeyError: 'b'
TypeError: 'NoneType' object has no attribute '__getitem__'
TypeError: list indices must be integers, not str

Clearly, we need our tools to catch up to our nested data.

Enter glom.

Restructuring Data

glom is a new approach to working with data in Python, featuring:

Path-based access for nested structures
Declarative data transformation using lightweight, Pythonic specifications
Readable, meaningful error messages
Built-in data exploration and debugging features

A tool as simple and powerful as glom attracts many comparisons.

While similarities exist, and are often intentional, glom differs from other offerings in a few ways:

Going Beyond Access

Many nested data tools simply perform deep gets and searches, stopping short after solving the problem posed above. Realizing that access almost always precedes assignment, glom takes the paradigm further, enabling total declarative transformation of the data.

By way of introduction, let's start off with space-age access, the classic "deep-get":

from glom import glom

target = {'galaxy': {'system': {'planet': 'jupiter'}}}
spec = 'galaxy.system.planet'

output = glom(target, spec)
# output = 'jupiter'

Some quick terminology:

target is our data, be it dict, list, or any other object
spec is what we want output to be

With output = glom(target, spec) committed to memory, we're ready for some new requirements.

Our astronomers want to focus in on the Solar system, and represent planets as a list. Let's restructure the data to make a list of names:

target = {'system': {'planets': [{'name': 'earth'}, {'name': 'jupiter'}]}}

glom(target, ('system.planets', ['name']))
# ['earth', 'jupiter']

And let's say we want to capture a parallel list of moon counts with the names as well:

target = {'system': {'planets': [{'name': 'earth', 'moons': 1},
                                 {'name': 'jupiter', 'moons': 69}]}}

spec = {'names': ('system.planets', ['name']),
        'moons': ('system.planets', ['moons'])}

glom(target, spec)
# {'names': ['earth', 'jupiter'], 'moons': [1, 69]}

We can react to changing data requirements as fast as the data itself can change, naturally restructuring our results, despite the input's nested nature. Like a list comprehension, but for nested data, our code mirrors our output.

And we're just getting started.

True Python-Native

Most other implementations are limited to a particular data format or pure model, be it jmespath or XPath/XSLT. glom makes no such sacrifices of practicality, harnessing the full power of Python itself.

Going back to our example, let's say we wanted to get an aggregate moon count:

target = {'system': {'planets': [{'name': 'earth', 'moons': 1},
                                 {'name': 'jupiter', 'moons': 69}]}}


glom(target, {'moon_count': ('system.planets', ['moons'], sum)})
# {'moon_count': 70}

With glom, you have full access to Python at any given moment. Pass values to functions, whether built-in, imported, or defined inline with lambda. But glom doesn't stop there.

Now we get to one of my favorite features by far. Leaning into Python's power, we unlock the following syntax:

from glom import T

spec = T['system']['planets'][-1].values()

glom(target, spec)
# ['jupiter', 69]

What just happened?

T stands for target, and it acts as your data's stunt double. T records every key you get, every attribute you access, every index you index, and every method you call. And out comes a spec that's usable like any other.

No more worrying if an attribute is None or a key isn't set. Take that leap with T. T never raises an exception, so worst case you get a meaningful error message when you run glom() on it.

And if you're ok with the data not being there, just set a default:

glom(target, T['system']['comets'][-1], default=None)
# None

Finally, null-coalescing operators for Python!

But so much more. This kind of dynamism is what made me fall in love with Python. No other language could do it quite like this.

That's why glom will always be a Python library first and a CLI second. Oh, didn't I mention there was a CLI?

Library first, then CLI

Tools like jq provide a lot of value on the console, but leave a dubious path forward for further integration. glom's full-featured command-line interface is only a stepping stone to using it more extensively inside application logic.

$ pip install glom
$ curl -s https://api.github.com/repos/mahmoud/glom/events \
  | glom '[{"type": "type", "date": "created_at", "user": "actor.login"}]'

Which gets us:

[
  {
    "date": "2018-05-09T03:39:44Z",
    "type": "WatchEvent",
    "user": "asapzacy"
  },
  {
    "date": "2018-05-08T22:51:46Z",
    "type": "WatchEvent",
    "user": "CameronCairns"
  },
  {
    "date": "2018-05-08T03:27:27Z",
    "type": "PushEvent",
    "user": "mahmoud"
  },
  {
    "date": "2018-05-08T03:27:27Z",
    "type": "PullRequestEvent",
    "user": "mahmoud"
  }
...
]

Piping hot JSON into glom with a cool Python literal spec, with pretty-printed JSON out. A great way to process and filter API calls, and explore some data. Something genuinely enjoyable, because you know you won't be stuck in a pipe dream.

Everything on the command line ports directly into production-grade Python, complete with better error handling and limitless integration possibilities.

Next steps

Never before glom have I put a piece of code into production so quickly.

Within two weeks of the first commit, glom has paid its weight in gold, with glom specs replacing Django Rest Framework code 2x to 5x their size, making the codebase faster and more readable. Meanwhile, glom's core is so tight that we're on pace to have more docs and tests than code very soon.

The glom() function is stable, along with the rest of the API, unless otherwise specified.

A lot of other features are baking or in the works. For now, we'll be focusing on the following growth areas:

Validation functionality, in the vein of schema and cerberus
CLI robustness, better error messages, etc.
Extension API, clean up some internal code, open up extensions
Automatic default registration of default behaviors for co-installed packages (e.g., Django)

We'll be talking about all of this and more at PyCon, so swing by if you can. In either case, I hope you'll try glom out and let us know how it goes!

Maintainerati 2017: GitHub Design

2017-10-18T05:00:00Z

Last week I attended a Maintainerati event, an unconference/mini-summit for maintainers of popular software, run as a prelude to the GitHub Universe conference. After being brought up to speed on this year's secret handshake of the software elite, I had a great time in the documentation breakout group, as well as moderating a lively discussion on diversity in open-source, both of which deserve their own write-ups at some point.

Once those were through and coffee breaks were had, what I consider the main event was upon us: An opportunity to discuss with GitHub designers and developers all the different ways projects use GitHub, and how GitHub might improve to match those use cases. I think these interactions have the most direct potential to bear fruit, so in my excitement I wrote a bunch of the proceedings down:

Contents

Digested emails
- Star spectrum
Code review permissions
Dashboard improvements
Thanks!

Digested emails

Discussion is one of the greatest things about GitHub, but as Jupyter developer Ian Rose brought up, an email for every comment can be overwhelming. Daily or weekly digests for issues, or even for all your GitHub activity would be a huge improvement, especially for more lightweight users of GitHub. This may not strike subscribers of The Weeklypedia as a surprise, but I am a big fan of email digests.

More control over engagement levels could open a great new avenue for driving traffic for GitHub, too.

Star spectrum

Right now you can either star repos or watch them, effectively getting no notifications or all of them.

I'd estimate about a third of the repos I star look interesting, but haven't yet reached the point where I'd use or contribute to them. So they mostly get starred and forgotten.

A friend of mine started a little project called Starminder, which emails a nightly selection of five of my starred repos. I've been having a grand time revisiting these old stars and seeing how far they've come, even reminding me of features I was waiting to build.

And while I love Nik's work, instead of relying on Starminder, it would be way better if I could tell GitHub roughly how often I'd like updates on a project, and then get Pulse-like info delivered to my inbox on a weekly or monthly basis.

Commit activity, high-traffic issues, and especially new tags/releases are all things I'd be very excited to get personalized, periodic updates on, without having to get every single notification as a separate email.

One off-the-cuff idea I had was to establish some sort of star gradient, with the basic star without notifications being an option on one end of the opt-in engagement spectrum, and full-blown, every-notification "Watching" on the other. Could there be one Star dropdown to rule them all?

Code review permissions

Requiring code review before merging is a pretty smart idea for any rigorous project, and now GitHub supports it natively. However, only developers with write permissions can actually perform a code review. Here are some real-life use cases that demonstrate why this is less than ideal:

A senior developer not involved with the project files an issue requesting a feature. I would like them to review the implementation to ensure it does what they want. The senior developer has a busy schedule and doesn't want to join the project and get a bunch of notifications, but would be qualified to review the code.
A novice developer finds an problem in the documentation, they could review the new documentation for clarity. Their lack of experience makes them best qualified to review.
The core maintainer implements a feature, but is actually the only developer on the project. Requesting a code review from non-project-member peers is a great way to get them to look at the code and become more involved with the project going forward.

For bonus points all permissions could have an option to be time-limited. Designated reviewr and expiration possibilities notwithstanding, I think the best flow would include the ability to add someone to a specific PR as reviewer, without giving them any project-wide permissions.

Dashboard improvements

I'm probably weird for doing this, but I habitually visit the normal github.com logged-in landing page, aka the dashboard, several times a day. Now, for the few of you who share my habit probably noticed, there's a new Discover tab, offering personalized suggestions of repos to star.

The event stream stayed mostly the same, however, as it has for many years. But despite its maturity there are a couple events that surprisingly don't show up anywhere, even when the dashboard seems like a natural fit:

Follows - I have the better part of a thousand followers, but I can't remember if I've ever seen a notification about this. They seem like nice folks!
Stars on org-owned repos - There are several repos I maintain and watch, but for which I've never seen on-dash notifications. What do they all have in common? They're all owned by organizations (e.g., python-hyper). Other types of notifications show up, but not stars.
Watches - Not sure I've ever gotten a notification for someone watching one of my repos, even though they're probably more interested in collaboration than the average stargazer.

Any or all of these would certainly make my github.com itch yield more interesting results, and I'm sure there are some enhancements I've missed, too!

Thanks!

Just wanted to say thanks to GitHub for putting together such a great event. Whether or not any of these features materializes in the near future, it was so nice to meet up with old friends and make some new ones, too.

Focused, cross-technology encounters like these are all too rare. For my Python readers, let this serve as a reminder to get out and interact with other stacks. Python's strength is its integrative nature, and I think that can be a strength for us Pythonists as well.

In any case, thanks for the event, GitHub! Hope to see you again next year!

Plugin Systems

2017-07-11T18:00:00Z

"What are plugins?" and other proceedings of the inaugural PyCon Comparative Plugin Systems BoF.

Update: This BoF and post inspired [a talk I gave at PyGotham 2017][pygotham2017].

Within the programming world, and the Python ecosystem in particular, there are a lot of presumptions around plugins. Specifically, we take them for granted. "It's just a plugin." "Oh, another plugin library?"

So for PyCon 2017, I resolved to dismiss the dismissals by revisiting plugins, and it may have been the best programming decision I've made all year.

Contents

Why plugins?
Setting examples
Taxonomizing
Drawing a line
A definition
Motivation
In conclusion

Why plugins?

For all types of software, open-source or otherwise, the scalability of development poses a problem long before scalability of performance and other technical challenges. Engaging more developers creates code contention and bugs. Too many cooks is all it takes to spoil the broth.

All growing projects need an API for code integration.

Call them plugins, modules, or extensions, from your browser to your kernel, they are the widely successful solution. Tellingly, the only thing wider than the success of plugin-based architecture is the variety of implementations.

Python's dynamic nature in particular seems to encourage inventiveness. The more the merrier, usually, but at some point we cloud a tricky space. How different could these plugin systems be? How wide is the range of functionalities, really? How does a developer choose the right plugin system for a given project? For that matter, what is a plugin system anyway? No one I talked to had clear answers.

So when PyCon 2017 rolled around, I knew exactly what I wanted to do: call together a team of developers to get to the bottom of the above, or at the very least, answer the question,

"What happens when you ask a dozen veteran Python programmers to spill their guts about plugins?"

Setting examples

Our group leapt into action by listing off plugin systems as fast as we could:

stevedore
twisted.plugin
Mercurial extensions
pytest plugins (pluggy)
gather
venusian
pluginbase
straight.plugin
pylint plugins
flake8 plugins
raw setuptools entrypoints
zope.component
Django command extensions
SQLAlchemy dialects/DBAPIs
Sphinx extensions
Buildout extensions
Pike
Dectate and Reg
Others that came and went a little too fast to jot down

With our plate heaping with examples like these, we all felt ready to dig into our big questions.

Taxonomizing

For our first bit of analysis, we asked: What practical and fundamental attributes differentiate these approaches? If we had to create a taxonomy, what characteristics would we look for?

Generalizability

You'll notice our list of example plugin systems included several very specialized examples, from pylint to SQLAlchemy. Many projects even use totally internal plugin systems to achieve better factoring.

Bespoke plugin systems like pylint's are a valuable reference for anyone looking to account for patterns in their own system, especially generic systems like pike and stevedore.

Discovery

A plugin system's first job is locating the plugins to load. The split here is whether plugins are individually specified, or automatically discovered based on paths and patterns.

In either case, we need paths. Some systems provide search functionality, exchanging explicitness for convenience. This can be a good trade, especially when plugins number in the double digits, or whenever less technical users are concerned.

Install location

Closely related to discovery, our next differentiator was the degree to which the plugin system leveraged Python's own package management facilities. Some systems, like venusian, were designed to encourage pip install-ing plugins, searching for them in site-packages, alongside the application itself.

Other systems have their own search paths, locating plugins in the user directory and elsewhere on the filesystem. Still other systems are designed for plugins inside the application tree, as is the case with Django apps.

Plugin independence

One of the most challenging parts of plugin development is finding ways of independently reusing and testing code, while keeping in mind the code's role as an optional component of another application.

In some systems, like Django's, the tailoring is so tightly coupled that reusability doesn't make sense. But other approaches, like gather's, keeps plugin code independently usable.

Dependency registration

Almost all plugins work by providing some set of hooks which are findable and callable by the core. We found another differentiator in whether and how plugins could gain access to resources from the core, and even other plugins.

Not all systems support this, preferring to keep plugins as leaf participants in the application. Those simplistic setups hit limits fast. The next best, and most common, solution is to simply pass the whole core state at the time of hook invocation, providing plugins with the same access as the core. It works, but the API becomes the whole system state.

More advanced systems allow plugins to publish an inventory of dependencies, which the core then injects. Higher granularity enables lazier evaluation for a performance boost, and more explicit structure helps create a more maintainable application overall.

Drawing a line

With our group feeling like we were approaching the nature of things, we reversed direction, asking instead: What isn't a plugin system?

Establishing explicit boundaries and specific counterexamples proved instrumental to producing a final definition.

Is eval() a plugin system? We thought maybe, at first. But the more we thought about it, no, because the code itself was not sufficiently abstracted through a loading or namespacing system.

Is DNS a plugin system? It has names and namespaces galore. But no, because code is not being loaded in. Remote services in general are beyond the boundary of what a plugin can be. They exist out there, and we call out to them. They're callouts, not plugins.

A definition

So with our boundaries established, we were ready to offer a definition:

A plugin system is a software facility used by a running program to discover and load code, often containing hooks called by the host application

But, by this definition, isn't Python's built-in import functionality a plugin system? Mostly, yes! Python's import system is a plugin system.

For discovery it uses sys.path, various "site" directories and ".pth" files, and much more.
For installation, it uses site-packages, user .local directories, and more.
As far as independent reusability, virtually every module can be made its own entrypoint.
As for dependency registration, every module is tossed into sys.modules with the others, but also has access to import and sys, making roughly every module an equal partner in application state.

Python's import system is a powerful one, with a plugin system of its own. But finders, loaders, and import hooks aren't Python's plugin system. For that, you need to look to the site module.

Motivation

With our hour nearly up, all these proximate details still needed to be distilled into an ultimate motivation behind plugins. To this end, we returned to one of software engineering's fundamental principles: Separation of concerns.

We want to reason about our software. We want to know what state it is in. What we all want is the ability to say, "the core is ready, proceeding to load modules/extensions/plugins." We want to defer loading some code so that we can add extra instrumentation, checks, resiliency, and error messages to that loading process. If something misbehaves, we can do better than a stack trace and an ImportError.

Python's import system is a plugin system of sorts, but because we use it all the time, we've already used up most of the concern separation potential of import. Hence, all the creativity around plugin systems, seeking a balance between feeling native to Python, while not still successfully separating concerns.

In conclusion

So now we have achieved a complete view of the Python plugin system ecosystem, from motivation to manifestation.

By numbers alone, it may seem on the face like there are more than enough Python plugin solutions. But looking at the motivation and taxonomy above, it's clear that there are still several gaps waiting to be filled.

By taking a holistic look at the implementations and motivations, the PyCon 2017 Plugins Open Session ended with the conclusion that even Python's wide selection could use expansion.

So, until next year, go forth and continue to build! The future of well-factored code depends on it.¹

For additional reading, I recommend doing what we did after our discussion, finding and reading this post from Eli Bendersky. While it focuses more on specific implementations and less about generalized systems, Eli's post overlaps in many very reaffirming ways, much to our relief and gratification. The worked example of building ReStructured Text plugins is a perfect complement to the post above. ↩

The Many Layers of Packaging

2017-05-09T20:47:00Z

The packaging gradient, and why PyPI isn't an app store.

Update: I turned this post into a talk. The video from PyBay is here, the slides are available here. The long-cut video from BayPiggies is coming, but the "Extended Edition" slides are here.

One lesson threaded throughout Enterprise Software with Python is that deployment is not the last step of development. The mark of an experienced engineer is to work backwards from deployment, planning and designing for the reality of production environments.

You could learn this the hard way. Or you could come on a journey into what I call the packaging gradient. It's a quick and easy decision tree to figure out what you need to ship. You'll gain a trained eye, and an understanding as to why there seem to be so many conflicting opinions about how to package code.

The first lesson on our adventure is:

Implementation language does not define packaging solutions.

Packaging is all about target environment and deployment experience. Python will be used in examples, but the same decision tree applies to most general-purpose languages.

Python was designed to be cross-platform and runs in countless environments. But don't take this to mean that Python's built-in tools will carry you anywhere you want to go. I can write a mobile app in Python, does it make sense to install it on my phone with pip? As you'll see, a language's built-in tools only scratch the surface.

So, one by one, I'm going to describe some code you want to ship, followed by the simplest acceptable packaging process that provides that repeatable deployment process we crave. We save the most involved solutions for last, right before the short version. Ready? Let's go!

Prelude: The Humble Script

Everyone's first exposure to Python deployment was something so innocuous you probably wouldn't remember. You copied a script from point A to point B. Chances are, whether A and B were separate directories or computers, your days of "just use cp" didn't last long.

Because while a single file is the ideal format for copying, it doesn't work when that file has unmet dependencies at the destination.

Even simple scripts end up depending on:

Python libraries - boltons, requests, NumPy
Python, the runtime - CPython, PyPy
System libraries - glibc, zlib, libxml2
Operating system - Ubuntu, FreeBSD, Windows

So every good packaging adventure always starts with the question:

Where is your code going, and what can we depend on being there?

First, let's look at libraries. Virtually every project these days begins with library package management, a little pip install. It's worth a closer look!

The Python Module

Python library code comes in two sizes, module and package, practically corresponding to files and directories on disk. Packages can contain modules and packages, and in some cases can grow to be quite sprawling. The module, being a single file, is much easier to redistribute.

In fact, if a pure-Python module imports nothing but the standard library itself, you have the unique option of being able to distribute it by simply copying the single file into your codebase.

This type of inclusion, known as vendoring, is often glossed over, but bears many advantages. Simple is better than complex. No extra commands or formats, no build, no install. Just copy the code¹ and roll.

For examples of libraries doing this, see bottle.py, ashes, schema, and, of course, boltons, which also has an architectural statement on the topic.

The pure-Python Package

Packages are the larger unit of redistributable Python. Packages are directories of code containing an __init__.py. Provided they contain only pure-Python modules, they can also be vendored, similar to the module above. Even very popular packages like pip itself can be found with vendor, lib, and packages directories.

Because these packages nest and sprawl, vendoring can lead to codebases that feel unwieldy. While it may seem awkward to have lib directories many times larger than your application, it's more common than some less-experienced devs might expect. That said, having worked on some very large codebases, I can definitely understand why core Python developers created other options for distributing Python libraries.

For libraries that only contain Python code, whether single-file or multi-file, Python's original built-in solution still works today: sdists, or "source distributions". This early format has worked for well over a decade and is still supported by pip and the Python Package Index (PyPI)².

The Python Package

Python is a great language, and one which is made all the greater by its power to integrate.

Many libraries contain C, Cython, and other statically-compiled languages that need build tools. If we distribute such code using sdists, installation will trigger a build that will fail without the tools, will take time and resources if it succeeds, and generally involve more intermediary languages and four-letter keywords than Python devs thought should be necessary.

When you have a library that requires compilation, then it's definitely time to look into the wheel format.

Wheels are named after wheels of cheese, found in the proverbial cheese shop. Aptly named, wheels really help get development rolling. Unlike source distributions like sdists, the publisher does all the building, resulting in a system-specific binary.

The install process just decompresses and copies files into place. It's so simple that even pure-Python code gets installed faster when packaged as a wheel instead of an sdist.

Now even when you upload wheels, I still recommend uploading sdists as a fallback solution for those occasions when a wheel won't work. It's simply not possible to prebuild wheels for all configurations in all environments. If you're curious what that means, check out the design rationale behind manylinux1 wheels.

Milestone: Outgrowing our roots

Now, three approaches in, we've hit our first milestone. So far, everything has relied on built-in Python tools. pip, PyPI, the wheel and sdist formats, all of these were designed by developers, for developers, to distribute code and tools to other developers.

In other words:

PyPI is not an app store.

PyPI, pip, wheels, and the underlying setuptools machinations are all designed for libraries. Code for developer reuse.

Going back to our first example, a "script" is more accurately described as a command-line application. Command-line applications can have a Python-savvy audience, so it's not totally unreasonable to host them on PyPI and install them with pip (or pipsi). But understand that we're approaching the limit for a good production and user-facing experience.

So let's get explicit. By default, the built-in packaging tools are designed to depend on:

A working Python installation
A network connection, probably to the Internet
Pre-installed system libraries
A developer who is willing to sit and watch dependencies recursively download at install-time, and debug version conflicts, build errors, and myriad other issues.

These are fine, and expected for development environments. Professionals are paid to do it, students pay to learn it, and there are even a few oddballs who enjoy this sort of thing.

Going into our next options, notice how we have shifted gears to support applications. Remember that distributing applications is more a function of target platform than of implementation language. This is harder than library distribution because we stop depending on layers of the stack, and the developer who would be there to ensure the setup works.

Depending on pre-installed Python

For our first foray into application distribution, we're going to maintain the assumption that Python exists in the target environment. This isn't the wildest assumption, CPython 2 is available on virtually every Linux and Mac machine.

Taking Python for granted, we can turn to bundling up all of the Python libraries on which our code depends. We want a single executable file, the kind that you can double click or run by prefixing with a ./, anywhere on a Python-enabled host. The PEX format gets us exactly this.

The PEX, or Python EXecutable, is a carefully-constructed ZIP archive, with just a hint of bootstrapping. PEXs can be built for Linux, Mac, and Windows. Artifacts rely on the system Python, but unlike pip, a PEX does not install itself or otherwise affect system state. It uses mature, standard features of Python, successfully iterating on a broadly-used approach.

A lot can be done with Python and Python libraries alone. If your project follows this approach, PEX is an easy choice. See this 15-minute video for a solid introduction.

Depending on a new Python/ecosystem

Plain old vanilla Python leaving you wanting? That factory-installed system software can leave a lot to be desired. Lucky for us there's an upgrade well within grasp.

Anaconda is a Python distribution with expanded support for distributing libraries and applications. It's cross-platform, and has supported binary packages since before the wheel. Anaconda packages and ships system libraries like libxml2, as well as applications like PostgreSQL, which fall outside the purview of default Python packaging tools. That's because while Anaconda might seem like an innocent Python distribution from the outside, internally Anaconda blends in characteristics of a full-blown operating system, complete with its own package manager, conda.

If you look inside of an Anaconda installation, or at the screenshot below, you'll find something that looks a lot like a root Linux filesystem (lib, bin, include, etc), with some extra Anaconda-specific directories.

What's remarkable is that the underlying operating system can be Windows, Mac, or basically any flavor of Linux. Just like that, Anaconda unassumingly blends Python libraries and system libraries, convenience and power, development and data science. And it does it all by using features built into Python and target operating systems.

Consider that the list of cross-platform and language-agnostic package managers includes only Steam, Nix, and pkgsrc, and you can start to understand why conda is often misunderstood. Adding onto that, conda is adding features fast. For instance, conda is the first Python-centric package manager to do its dependency resolution up front (using a SAT solver), unlike pip. More recently, conda 4.3 fulfilled the wishes of many by matching apt and yum with transactional package installation. Now conda matches operating system package managers in critical technical respects, except the wide-open social components of anaconda.org make it even easier to use than, say PPAs.

In short, Anaconda makes a compelling and effective case, both as a development environment comparable to pip + virtualenv, and even as part of the equation in production server environments. Python is lucky to host to such a rare breed.

Bringing your own Python

Can you imagine deploying to an environment without Python? It's a hellish scenario, I know. Luckily, your code can still bring your own, and it's ice cold. Freezing, in fact.

When I wrote my first Python program, I naturally shared news of the accomplishment with my parents, who naturally wanted to experience this taste of The Future firsthand.

Of course all I had a .py file I wrote on Knoppix, and they were halfway around the world on a Windows 2000 machine. Luckily, this new software called cx_Freeze was just announced a couple months earlier. Unluckily, no one told me, and the better part of a decade would pass before I learned how to use it.

Fifteen years later, the process has evolved, but retained the same general shape. Dropbox, EVE Online, Civilization IV, kivy, and countless other applications and frameworks rely on freezing to ship applications, generally to personal computing devices. Interpreter, libraries, and application logic, all rolled into an independent artifact.

These days the list of open-source tools has expanded beyond cx_Freeze to include PyInstaller, osnap, bbFreeze, py2exe, py2app, pynsist, nuitka, and more. There is even a conda-native option called constructor. A partial feature matrix can be found here.

Most of these systems give you some latitude to determine exactly how independent an executable to generate. Frozen artifacts almost always ends up depending somewhat on the host operating system. See this py2exe tutorial discussion of Windows system libraries for a taste of the fun.

If you're wondering about the chilly moniker, freezers owe their name to their reliance on the "frozen module" functionality built into Python. It's sparsely documented, but basically Python code is precompiled into bytecode and frozen into the interpreter. As of Python 3.3, Python's import system was ported from C to a frozen pure-Python implementation.

Servers ride the bus

Freezing tends to be targeted more toward client software. They're great for GUIs and CLI applications run by a single user on a single machine at a time. When it comes to deploying server software bundled with its own Python, there is a very notable alternative: the Omnibus.

Omnibus builds "full-stack" installers designed to deploy applications to servers. It supports RedHat and Debian-based Linux distros, as well as Mac and Windows. A few years back, DataDog saw the light and made the switch for their Python-based agent. GitLab's on-premise solution is perhaps the largest open-source usage, and has been a joy to install and upgrade.

Unlike our multitude of freezers, Omnibus is uniquely elegant and mature. No other system has natively shipped multi-component/multi-service packages as sleekly for as long.

Bringing your own userspace

Probably the newest and fastest-growing class of solution has actually been a long time coming. You may have heard it referenced by its buzzword: containerization, sometimes crudely described as "lightweight virtualization".

Better descriptions exist, but the important part is this: Unlike other options so far, these packages establish a firm border between their dependencies and the libraries on the host system. This is a huge win for environmental independence and deployment repeatability.

In our own image

Let's illustrate with one of the simplest and most mature implementations, AppImage.

Since 2004, the aptly-named AppImage (and its predecessor klik) have been providing distro-agnostic, installation-free application distribution to Linux end users, without requiring root or touching the underlying operating system. AppImages only rely on the kernel and CPU architecture.

An AppImage is perhaps the most aptly-named solution in this whole post. It is literally an ISO9660 image containing an entrypoint executable, plus a snapshot of a filesystem comprising a userspace, full of support libraries and other dependencies. Looking inside a mounted Kdenlive image, it's easy to recognize the familiar structure of a Unix filesystem:

Dozens of headlining Linux applications ship like this now. Download the AppImage, make it executable, double-click, and voila.

If you're reading this on a Mac, you've probably had a similar experience. This is one of those rare cases where there's some consensus: Apple was one of the pioneers in image-based deployments, with DMGs and Bundles.

An image by any other name

No class of formats would be complete without a war. AppImage inspired the Flatpak format, which was adopted by RedHat/Fedora, but was of course insufficient for Canonical/Ubuntu, who were also targeting mobile, and created Snappy. A shiny update to our deb-rpm split tradition.

Both of these formats introduce more features, as well as more complexity and dependence on the operating system. Both Snaps and Flatpaks expect the host to support their runtime, which can include dbus, a systemd user session, and more. A lot of work is put into increased namespacing to isolate running applications into separate sandboxes.

I haven't actually seen these formats used for deploying server software. Flatpak might never support servers, Snappy is trying, but personally, I would really like to hear about or experiment with server-oriented AppImages.

The whale in the room

Some call the technology sphere a marketplace of ideas, and that metaphor is certainly felt in this case. Whether you've heard good things or bad, we can all agree Docker is the format sold the hardest. What else would you do when you've got $180 million of VC breathing down your neck.

Docker lets you make an application as self-contained as AppImage, but exceeds even Snapcraft and Flatpak in the assumptions it makes. Images are managed and run by yet another service with a lot of capabilities and tightly coupled components.

Docker's packaging abstraction reflects this complexity. Take for instance how Docker applications default to running as root, despite their documentation recommending against this. Default root is particularly unfriendly because namespacing is still not a reliable guard against malicious actors attacking the host system. Root inside the container is root outside the container. Always check the CVEs. The Docker security documentation also includes some good, frank discussion of what one is getting into.

Checking in with our trendline, so far we have been shipping larger larger, more-inclusive artifacts for more independent, reliable deployments. Some container systems present us with our first clear departure from this pattern. We no longer have a single executable that runs or installs our code. Technically we have a self-contained application, but we're also back to requiring an interpreter other than the OS and CPU.

It's not hard to imagine instances where the complexity of a runtime can overrun the advantages of self-containment. To quote Jessie Frazelle's blog post again, "Complexity == Bugs". This dynamic leads some to skip straight to our next option, but as AppImage simply demonstrates, this is not an impeachment of all image-based approaches.

Bringing your own kernel

Now we're really packing heavy. If having your Python code, libraries, runtime, and necessary system libraries isn't enough, you can add one more piece of machinery: the operating system kernel itself.

While this type of distribution never really caught on for consumers, there is a rich ecosystem of tools and formats for VM-based server deployment, from Vagrant to AMIs to OpenStack. The whole dang cloud.

Like our more complex container examples above, the images used to run virtual machines are not runnable executables, and require a mediating runtime, called a hypervisor. These days hypervisor machinery is very mature, and may even come standard with the operating system, as is the case with Windows and Mac. The images themselves come in a few formats, all of which are mature and dependable, if large. Size and build time may be the only deterrent for smaller projects prioritizing development time. Thanks to years of kernel and processor advancement, virtualization is not as slow as many developers would assume. If you can get your software shipped faster on images, then I say go for it.

Larger organizations save a lot from even small reductions to deployment and runtime overhead, but have to balance that against half a dozen other concerns worthy of a much longer discussion elsewhere.

Bringing your own hardware

In a software-driven Internet obsessed with lighter and lighter weight solutions, it can be easy to forget that a lot of software is literally packaged.

If your application calls for it, you can absolutely slap it on a rackable server, Raspberry Pi, or even a micropython and physically ship it. It may seem absurd at first, but hardware is the most sensible option for countless cases. And not limited to just consumer and IoT use cases, either. Especially where infrastructure and security are concerned, hardware is made to fit software like a glove, and can minimize exposure for all parties.

But what about...

Before concluding, there are some usual suspects that may be conspicuously absent, depending on how long you've been packaging code.

OS packages

Where do OS packages like deb and RPM fit into all of this? They can fit anywhere, really. If you are very sure what operating system(s) you're targeting, these packaging systems can be powerful tools for distributing and installing code. There are reasons beyond popularity that almost all production container and VM workflows rely on OS package managers. They are mature, robust, and capable of doing dependency resolution, transactional installs, and custom uninstall logic. Even systems as powerful as Omnibus target OS packages.

In ESP's packaging segment, I touch on how we leveraged RPMs as a delivery mechanism for Python services in PayPal's production RHEL environment. One detail, that would have been minor and confusing in that context, but should make sense to readers now, is that PayPal didn't use the vanilla operating system setup. Instead, all machines used a separate rpmdb and install path for PayPal-specific packages, maintaining a clear divide between application and base system.

virtualenv

Where do virtualenvs fit into all of this? Virtualenvs are indispensible for many Python development workflows, but I discourage direct use of virtualenvs for deployment. Virtualenvs can be a useful packaging primitive, but they need additional machinery to become a complete solution. The dh-virtualenv package demonstrates this well for deb packaging, but you can also make a virtualenv in an RPM post-install step, or by virtue of using an installer like osnap. The key is that the artifact and its install process should be self-contained, minimizing the risk of partial installs.

This isn't virtualenv-specific, but lest it go unsaid, do not pip-install things, especially from the Internet, during production deploys. Scroll up and read about PEX.

Security

The further down the gradient you come, the harder it gets to update components of your package. Everything is more tightly bound together. This doesn't necessarily mean that it's harder to update in general, but it is still a consideration, when for years the approach has been to have system administrators and other technicians handle certain kinds of infrastructure updates.

For example, if a kernel security issue emerges, and you're deploying containers, the host system's kernel can be updated without requiring a new build on behalf of the application. If you deploy VM images, you'll need a new build. Whether or not this dynamic makes one option more secure is still a bit of an old debate, going back to the still-unsettled matter of static versus dynamic linking.

Closing

Packaging in Python has a bit of a reputation for being a bumpy ride. This is mostly a confused side effect of Python's versatility. Once you understand the natural boundaries between each packaging solution, you begin to realize that the varied landscape is a small price Python programmers pay for using the most balanced, flexible language available.

A summary of our lessons along the way:

Language does not define packaging, environment does. Python is general-purpose, PyPI is not.
Application packaging must not be confused with library packaging. Python is for both, but pip is for libraries.
Self-contained artifacts are the key to repeatable deploys.
Containment is a spectrum, from executable to installer to userspace image to virtual machine image to hardware. "Containers" are not just one thing, let alone the only option.

Now, with map in hand, you can safely navigate the rich terrain. The Python packaging landscape is converging, but don't let that narrow your focus. Every year seems to open new frontiers, challenging existing practices for shipping Python.

Don't forget to include respective free software licenses, where applicable. ↩
Despite being called the Python Package Index, PyPI does not index packages. PyPI indexes distributions, which can contain one or more packages. For instance, pip installing Pillow allows you to import PIL. Pillow is the distribution, PIL is the package. The Pillow-PIL example also demonstrates how the distribution-package separation enables multiple implementations of the same API. Pillow is a fork of the original PIL package. Still, as most distributions only provide one package, please name your distribution after the package for consistency's sake. ↩

Developer variants

2016-08-09T10:00:00Z

Software development takes all kinds. I'm not talking about appearances or job titles. I'm talking about motivations and fulfillment.

In my years of writing code and leading projects, I've come to learn a bit about how my teammates, and I, experience success, through a few manifest archetypes.

The Developer-Mathematician

Always a source of conversation, the Developer-Mathematician, seeks truth, pure and provable. They don't want to create software. They want to unearth timeless, universal absolutes that happen to be in the neighborhood of computers.

Catch them crafting functional code, writing property-based tests, or exhaustively searching their bookmarks for that one paper on arXiv.

To be honest, purity and formalism can chafe when building most software. Proofs are still more suited to dissertations than development. Still, it's good to strike a healthy balance between research and development. Make time to try new testing strategies, start a weekly paper club, and keep those fundamentals sharp.

The Developer-Architect

Less formal than the mathematician, but not always more practical, the Developer-Architect is brimming with potential. They want to create something original, important, and particularly elegant. They want to create something that outlasts them, something worthy of use, maintenance, and study. The creation need not be immortal or universal; the more of their mark that is left on it the better.

Find them making high-concept pitches in response to clear gaps in the open-source ecosystem, or discussing best practices that are suspiciously similar to their own practices. If your Developer-Architect is low on ideas or recently saw one of their ideas superseded or implemented without them, they may become despondent.

Software designers derive a lot of pleasure from the design process, but need to be reminded that architecture is far from the hardest part. To avoid turmoil and despondency, Developer-Architects must code their own implementations and design only a few steps ahead. Creative code can be very good code, and may well be worth the risk and wait.

The Developer-Engineer

Least formal, but no less professional, the Developer Engineer is the workhorse of the software industry. Engineers build for the sake of building. Recognize them by their willingness to experiment with code, and their lack of attachment to code. If it doesn't work, the engineer has confidence: Toss it, we can build it better again.

For motivation, the engineer needs clear requirements and a modicum of appreciation for a spec well-met. For fulfillment, the build itself often suffices, so avoid process and interruptions.

Proofs and designs aside, I still believe when we channel the Developer Engineer, we channel our best selves. A sense of confident understanding of the problem, married with unbounded pragmatism, leading to working, shippable code. It will have bugs, and it may not be abstracted quite right for future extensibility, but it will work.

A Winning Combination

We all go through phases, play different roles, and work with all sorts. Embracing the mathematician, architect, and engineer, as well as others, from tinkerers to hustlers, has taught me more than I could have learned by my undifferentiated self.

The key is recognizing your current motivations and finding alignment of these angles within a company, within a team, and within oneself.

Announcing CalVer

2016-06-22T17:30:00Z

It's about time.

Technologists expect things to get better with time. Your current laptop has more RAM than the last, your current car is safer than its predecessor, and the latest version of your code is certainly the best ever.

What if the same be said of versioning systems?

Software versioning systems also get better with time. That's why today I'm pleased to announce CalVer, a calendar versioning convention based on project release dates, formally hosted on calver.org.

Calendar versioning represents a powerful alternative to Semantic Versioning (SemVer). CalVer combines with or even replaces SemVer versioning systems, based on the needs of the project.

Features

The calver.org site speaks for itself, but there you'll find:

Terms and definitions
Case studies, including Ubuntu, Twisted, Teradata, and more
And a short guide on when to use CalVer for your future projects

Case studies feature badges like this one, for Ubuntu's versioning scheme:

You'll also find a project list, always seeking new additions.

Rationale

Many projects have designed their version schemes to better match the needs of their developers and customers. CalVer formalizes those practices. calver.org began as a resource to help maintainers communicate the design choices in their versioning scheme.

CalVer has grown to showcase prominent uses and provide a way for more projects to adopt calendar versioning in their projects. It even hosts a community-curated list of projects using calendar versioning.

Even more background on the project can be found on the calver.org About page, as well as my previous versioning essay, Designing a version.

Compared to SemVer

Some comparisons are inevitable. SemVer, hosted at semver.org, is a big name in software versioning conventions. CalVer combines well with incremental-number schemes, so it's not strictly a competition. That said, here is how CalVer outshines SemVer.

🕐 CalVer integrates objective, intuitive calendar dates.
⊠ SemVer subjectively increments numbers.

🕑 CalVer encompasses real-world usage through a formal vocabulary.
⊠ SemVer imitates the form of a specification, albeit a confrontational one. Unlike real specifications, SemVer lacks objective verifiability, exemplars, or reference implementations.

🕒 CalVer makes maintenance easier through powerful, objective semantics. Look at a library's version number, immediately know how recent your copy. Compare across libraries, checking that dependencies are in sync. Deprecate versions based on time.
⊠ SemVer has Tom Preston-Werner's semantics.

🕓 CalVer's use of release dates allows for automatable, immutable versions on which everyone can agree.
⊠ SemVer introduces one more place a bug can enter a projects. Versions only go up, and a release which violates SemVer guidelines cannot be undone. That pressure means more projects perpetually stuck in 0.x.

The list goes on, but the message is clear. There is an alternative to SemVer, and it's about time!

Next steps

Have a look at the Users list and help add any projects I may have missed. It's a big ecosystem out there, and the initial list reflects my own Linux and Python tendencies.

For current maintainers using calendar versioning, next time you get a raised eyebrow, just let them know: It's CalVer. Or save yourself a step and add one of the badges, linking to calver.org.

For developers of new libraries, CalVer is here to stay, and calver.org will be there next time you're designing your versioning scheme. It's a big ecosystem out there, and once you try CalVer, I think you'll agree. Software versioning get better with time.

Running from software

2016-05-27T11:11:00Z

So while PyCon 2016 starts in less than 48 hours, some kind of anticipation compelled me to polish off the last of the talks from last year. For some reason I went for a keynote. I'm not typically a keynote attendee, and this time I'd missed something big.¹

Jacob Kaplan-Moss, the herald of Django, really laid something out. I'll give you the short version, but here's a video in case you want a look:

To summarize, Jacob sets out to explain why mediocrity is acceptable. Bell curves rule everything around us. He holds up his record as a middling ultramarathon runner as proof. He surmises that lack of passion for work is leading people to feel untalented. This, combined with "brilliant asshole" programmers, is shaming people out of the industry. He wraps up with a message of inclusivity, especially toward women. Now, you can probably make sense of any other details with the slides.

Above all, Jacob and I are in complete agreement with his opening and closing. If you consider yourself an average programmer, that is fine and probably better than the alternatives. Also, as a field, software must continue reaching out to and integrating more underrepresented groups, especially women.

That said, I'm not sure how one could have put more missteps between those two points.²

The 10x Programmer

If Jacob makes one thing clear from the keynote, it's that years of being called a 10x programmer has made him very uncomfortable. He rejects the concept, as many have. Now I, too, have at various points been called a rockstar, ninja, and 10xer, and even though I also don't identify with those labels, I will tell you that the 10x programmer is very real.³

Every 10x programmer I know spends most days as a 1x something else. Most 10x code is the result of observing and accumulating 10x more domain knowledge, then being in the right place at the right time. You do what ten developers off the street could never. I've been there, and I have the commits to prove it. And when other aspects of my life take priority, I'm an average programmer, focusing on my job and its share of 1x work.

10x programming is a matter of insight and inspiration, confidence and autonomy. This is a circumstance so unique that it creates an obligation to teach software to the world. You never know when the right 1x programmer is going to be in the right place to transform their surroundings with a 10x moment. Many of the most creative people I know understand very little about programming, and one can't help but wonder what programming skills or insight might bring to their process.

The great thing about Python is that you can teach so much programming with so little overhead. You give those highly creative people even a taste of programming and it opens up vast opportunities. Even just the shared vocabulary is a huge boost to cross-pollination of ideas between disciplines.

Look at Python use among biologists, neuroscientists, and other academics and analysts. Their amazing results speak volumes. Yet by strict accounts their engineering skill wilts next to experienced Python systems engineers working at YouTube, PayPal, Dropbox, Continuum Analytics, etc.

It's inexcusable to put such a diverse group on this single bell curve when their goals and disciplines are so different. Our language is the same and our cultures are mutually beneficial. Seeing people measured along this single dimension keeps me up at night.

Putting it all in terms of employment is harmful. Maximizing employee utilization only creates more 1x programming. Software is more than the industry of churning out code. A programmer is more than someone who is paid to write software. A person is more than their profession.

The Privilege

It's said that the most sure sign of privilege is ignorance. Jacob drives this all the way home, but not for lack of trying

From the beginning of the talk, he considers the immediate situation. He disclaims most of his reputation, describes his origins as unremarkable, and points out that his biggest contributions weren't actually his. Later on in the talk, while showcasing the face of the privileged programmer, the 10x archetype, the person most likely to be able to ride on their identity, he shares a chuckle at his own resemblance.

Moving into Jacob's running-programming analogy, the anecdote got off to a false start, but just kept going. Nobody stopped him to point out that by virtue of simply being an ultra-runner, he is the top tier. If you're in the 68th percentile of ultrarunners, then you're in the top 1% of people who run, period. Even finishing a normal marathon faster than the median time demonstrates talent and tremendous physical gifts.

Jacob trimmed the y-axis, measured himself among the top tier, and found himself only slightly better than mediocre. The sort of guilt-inducing behavior that he claims leads people to leave the field, unfolding right on stage.

The Corporatism

Throughout the talk, Jacob cites some statistics. The one that stuck with me was about an impending employment deficit. The U.S. government projects 1.5 million unfilled programming jobs in the year 2020. This becomes a central motivation for Jacob encouraging people to go into software⁴. Programming is immediately linked to coding for money.

Jacob says software is a skill, like any other. Programming is like running marathons. Individuals are responsible for their own training. But Jacob bears a message of hope: bosses will pay you to run, even if you're not the fastest.

Too many managers are like Jacob, subtly redirecting the creative potential of software into commodity labor. "We" need as many people as possible to learn and teach programming because some a small portion of society has decided to gamble money on software eating everything in a very particular way.

On the contrary, people need exposure to programming for its fundamental concepts. Software offers new ways of decomposing problems and creating solutions, new approaches that are necessary to understand an increasingly fast-paced and connected world. That is totally irrespective of employment. Software design is a new way of thinking, for all people, employed as programmers or not.

In short

Jacob is a much better runner than he gives himself credit for, but programming is not running.

Software is much more than an industry. You don't need a programming job to be a good programmmer.

This brings me back to reiterate the central thought we share: One doesn't need to compare favorably to other programmers in order to make a difference with software. So, we must accept and support programmers of all walks and skill levels.

Suffice to say, I'm already subscribed to PyCon 2016 ↩
Dear Jacob, if you are reading this, I just wanted to say no harsh feelings. It was a moving talk and I'm sure that most people got the good messages that bookended the talk. I hope you don't mind the criticism and still find it as interesting as you mentioned on stage. Hope it helps with future keynotes, and I'll be right here if you have any followups. ↩
This also came up in Episode #54 of Talk Python to Me, while discussing my course, Enterprise Software with Python. ↩
"The US Bureau of Labor Statistics estimates that by 2020 there will be a 1.5 million programming job gap, which means there will be that many jobs unfilled. That's in five years. The EU has published similar numbers, 1.2 million in 2018—three years. That means we need to be doing something to get more people into our industry." ↩

Managing Python Ecosystems

2016-05-24T17:00:00Z

You know that old quote:

The wider the net you cast, the wider the variety you catch.

Was it a wise old fisherman? Or a dogged Python programmer? Either way, words don't come much truer than those.

Few, if any, programming languages have embodied the description "general-purpose" as wholly as Python. And with the wide net of that applicability comes a wide variety in use -- and environments.

Library and framework developers rarely get to control how their code is used, and thus have to think about how their code fits into the whole ecosystem. From writing hybrid code for Python 2 and 3 to inserting shims for Pythons without threading support, there's no rest for the rigorous. Until now.

Announcing `ecoutils`

Ecosystems differ. Widely. Academic Python tends to be more Windows-heavy, corporate Python will probably forever be entrenched in Python 2, and one can never predict the arrival of that oddball user with the super old version of Python on Cygwin. But these are generalities and we can do better.

Enter ecoutils. ecoutils is a pure-Python module that, using nothing but builtins, generates a semantic, Python-centric profile of the environment that's running it. This includes:

Host operating system: Windows, OS X, Ubuntu, Debian, CentOS, RHEL, etc.
Language version: 2.5, 2.6, 2.7, ..., 3.4, 3.5, ..., etc.
Executable runtime: CPython, PyPy, Jython, etc., (plus build date and compiler)
Features: 64-bit, IPv6, Unicode character support (UCS-2/UCS-4)
Built-in library support: OpenSSL, threading, SQLite, zlib, and more
User environment: umask, ulimit, working directory
Machine info: CPU count, hostname, filesystem encoding

Now, instead of crossing platform support bridges when users bring them to you, you can be proactive. Now, instead of guessing how developers are using the code, you can design for their needs and watch those needs change.

ecoutils only gets more valuable when code goes to production. If you manage your own machines, you know the risk of version drift and missed boxes only goes up with machine number and time. If you don't manage your machines, it's just a matter of time until someone is being trained on your boxes.

So what does a profile look like?

Generating a profile

Profiles are generated by ecoutils.get_profile().

When run as a module, ecoutils calls get_profile() and prints a JSON-formatted profile. On my fully-updated Ubuntu 14.04LTS machine, python -m boltons.ecoutils yields:

{
  "_eco_version": "1.0.0",
  "cpu_count": 4,
  "cwd": "/home/mahmoud/projects/boltons",
  "fs_encoding": "UTF-8",
  "guid": "6b139e7bbf5ad4ed8d4063bf6235b4d2",
  "hostfqdn": "mahmoud-host",
  "hostname": "mahmoud-host",
  "linux_dist_name": "Ubuntu",
  "linux_dist_version": "14.04",
  "python": {
    "argv": "boltons/ecoutils.py",
    "bin": "/usr/bin/python",
    "build_date": "Jun 22 2015 17:58:13",
    "compiler": "GCC 4.8.2",
    "features": {
      "64bit": true,
      "expat": "expat_2.1.0",
      "ipv6": true,
      "openssl": "OpenSSL 1.0.1f 6 Jan 2014",
      "readline": true,
      "sqlite": "3.8.2",
      "threading": true,
      "tkinter": "8.6",
      "unicode_wide": true,
      "zlib": "1.2.8"
    },
    "version": "2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2]",
    "version_info": [2, 7, 6, "final", 0]
  },
  "time_utc": "2016-05-24 07:59:40.473140",
  "time_utc_offset": -8.0,
  "ulimit_hard": 4096,
  "ulimit_soft": 1024,
  "umask": "002",
  "uname": {
    "machine": "x86_64",
    "node": "mahmoud-host",
    "processor": "x86_64",
    "release": "3.13.0-85-generic",
    "system": "Linux",
    "version": "#129-Ubuntu SMP Thu Mar 17 20:50:15 UTC 2016"
  },
  "username": "mahmoud"
}

Weighing in at just over 1KB, it's not too daunting! ecoutils is part of the boltons package, so pip install boltons and see how yours compares.

By virtue of being in boltons, the ecoutils module is also fully standalone, and can be used without the rest of the boltons package. ecoutils has been tested with Python 2.6, 2.7, 3.4, 3.5, and PyPy on Ubuntu, Debian, RHEL, OS X, FreeBSD, and Windows. File an issue if something seems to be broken. Compatibility is the goal.

Transmission and collection

Now, ecoutils is really just part of the solution. Sure you can write out a quick profile it at the top of every log file, and you won't regret it. However, real ecosystem management means running a sort of Python analytics shop.

For those familiar with browsing the Internet, your browser is a virtual machine that has likely been participating in a similar arrangement all day today. Like Google Analytics or Piwik, the setup involves collecting relevant data, and then sending it to a central server for storage and querying.

Collection is handled by ecoutils. As far as transmission is concerned, in development environments, we have a dead-simple, side-effect-minimizing, single-file HTTP client that sends ecoutils profiles to a central analytics server on application startup.

In production environments, our framework serves this information for queries on a special port, through SuPPort's MetaService, through clastic's MetaApplication, where this all started. Here's an example of it running in Wikipedia Hashtags Search, on a managed Wikimedia environment, over which I have minimal control, and need maximum information.¹

Push or pull, all the data is stored in a simple SQL (or JSONL) format, as demonstrated by espymetrics, the example project for my Enterprise Software with Python course. Nothing more enterprise than having literally dozens of environments by design, and even more than that by debt.

One last note, data management is all about audience and context. If you're an administrator in a professional setting, the data above is great. But there are understandably some cases where you might want something less identifiable. get_profile has a scrub flag that handles that. See the docs for details.

Success stories

Originally designed for easier remote administration across multiple environments, a little bit of info has had far-reaching impacts. For a few examples from my work at PayPal, this approach enabled us to:

Deprecate and remove production Python 2.6 support from our framework, simplifying our build matrix without customer impact.
Actively engage new users attempting to use our framework with unsupported Pythons or OSes.
Improve utilization through designing for observed CPU counts.

In practice, ecoutils combines well with psutil data to go even further in utilization.

Building for variation

Some of you probably came here expecting to read yet another great post about virtualenv, tox, and maybe even conda envs. I'm glad you've already heard of them, because they're a big part of the story. If you haven't yet explored these tools, check them out, because they are invaluable for cross-version Python testing and packaging.

Also, if you're working on an open-source library, I can vouch for Travis CI (Linux) and Appveyor (Windows) as very valuable providers for cross-platform testing. I use both of them on boltons, and it makes it easier, not harder, for contributors to submit pull requests with confidence. Most outfits can't afford to have a team member leading support for each platform, like we do at PayPal.

Conclusion

Python is more than just an expressive, succinct programming language. In a diverse world, Python is a tremendous force, made so by its wide deployment, cross-platform support, and external library integrations. Python gives you SQLite, JSON, SSL, Unicode, and much more, but with many necessary strings attached to Python version, build, or environment. ecoutils offers an experienced look at the real features that affect the value of Python components and teams.

Don't leave ecosystems and their constituents to chance, whim, or fad. Collect the data that makes your ecosystem unique, and make measured decisions based on the realest demand: actual usage.

When that server seems slow, remember to donate to Wikipedia. And maybe volunteer, because money alone does not make servers run fast. ↩

Enterprise Software with Python

2016-03-22T11:04:00Z

When I first published 10 Myths of Enterprise Python on the PayPal Engineering blog, there were a lot of reactions. Some I expected:

Surprise at Python in the enterprise space.
Relief at more attestation of Python's use in the enterprise.
And, as with all the best, a few flamewars.

But there was one I missed: new developers interested in professional software development.

Really I should have seen it coming. For the better part of a decade, Python has provided me the best vocabulary for answering questions from motivated individuals looking for programming productivity. It's only logical that once they got the basics down, they'd want to take it to the next level.

With this end in mind, I'm pleased to announce Enterprise Software with Python (ESP), a bridging class from beginner to pro¹, brought to you by O'Reilly Media and yours truly.

It's got something for everyone, but really it's designed with three groups in mind:

Recently-graduated and self-taught developers, looking for a holistic introduction to enterprise software.
Experienced developers at large organizations, looking for a relatable orientation to Python industry standards.
Technical team leaders with priorities, looking to quickly get groups on the same page of vocabulary, expectations, and practice.²

As the title suggests, ESP is more than a Python class. While the perspective is Pythonic and there are several examples in Python, this is a full software development course. You will find a serious effort has been made to set expectations and develop the soft skills large organizations demand. You need architectural skills to form a technical opinion, engineering skills to implement and maintain it, and managerial skills to defend it all along the way. I can't resist a good table of contents, so this is how the course is factored to address all of these:

Introductions and definitions - A bit about me, a bunch about the course.
Overview
Prerequisites and viewing guide
Definitions and foundations - Know your domain, know your platform.
What is Enterprise Software? - 9 Hallmarks of the Enterprise
What is Python? 3 Perspectives for the Organization
What is Python Not? 4 Common Misconceptions
When to Use Python? Motivations and Applications
Architecture and design - Do your research, present your findings.
Designing Architectures: Professional Planning
Gathering Requirements: Understanding the 6 Aspects of Software
Researching Environments: From Production to Development
Choosing Dependencies: Evaluating Building Blocks
Getting Assistance: Finding Help in the Software World
Presenting Designs: Navigating the Organizational and Interpersonal
Engineering practices - Execution and delivery with minimal regret.
Development Environments: Editors and Dev Tools
Source Control, Issue Tracking, and Continuous Integration
Workflow: Starting a Python Project
Design Patterns: Idioms for Python Projects
Debugging: Solving Problems in Python projects
Security: Software Risk Management Fundamentals
Code Review: Python Antipatterns and Collaboration
Testing: Practical Python Quality Engineering
Logging and Monitoring: Introspectable Python Projects
Profiling and Performance: Strategies for High-Speed Python
Documentation: Preserving the Legacy
Packaging and Deployment: Going Live
Career development and further study - A good end offers a dozen new beginnings.
Project Ideas: Building Experience
Technology Evangelism: Building a Community
Other Resources: Building Skills
Closing

Yes, it is a lot. I never pass on an opportunity to give a comprehensive treatment, but I'll save the whole motivation and process essay for later. For now, keep in mind that most segments are under 20 minutes, and the longest, Profiling and Performance, is only 45 minutes — shorter than most orgs' tech talks. It's all compact and practical, right down to the example repo.

Actual footage from the intro. Not a prerelease render.

The first three parts are free, and will give you a good sense of the format, tone, and content. I kept it pretty light and approachable, complete with dozens of illustrations. Purchasers can stream the rest, and download DRM-free copies whenever you want (my personal favorite). If you have any questions or concerns, don't hesitate to reach out to me, personally, or O'Reilly Media.

I hope you'll take a look! It's already making waves at PayPal, and chances are there's someone you know who could use it, too.

This link has a 50% off coupon code, applied at checkout. Check if your organization has Safari, first. If not, use this coupon-less link and expense it! :) Safari users, try the SBO site. If you're not sure if you have Safari access, contact your technology education and training department. ↩
This target audience is me, but I know there are others out there. Send me your tiring, huddled masses yearning to learn Python. Seriously though, I can't fully quantify how much time it saves me to send a new Python initiate to a video, then have them come back with the foundations necessary to have a productive conversation. ↩

Designing a version

2016-02-23T18:27:00Z

In modern software development, a project isn't a project without a proper versioning scheme.

Weak version management neglects clients like lack of source control neglects collaborators. Dependency management and migration rely on versions. Beyond the technical, a project's version bears a huge impact on the perception of the project. It informs adoption and entices users to upgrade. The version is attached to the name of the project — appearing closer and more often than the names of the maintainers. Versions are how a project builds a legacy.

So why do projects leave versioning to afterthought? What do clients expect and what do projects need?

Followup: This post culminated in the announcing CalVer and launching calver.org. This page provides a thorough background to the CalVer best practices.

Contents

Semantic Versioning
Collective Expectations
Case Study: Chrome vs. Firefox
Calendar Versioning
Summary

Semantic Versioning

Currently, the go-to versioning system for open-source software is referred to as Semantic Versioning, or SemVer.

Take a quick look at the 40 most recent updates on the Python Package Index (PyPI). My glance showed all but six packages had the comfortable three-part versioning scheme, major.minor.micro. Among those packages the highest minor version was 108. The highest micro version was all the way up to 595.

So, if SemVer is so popular, it must be easy, right? Follow a couple straightforward steps. Pick a number, add one to it. With arithmetic that simple, what could go wrong?

SemVer and code breakage

Everyone knows it's more exciting to announce 2.0 than 1.7.0, even if there's more user demand for the latter than the former. This is especially true with SemVer, because a SemVer major version change implies breaking the API.

As we will see, there are consequences to this. People judge value based on version number. SemVer supports this opaque apples-and-oranges comparison, punishing libraries that get it right on the first try, and encouraging libraries to break APIs to appear more mature and get that coveted 2.0.

SemVer and release blockage

More damaging than the fatuous 2.0 is the epidemic of Zeno's 1.0.

Witness the version, racing to numeric motionlessness. (Image based on Martin Grandjean's.)

To quote the second answer in SemVer's own FAQ:

If your software is being used in production, it should probably already be 1.0.0. If you have a stable API on which users have come to depend, you should be 1.0.0. If you’re worrying a lot about backwards compatibility, you should probably already be 1.0.0.

On this count, SemVer might be found not guilty.² If so, it's the SemVer users that didn't get the memo — myself included. Maybe if it had been in the spec itself.

The problem is the heavy emphasis on "public API" breakage. Conservative library authors end up indefinitely preferring the semantic power of 0.x: The ability to break APIs. Whether the cause is conservatism, humility, or misunderstanding, the effect is misrepresenting the release state of many major libraries.

A more practical scheme might help represent accurate versions for mature, production libraries like Cython (0.23) and SciPy (0.17), both of which have books and nearly a decade of release history still on PyPI.

SemVer and certifiability

Appealing to engineering aesthetics, SemVer is presented as a "specification". But, unlike the vast majority of successful RFCs, there is no validation or certification that can determine whether a project has a correct implementation. Yes, if a project API changes, but the major version is not incremented, the SemVer specification has been violated. But there's no way to test that generally, and no one does it specifically.

SemVer is a detailed suggestion. Software breaks as quickly as SemVer's promise. The remediations do not happen. Better to embrace the realities of versioning, rather than argue over the MUSTs and MUST NOTs of an unenforceable specification.

Collective Expectations

Let's take a brief moment to reconsider the humble version.

We encounter far more software than we write. Few, if any, expect compliance with all the suggestions in SemVer. So what do we expect from our versions?

There are three main expectations driving modern software versioning:

#1 Versions go up

The later the release, the greater the version. Sofware should not change without a version change, and the version must go up, and never come down.

#2 Versions correlate to software quality

A project name communicates an ideal. The project version communicates current progress toward that ideal. Vision pursued by version: The greater the version, the greater the software.

#3 Versions are numeric, except when they're not

Here's where things get hairy. Numeric versions are the default, but non-numeric versions and version components abound.

Version vernacular is now thoroughly mainstream: "alpha", "beta", "dev", "nightly", "stable", and so on. There are also named project versions, like those used in Linux distributions, such as Debian's "jessie", Ubuntu's "trusty", and Windows' "longhorn". Non-numeric versions are often hijacked for branding purposes. Numerical versions' technical utility is much more important to preserve.

Case Study: Chrome vs. Firefox

We take our version expectations for granted, but a convention this fundamental has profound effects at scale. As mentioned above, higher versions are expected to be better, especially within a project. But there is at least one case where this impact very publically spilled out across projects: The Chrome-Firefox Version Wars.

When Google Chrome entered the browser race, it brought with it a fast feature release schedule and a versioning system to match. This versioning system had Chrome see a dozen major releases while Firefox was still 3.x. Firefox looked like it was being left in the dust, despite the fact that Chrome was less mature and, as anyone who used it at the time can attest, Chrome 4 wasn't half the browser Firefox 4 ended up being.

After a couple years of this onslaught, Firefox switched its versioning system to match. Now, despite browsing for hours a day, few users or even developers could tell you off the top of their heads what version of Firefox/Chrome they use.³

SemVer ignored this huge precedent, harshly judging fast-moving projects. Let's call that our last straw and look at an alternative.

Calendar Versioning

If you're an earnest engineer with honest intents of creating, releasing, and maintaining a project, then calendar versioning may be for you. CalVer fulfills all of the versioning expectations, so what advantages does it bring?

CalVer leverages natural understanding

People are calendar-oriented. Practically, it's just easier to remember that a library was causing a live issue back in 2013 than it is to remember that up until version 1.6.18 that library had a lot of bugs.

Furthermore, in long-term development, releases pile up and increasingly large major versions blur together. Browser versions have been rendered meaningless. But the calendar is one construct where numbers increase and cycle regularly. Leveraging that natural understanding anchors otherwise arbitrary versions.

CalVer has better semantics

Ironically yes.

"Semantic" Versioning is all relative. One developer's 1.0.0 is another's 0.0.1alpha. As authors, we try to ignore this and write others off as wrong. But as clients, we make snap judgments, and SemVer lets us forget and pretend. Calendar versioning is absolute and neutral, with practical advantages to boot.

As application developers adding functionality, evaluating a new library means ascertaining maintenance status, usually by looking at the most recent release date. CalVer puts us in the ballpark right away. As maintainers depending on many libraries, calendar versioning allows us to look at the dependency list and quickly ascertain which libraries are good candidates for updating. CalVer even lets us take that a step further, with date-based deprecation.

Many might not realize it, but the oh-so ubiquitous Ubuntu is in fact calendar versioned. For example, version 15.04 came out in April, 2015. It gets better when you remember Long-Term Support. Ubuntu's LTS support lasts for five years. So, 14 + 5: Ubuntu 14.04's end of life will be in 2019. You don't have to look anything up. It's all right there in the CalVer semantics.⁴

CalVer protects projects

If you care about the future of the project, then guard it against one of the worst fates: the fatuous 2.0. Give your project a future. Guard against the learned expectation of 2.0 or death.

A 1.x always carries one advantage over a 2.0: the code is deployed and working. Avoid contempt for past decisions and current users. In engineering, utility is half of correctness.

SemVer is set up so that every major release implies a minimum threshold of change. If the project is founded on and aiming for correctness, fewer and fewer changes are required. Donald Knuth embraced this in the extreme by having TeX's version approach π asymptotically. Suffice to say with CalVer, you are safe to add as much or as little functionality as needed.

Too often projects become a victim of versioning. New projects end up masquerading as new versions. D3 could have been Protovis 2.0, but instead, a successor was created. Both projects coexisted and we are all the better for it. Same with characteristic and attrs. Successors and CalVer protect projects and do justice by clients and code.

Summary

Consider adding a calendar component to your next library's versioning schemes. As for my opinion, I've joined other maintainers in doing so for boltons and ashes. I've found it makes a lot of sense for libraries, and a little less sense for protocols and services.⁵

Either way, think about project versions. The version is part of your project's face and your clients' integration. After spending days, weeks, and months on a project, it's worthwhile to spend a few minutes or hours designing a versioning system tailored to the needs of project users and maintainers.

If you're into enterprise software considerations like these, subscribe or follow me on Twitter for some details about my upcoming O'Reilly project.

Astute readers will note that it's Semantic Versioning 2.0.0. "Oh, cute, Tom used his own scheme for the document." But did you wonder what public API changed to trigger that major version bump? SemVer's public API has been semver.org since before 1.0. How about those semantics? ↩
I've actually been saying something similar, but more practical, for a long time:

If both you (or your team) and a stranger (someone not directly advised) are both using a library in a production environment, the time for a major version has come.

If it's just you and yours, that's understandable. Many great scientists took great risks with themselves for the sake of progress. If it's just a stranger going against your explicit advice, then there's no accounting for such wildcards. But, if both of groups are using something in production, then it's time to face the facts. Tie up the loosest of ends and give it a major version. ↩
Here are some more resources for those interested in the Firefox release switch up:
At the very least this should illustrate that versions matter. They're part of your project's identity. Design them to help your user. ↩
To illustrate the prevalence, there are actually many other examples of calendar versioning we take for granted. Off the top of my head I could think of Twisted, Windows 95/98/2000, and probably most ubiquitous: every mainstream car in circulation. Email me with more examples and I'll compile them somewhere. ↩
To illustrate, if I could have it my way, we'd have OpenSSL 16.x.x. That way I can easily complain if I find someone using 10.x.x in production. That said, TLS/1.3 seems better than TLS/16.0.

My current thought is that protocols live outside of time, because I believe it's possible to complete a protocol, but an implementation is never done. ↩

Getting a Python job

2016-01-25T21:02:00Z

Every day, Python is the primary programming language for tens if not hundreds of thousands of professional engineers, analysts, and researchers, including yours truly. Given Python's "language of choice" status, what can you do to join those lucky ranks?

It's a good question, and one I get often. Recently I was asked more publically than usual. Michael Kennedy, host of the Talk Python to Me podcast, asked me five questions on behalf of people early in their Python/programming careers:

What kind of Python devs do you work with and interview?
What is the most important piece of experience that you look for in a candidate?
If someone is applying for their first job with you, what can they present to show they have the right skillset/education?
Open-source contributions
Side projects
Mobile phone apps
Websites
Code competitions
If you are presented with two candidates, one with a solid CS degree, and the other with 1-2 years of experience, which would you value more?
Why did you hire the last person you hired?

Here are my answers, the enterprise hiring perspective, as transcribed from my parts of the panel discussion.

Contents

Intro
My type of hiring
The most important experience
Side experience
Formal education
Last hire
Takeaways

Intro

Hi my name is Mahmoud Hashemi. I'm lead developer of Python Infrastructure at PayPal, and I'm also the presenter of Enterprise Software with Python, coming soon from O'Reilly. Dedicated listeners may recognize my voice from episode #4 of Talk Python to Me, and it's great to be back on the show.

My type of hiring

What kind of Python devs do you work with and interview?

I work with Python infrastructure engineers. Software infrastructure is the foundation of all sorts of software development, from web to backend to batch to automation and tools. To do it well you have to have personal experience developing in two or more of those categories. For the past year or so, my team has been adjunct to the PayPal application security team, and that's who I'm hiring for right now. So a little plug, if you have at least five years of industry experience and want to get into some ultrahigh performance Python security work, shoot me an email at mahmoud@paypal.com.

All that said, one of the services the Python infrastructure team also performs is to do phone and in-person interviews for PayPal teams looking to expand their Python talent through hiring.

The most important experience

What is the most important piece of experience that you look for in a candidate?

The most important fundamental skills I look for are closely related to experience: environmental fluidity and personal learning abilty.

Wait, not Python? That's right. The fact is, for more junior jobs, the Python is going to be the easiest part of the job, and new hires have plenty of time to learn, plus the team is there to help. New developers will come up to speed quickly provided they're comfortable learning in the environment.

As for environmental fluidity, specifically, PayPal uses a lot of Linux, so I look for candidates that can demonstrate familiarity at the console, interacting with the operating system. So while I don't usually give candidates complex algorithmic questions on the spot, I do log them into one of PayPal's test servers and have them do some basic debugging. For the experienced, you can almost feel them relaxing into a familiar environment. For the inexperienced, the terminal can be an aptly named dark and scary place. Either way, the command line is a foundational technology critical to enterprise work, and is not going away anytime soon. Lack of command line comfort is a big yellow flag, especially when Linux is so widespread and easy to experiment with on your own.

The other characteristic I look for is learning ability. The skills to read and research naturally, absorbing and arranging information automatically. I've been burned once or twice by talented people who were too lazy to read the docs, or too intimidated to read the source code. You don't have to do it in big gulps, but you do need to do it consistently. So I usually look at what candidates have done to learn lately, and the sources they've been consulting. Show me some code you've written and what you learned during the process. Tell me about a project that sounds much simpler than it was. What sites taught you the web? Seen any noteworthy source code lately?

On the other hand, I watch out for HackerNewsy types. My projects have topped HN several times in the last couple years, and some lurking is fine, but I want someone ready to outgrow that consumption and commodification of creative work interleaved with press releases. Someone ready to dedicate time to actually create the sorts of things that others will upvote.

Side experience

If someone is applying for their first job with you, what can they present to show they have the right skillset/education?

When it comes to first jobs and concrete projects, I'll look at anything and everything. With new developers it's just so rare to get someone with anything interesting in their GitHub or Bitbucket account, but that is definitely my first stop. Software is increasingly portfolio driven, and I do get a bit discouraged when I see a developer who doesn't have a GitHub, or a site, or even a blog. You can cram for an interview, and you can exaggerate on a resume, but you can't really fake a meaningful commit timeline going back a year or two. Even if it's just school projects, at least I could see you've tried and you have some basic git skills. Contributions to other projects tell a good story, too. You were probably using the project for something, being productive. You took the time to understand how it worked, you were able to communicate, and lived up to someone else's standards. That's stressful for a lot of people, but that's got a lot in common with enterprise development, too.

Side projects and apps that run in environments similar to our own are very interesting. Mobile phone apps not as much. Code competitions and scores from reddit/stackoverflow/HN are OK, but honestly those skills don't apply that well internally. This may make me unpopular, but people who have high scores on all those sites are playing games that can lead them to be impatient and unhelpful with internal people and processes. That said, if you're someone who helps out with mentorship or even get on IRC and answer questions, that could be great!

Formal education

If you are presented with two candidates, one with a solid CS degree, and the other with 1-2 years of experience, which would you value more?

Of the three hires I'd truly consider my "star" hires, none of them had a CS degree. Electrical engineering, math, and comparative literature. The things they had in common were voracious reading and extensive hours spent in some Python or POSIX environment.

Computer science degrees aren't really necessary for the majority of enterprise work. Like I said before, environmental fluidity and willingness to read docs are far more important. A couple of CS classes get you some useful vocabulary and teach you time complexity.

As for the concept of a degree in general, if you want to work at a big company, it's a lot easier to get in with a bachelors. You don't need much more than that. The right two years of experience can go a long way in terms of skills development, but in terms of management marketability, no degree raises eyebrows in many cases. So, in short, for enterprise software, my observation is that a computer science degree is about as good as a non-CS degree plus 2 years experience which is about as good as no degree plus 4-5 years experience, at least.

Most professors and academic programs don't give you all that much pragmatic knowledge, even if it's pretty old stuff like emacs and terminal usage. Basically everything is about how you approach your assignments and free time. If you push beyond the requirements, you will learn much more.

So, if you're in school, take an operating systems class. Take a networking class. Maybe a crypto class. You'll learn almost as much as running a shared server in your dorm. No, those are different types of knowledge, so consider doing both. If you're not in school, Coursera and other options are far better than nothing, and I'd like to hear about those experiences in interviews.

Last hire

Why did you hire the last person you hired?

I gave my most recent thumbs up to a developer who knew Django and was willing to continue working with it, but most importantly he could start on-site before the req closed. In large companies, empty seats have expiration dates, and everyone is willing to gamble. Because somebody is better than nobody, and even if they're worse than nobody, then you still get a backfill when they leave or are pushed out. But this developer seems to be working out, but I only helped hire him for another team.

The last engineer I hired onto my team was recruited over the course of two years. I met him at PyCon 2012 and we collaborated on a few open-source projects. Real recruiting can be a long process, not the least of which is due to weird budgeting and bureaucracy. So please don't get frustrated if you're still waiting on an email reply from me! :)

Takeaways

Reduced to a few bullet points, here are the key characteristics:

Environmental fluidity
Reading ability and conceptual familiarity
Command line comfort
Not HackerNewsy
Dedication. No technical butterflies here, please.
Pragmatism, lack of frustration
Management marketability
Ability/willingness to work/train/visit onsite

The other interviewees had some interesting things to say, as well. I recommend checking out the full podcast, now featuring transcripts for everyone, not just me. Thanks again to Michael for having me back!

RWC 2016 Lightning Talk

2016-01-07T20:20:00Z

Today I had the pleasure of talking on stage for ~2 minutes at the Real World Crypto 2016 conference in Stanford, CA. This is a pseudotranscript of that lightning talk.

I'm Mahmoud Hashemi and I work as a Lead Developer at PayPal. I mostly focus on Python frameworks and software infrastructure, but for the last couple years I've been working on Application Security. In fact, my first assignment, back in late 2012, was reverse engineering and reimplementing Max Levchin's Certicom elliptic curve integration, in Python.

These days I work on PayPal's comprehensive key management (and HSM integration) system. Suffice to say, we work a lot with encryption and secure sockets. Also suffice to say, we're a bit nervous about OpenSSL. With all the news lately we've started design discussions with regard to how we can hedge our OpenSSL bets.

In Python, this translates to a DBAPI 2.0-like abstraction layer to enable swapping out security implementations. Like many ORMs (e.g., SQLAlchemy), but for security. Honestly, there are usually better/more reasons to switch SSL implementations than relational databases. We want an API that allows us to leverage other great SSL implementations, including OpenSSL-derivatives like LibreSSL, as well as other implementations like WolfSSL. PayPal already has a diverse SSL ecosystem, with multiple versions of OpenSSL and tons of JVM-based implementations, making it a great testbed ecosystem.

To achieve this we're hoping to have some productive discussions with the experienced engineers and cryptographers that attend RWC. It's still very early days, and there are a lot of corner cases, so we'll need all the advice we can get. Help us invest in the algorithms, not the implementations. Design for replaceability, to avoid having 17-year-old libraries serving today's security-hungry Internet. You can contact me at github.com/mahmoud, twitter.com/mhashemi, or mahmoud@paypal.com.

A partially obfuscated view from the stage of RWC2016

Enterprise Overhaul: Resolving DNS

2015-12-21T11:13:00Z

Originally published on the PayPal Engineering blog. Republished here with minor modifications and updates.

Everyone assumes all software engineers are great with numbers. If only they knew the truth. How many people's phone numbers can you recite? No peeking and emergency numbers don't count! Don't worry if you couldn't name that many. Here's the real embarrassing test of the day: How many sites' IP addresses can you name? No pinging and local subnets don't count!

Most telephones still looked like this when DNS was invented. Not pictured: the phonebook.

Back in the mid-1980s, the first Domain Name System (DNS) implementations started putting our IP addresses into server-based contact lists and the Internet has never looked the same since. These days, we may associate DNS with large-scale networks, but it's important to remember that DNS really came from a very human distaste for numbers. Thirty years later, we engineers use it so much in normal Internet usage that it's easy to take for granted.

DNS may be a mature, but the fact of networks is that it always takes at least two to tango. As new technologies and deployments emerge, the implications of integrating with DNS must still be revisited. Your datacenter is not the Internet, even if it's in the cloud. This post looks at how to resolve a few of the DNS pitfalls preying on precious reliability and performance.

A protocol precaution

The client side of DNS, resolution, is virtually all UDP. This is interesting because UDP is designed as a lightweight, unreliable transport. However, in many of the most common use cases, DNS calls precede TCP-backed HTTP and other protocols based on reliable transports. This fundamental difference changes many things. Looking upstream, UDP does not load-balance like TCP. Because UDP is not connection-oriented or congestion-controlled, DNS traffic will act very differently at scale.

So our first lesson is to stay true to the stateless nature of UDP and avoid putting stateful load balancers in front of DNS infrastructure. Instead, configure clients and servers to conform to the built-in load-handling architecture of DNS. The Internet's DNS "deployment" is load balanced via its inherent hierarchy and IP Anycast.

Client integration

Back on the client side, you can do a lot to optimize and robustify your application's DNS integration. The first step is to take a hard look at your stack. Whether you're running Python, Java, JavaScript, or C++, the defaults may not be for you, especially when working with traffic within the datacenter.

For example, while not supported here at PayPal, it's safe to say Tornado is a popular Python web framework, with many asynchronous networking features. But, silently and subtly, DNS is not one of them. Tornado's default DNS resolution behavior will block the entire IO event loop, leading to big issues at scale.

And that's just one example of library DNS defaults jeopardizing application reliability. Third-party packages and sometimes even builtins in Java, Node.js, Python, and other stacks are full of hidden DNS faux pas.

For instance, the average off-the-shelf HTTP client seems like a neutral-enough component. Where would we be without reliable standbys like wget? And that is how the trouble starts. The DNS defaults in most tools are designed to make for good Internet citizens, not reliable and performant enterprise foundations.

The hops Internet-connected applications make for you. It's no wonder the default timeout is 5000 milliseconds.

The first difference is name resolution timeouts. By default, resolve.conf, netty, and c-ares (gevent, node.js, curl) are all configured to a whopping 5 seconds. But this is your enterprise, your service, and your datacenter. Look at the SLA of your service and the reliability of your DNS. If your service can't take an extra 5000 milliseconds some percentage of the time, then you should lower that timeout. I've usually recommended 200 milliseconds or less. If your infrastructure can't resolve DNS faster than that, do one or more of the following:

Put the authoritative DNS servers topologically closer.
Add caching DNS servers, maybe even on the same machine.
Build application-level DNS caching.

Option #1 is purely a network issue, and a matter for network operations to discuss. For brevity's sake, option #2 is outside the scope of this article. But option #3 is the one we recommend most, because it is bureaucracy-free and relatively easy to implement, even with enterprise considerations.

Application-level DNS caching

When designing an enterprise application-level DNS cache, we must recognize that we are not discussing standard-issue web components like scrapers and browsers. Most enterprise services talk to a fixed set of relatively few machines. Even the most powerful and complex production PayPal services communicate with fewer than 200 addresses, partly due to the prevalence of load balancing LTMs in our architecture.

For our gevent-based Python stack, we use an asynchronous DNS cache that refreshes those addresses every five minutes. Plus, the stack warms up our application's DNS cache by kicking off preresolution of many known DNS-addressed hosts at startup, ensuring that the first requests are as fast as later ones.

Some may be asking, why use a custom, application-level DNS cache when virtually every operating system caches DNS automatically? In short, when the OS cache expires, the next DNS resolution will block, causing stacks without this asynchronous DNS cache to block on the next resolution. Our DNS cache allows us to use mildly stale addresses while the cache is refreshing, making us robust to many DNS issues. For our use cases both the chances and consequences of connecting to the wrong server are so minute that it's not worth inflating outlier response times by inlining DNS. This arrangement also makes services much more robust to network glitches and DNS outages, as well as allowing for more logging and instrumentation around the explicit DNS resolution so you can see when DNS is performing badly.

Denecessitizing DNS

The overhaul wouldn't be complete without exploring one final scenario. What's it like to not use DNS at all? It may sound odd, given the number of technologies built on DNS in the last 30 years. But even today, PayPal production services still communicate to each other using a statically generated IP-address-based system, like a souped-up hosts file. This design decision long predates my tenure here, and for a long time I considered it technical debt. But after collaborating with architects here and at other enterprise datacenters, I've come to appreciate the advantages of skipping DNS. DNS was designed for multi-authority, federated, eventually-consistent networks, like the Internet. Even the biggest datacenters are not the Internet. A datacenter is topologically smaller, has only one operational authority, and must meet much tighter reliability requirements.

A little peek at PayPal's midtier-to-midtier traffic. Each shrunken line of text is a service endpoint. It looks like a lot, but each endpoint only talks to a few others.

Whether or not your system uses DNS, when you own the entire network it's still best practice to maintain a central, version-controlled, "single source of truth" repository for networking configurations. After all, even DNS server configurations have to come from somewhere. If it were possible to efficiently and reliably push that same information to every client, would you? Explicit preresolution of all service names reduces the window of inconsistency while saving the datacenter billions of network requests. If you already have a scalable deployment system, could it also fill the network topology gap, saving you the trouble of overhauling, scaling, and maintaining an Internet system for enterprise use? There's a lot packed in a question like that, but it's something to consider when designing your service ecosystem.

In short

So, to sum it all up, here are the key takeaways:

Beware the pitfalls of stateful load-balancing for DNS and UDP.
Tighten up your timeouts according to your SLAs.
Consider an in-application DNS cache with explicit resolution.
The fastest and most reliable request is the request you don't have to make.
A datacenter is not the Internet.

If you're not careful, out-of-box solutions will fill your inbox with avoidable problems. Quality enterprise engineering means taking a microscope to libraries, with deliberate overhauling for your organization's needs.

Announcing the Hatnote Top 100

2015-12-14T13:00:00Z

Originally published on the Hatnote blog.

Moreso than any other major site, Wikipedia is centered around knowledge, always growing, and brimming with information. It's important to remember that the insight of our favorite community-run encyclopedia often follows the focus of its massive readership. Here at Hatnote, we've often wondered, what great new topics is the community learning about now?

To shed more light on Wikipedia's reading habits, we're pleased to announce the newest addition to the Hatnote family: The Hatnote Top 100, available at top.hatnote.com. Because we can't pass up a good headwear-based pun.

Updated daily, the Top 100 is a chart of the most-visited articles on Wikipedia. Unlike the edit-oriented Listen to Wikipedia and Weeklypedia, Top 100 focuses on the biggest group of Wikipedia users: the readers. Nearly 20 billion times per month, around 500 million people read articles in over 200 languages. Top 100's daily statistics offer a window into where Wikipedia readers are focusing their attention. It also makes for a great way to discover great chapters of Wikipedia one wouldn't normally read or edit.

Clear rankings, day-to-day differences, social media integration, permalinks, and other familiar simple-but-critical features were designed to make popular Wikipedia articles as relatable as albums on a pop music chart. In practice, popular news stories and celebrities definitely make the Top 100, but it is satisfying to see interesting corners of history and other educational topic sharing, if not dominating, the spotlight.

In addition to a clear and readable report, Top 100 is also a machine-readable archive, with reports dating back to November 2015, including JSON versions of the metrics, as well as RSS feeds for all supported languages and projects. It's all available in over a dozen languages (and we take requests for more). The data comes from a variety of sources, most direct from Wikimedia, including a new pageview statistics API endpoint that we've been proud to pilot and continue to use. And yes, as with all our projects the code is open-source, too.

For those of you looking to dig deeper than Wikipedia chart toppers, there are several other activity-based projects worth mentioning:

stats.grok.se - The original, venerable pageview grapher and API
Wikimedia Report Card - Advanced metrics and data used by the Wikimedia Foundation
The Open Wikipedia Ranking - Traffic stats and more
@WikipediaTrends - A bot posting notable upward traffic spikes
The Top 25 Report - A manually-compiled weekly report of views and likely reasons
The Weeklypedia - Weekly edit statistics, emailed and archived by Hatnote

And there are other visualizations on seealso.org as well. But for those who like to keep it simple, hit up the Hatnote Top 100, subscribe to a feed, and/or follow us on Twitter. See you there!

Repeat the obvious

2015-11-09T08:05:00Z

Bad things happen when we don't repeat the obvious.

It's 9pm and I'm writing a post for the company engineering blog. Every sentence is a slog. Not because I'm exacting and conciseness isn't my strong suit. My writing is slow because every word is obvious, almost patronizing.

Obvious realities bear repetition, and so must you. Common sense is not so common. The majority of ideas floating around try too hard. They're designed to confuse, seduce, and sell. Press releases and ads push to the forefront, while reviewed articles and texts sit on shelves and in queues.

Repeat the obvious, so we stay on the same page. The ways we rush people into technology leaves little time for foundations. Software is so new and developers so in-demand, every wave brings more fresh minds than the last. Developers are arriving faster than knowledge can diffuse.

Repeat the obvious, to keep perspective. Technology may favor the new, but fundamentals do exist. Without reminders, time buries working technologies in the dust of silence.

Repeat the obvious, to avoid bizarre dark ages. Take functional programming's disappearance in the 1990s/2000s, cast aside in favor of object orientated hype. Or that one time when not enough programmers talked about and taught event-driven servers programming and Frankenstein was cast as revolutionary.

So I hope you'll forgive the repetition. It hurts me more than it hurts you, and believe me when I say it helps many. Documentation does not equal disussion. The modern media landscape demands a technology have both docs and discourse to remain useful.

Until we live in a world where reference rules over repetition, you can help by writing about something painfully obvious to you. Bad things happen when we don't repeat the obvious.

Remap: Nested Data Multitool for Python

2015-09-24T19:25:00Z

This entry is the first in a series of "cookbooklets" showcasing more advanced Boltons. If all goes well, the next 5 minutes will literally save you 5 hours.

Contents

Intro
Normalize keys and values
Drop empty values
Convert dictionaries to OrderedDicts
Sort all lists
Collect interesting values
Add common keys
Corner cases
Wrap-up

Intro

Data is everywhere, especially within itself. That's right, whether it's public APIs, document stores, or plain old configuration files, data will nest. And that nested data will find you.

UI fads aside, developers have always liked "flat". Even Python, so often turned to for data wrangling, only has succinct built-in constructs for dealing with flat data. List comprehensions, generator expressions, map/filter, and itertools are all built for flat work. In fact, the allure of flat data is likely a direct result of this common gap in most programming languages.

Let's change that. First, let's meet this nested adversary. Provided you overlook my taste in media, it's hard to fault nested data when it reads as well as this YAML:

reviews:
  shows:
    - title: Star Trek - The Next Generation
      rating: 10
      review: Episodic AND deep. <3 Data.
      tags: ['space']
    - title: Monty Python's Flying Circus
      rating: 10
      tags: ['comedy']
  movies:
    - title: The Hitchiker's Guide to the Galaxy
      rating: 6
      review: So great to see Mos Def getting good work.
      tags: ['comedy', 'space', 'life']
    - title: Monty Python's Meaning of Life
      rating: 7
      review: Better than Brian, but not a Holy Grail, nor Completely Different.
      tags: ['comedy', 'life']
      prologue:
        title: The Crimson Permanent Assurance
        rating: 9

Even this very straightforwardly nested data can be a real hassle to manipulate. How would one add a default review for entries without one? How would one convert the ratings to a 5-star scale? And what does all of this mean for more complex real-world cases, exemplified by this excerpt from a real GitHub API response:

[{
    "id": "3165090957",
    "type": "PushEvent",
    "actor": {
      "id": 130193,
      "login": "mahmoud",
      "gravatar_id": "",
      "url": "https://api.github.com/users/mahmoud",
      "avatar_url": "https://avatars.githubusercontent.com/u/130193?"
    },
    "repo": {
      "id": 8307391,
      "name": "mahmoud/boltons",
      "url": "https://api.github.com/repos/mahmoud/boltons"
    },
    "payload": {
      "push_id": 799258895,
      "size": 1,
      "distinct_size": 1,
      "ref": "refs/heads/master",
      "head": "27a4bc1b6d1da25a38fe8e2c5fb27f22308e3260",
      "before": "0d6486c40282772bab232bf393c5e6fad9533a0e",
      "commits": [
        {
          "sha": "27a4bc1b6d1da25a38fe8e2c5fb27f22308e3260",
          "author": {
            "email": "mahmoud@hatnote.com",
            "name": "Mahmoud Hashemi"
          },
          "message": "switched reraise_visit to be just a kwarg",
          "distinct": true,
          "url": "https://api.github.com/repos/mahmoud/boltons/commits/27a4bc1b6d1da25a38fe8e2c5fb27f22308e3260"
        }
      ]
    },
    "public": true,
    "created_at": "2015-09-21T10:04:37Z"
}]

The astute reader may spot some inconsistency and general complexity, but don't run away.

Remap, the recursive map, is here to save the day.

Remap is a Pythonic traversal utility that creates a transformed copy of your nested data. It uses three callbacks -- visit, enter, and exit -- and is designed to accomplish the vast majority of tasks by passing only one function, usually visit. The API docs have full descriptions, but the basic rundown is:

visit transforms an individual item
enter controls how container objects are created and traversed
exit controls how new container objects are populated

It may sound complex, but the examples shed a lot of light. So let's get remapping!

Normalize keys and values

First, let's import the modules and data we'll need.

import json
import yaml  # https://pypi.python.org/pypi/PyYAML
from boltons.iterutils import remap  # https://pypi.python.org/pypi/boltons

review_map = yaml.load(media_reviews)

event_list = json.loads(github_events)

Now let's turn back to that GitHub API data. Earlier one may have been annoyed by the inconsistent type of id. event['repo']['id'] is an integer, but event['id'] is a string. When sorting events by ID, you would not want string ordering.

With remap, fixing this sort inconsistency couldn't be easier:

from boltons.iterutils import remap

def visit(path, key, value):
    if key == 'id':
        return key, int(value)
    return key, value

remapped = remap(event_list, visit=visit)

assert remapped[0]['id'] == 3165090957

# You can even do it in one line:
remap(event_list, lambda p, k, v: (k, int(v)) if k == 'id' else (k, v))

By default, visit gets called on every item in the root structure, including lists, dicts, and other containers, so let's take a closer look at its signature. visit takes three arguments we're going to see in all of remap's callbacks:

path is a tuple of keys leading up to the current item
key is the current item's key
value is the current item's value

key and value are exactly what you would expect, though it may bear mentioning that the key for a list item is its index. path refers to the keys of all the parents of the current item, not including the key. For example, looking at the GitHub event data, the commit author's name's path is (0, 'payload', 'commits', 0, 'author'), because the key, name, is located in the author of the first commit in the payload of the first event.

As for the return signature of visit, it's very similar to the input. Just return the new (key, value) you want in the remapped output.

Drop empty values

Next up, GitHub's move away from Gravatars left an artifact in their API: a blank 'gravatar_id' key. We can get rid of that item, and any other blank strings, in a jiffy:

drop_blank = lambda p, k, v: v != ""
remapped = remap(event_list, visit=drop_blank)

assert 'gravatar_id' not in remapped[0]['actor']

Unlike the previous example, instead of a (key, value) pair, this visit is returning a bool. For added convenience, when visit returns True, remap carries over the original item unmodified. Returning False drops the item from the remapped structure.

With the ability to arbitrarily transform items, pass through old items, and drop items from the remapped structure, it's clear that the visit function makes the majority of recursive transformations trivial. So many tedious and error-prone lines of traversal code turn into one-liners that usually remap with a visit callback is all one needs. With that said, the next recipes focus on remap's more advanced callable arguments, enter and exit.

Convert dictionaries to OrderedDicts

So far we've looked at actions on remapping individual items, using the visit callable. Now we turn our attention to actions on containers, the parent objects of individual items. We'll start doing this by looking at the enter argument to remap.

# from collections import OrderedDict
from boltons.dictutils import OrderedMultiDict as OMD
from boltons.iterutils import remap, default_enter

def enter(path, key, value):
    if isinstance(value, dict):
        return OMD(), sorted(value.items())
    return default_enter(path, key, value)

remapped = remap(review_list, enter=enter)
assert remapped['reviews'].keys()[0] == 'movies'
# True because 'reviews' is now ordered and 'movies' comes before 'shows'

The enter callable controls both if and how an object is traversed. Like visit, it accepts path, key, and value. But instead of (key, value), it returns a tuple of (new_parent, items). new_parent is the container that will receive items remapped by the visit callable. items is an iterable of (key, value) pairs that will be passed to visit. Alternatively, items can be False, to tell remap that the current value should not be traversed, but that's getting pretty advanced. The API docs have some other enter details to consider.

Also note how this code builds on the default remap logic by calling through to the default_enter function, imported from the same place as remap itself. Most practical use cases will want to do this, but of course the choice is yours.

Sort all lists

The last example used enter to interact with containers before they were being traversed. This time, to sort all lists in a structure, we'll use the remap's final callable argument: exit.

from boltons.iterutils import remap, default_exit

def exit(path, key, old_parent, new_parent, new_items):
    ret = default_exit(path, key, old_parent, new_parent, new_items)
    if isinstance(ret, list):
        ret.sort()
    return ret

remap(review_list, exit=exit)

Similar to the enter example, we're building on remap's default behavior by importing and calling default_exit. Looking at the arguments passed to exit and default_exit, there's the path and key that we're used to from visit and enter. value is there, too, but it's named old_parent, to differentiate it from the new value, appropriately called new_parent. At the point exit is called, new_parent is just an empty structure as constructed by enter, and exit's job is to fill that new container with new_items, a list of (key, value) pairs returned by remap's calls to visit. Still with me?

Either way, here we don't interact with the arguments. We just call default_exit and work on its return value, new_parent, sorting it in-place if it's a list. Pretty simple! In fact, very attentive readers might point out this can be done with visit, because remap's very next step is to call visit with the new_parent. You'll have to forgive the contrived example and let it be a testament to the rarity of overriding exit. Without going into the details, enter and exit are most useful when teaching remap how to traverse nonstandard containers, such as non-iterable Python objects. As mentioned in the "drop empty values" example, remap is designed to maximize the mileage you get out of the visit callback. Let's look at an advanced usage reason that's true.

Collect interesting values

Sometimes you just want to traverse a nested structure, and you don't need the result. For instance, if we wanted to collect the full set of tags used in media reviews. Let's create a remap-based function, get_all_tags:

def get_all_tags(root):
    all_tags = set()

    def visit(path, key, value):
        all_tags.update(value['tags'])
        return False

    remap(root, visit=visit, reraise_visit=False)

    return all_tags

print(get_all_tags(review_map))
# set(['space', 'comedy', 'life'])

Like the first recipe, we've used the visit argument to remap, and like the second recipe, we're just returning False, because we don't actually care about contents of the resulting structure.

What's new here is the reraise_visit=False keyword argument, which tells remap to keep any item that causes a visit exception. This practical convenience lets visit functions be shorter, clearer, and just more EAFP. Reducing the example to a one-liner is left as an exercise to the reader.

Add common keys

As a final advanced remap example, let's look at adding items to structures. Through the examples above, we've learned that visit is best-suited for 1:1 transformations and dropping values. This leaves us with two main approaches for addition. The first uses the enter callable and is suitable for making data consistent and adding data which can be overridden.

base_review = {'title': '',
               'rating': None,
               'review': '',
               'tags': []}

def enter(path, key, value):
    new_parent, new_items = default_enter(path, key, value)
    try:
        new_parent.update(base_review)
    except:
        pass
    return new_parent, new_items

remapped = remap(review_list, enter=enter)

assert review_list['shows'][1]['review'] == ''
# True, the placeholder review is holding its place

The second method uses the exit callback to override values and calculate new values from the new data.

def exit(path, key, old_parent, new_parent, new_items):
    ret = default_exit(path, key, old_parent, new_parent, new_items)
    try:
        ret['review_length'] = len(ret['review'])
    except:
        pass
    return ret

remapped = remap(review_list, exit=exit)

assert remapped['shows'][0]['review_length'] == 27
assert remapped['movies'][0]['review_length'] == 42
# True times two.

By now you might agree that remap is making such feats positively routine. Come for the nested data manipulation, stay for the number jokes.

Corner cases

This whole guide has focused on data that came from "real-world" sources, such as JSON API responses. But there are certain rare cases which typically only arise from within Python code: self-referential objects. These are objects that contain references to themselves or their parents. Have a look at this trivial example:

self_ref = []
self_ref.append(self_ref)

The experienced programmer has probably seen this before, but most Python coders might even think the second line is an error. It's a list containing itself, and it has the rather cool repr: [[...]].

Now, this is pretty rare, but reference loops do come up in programming. The good news is that remap handles these just fine:

print(repr(remap(self_ref)))
# prints "[[...]]"

The more common corner case that arises is that of duplicate references, which remap also handles with no problem:

my_set = set()

dupe_ref = (my_set, [my_set])
remapped = remap(dupe_ref)

assert remapped[0] is remapped[-1][-1]
# True, of course

Two references to the same set go in, two references to a copy of that set come out. That's right: only one copy is made, and then used twice, preserving the original structure.

Wrap-up

If you've made it this far, then I hope you'll agree that remap is useful enough to be your new friend. If that wasn't enough detail, then there are the docs. remap is well-tested, but making something this general-purpose is a tricky area. Please file bugs and requests. Don't forget about pprint and repr/reprlib, which can help with reading large structures. As always, stay tuned for future boltons cookbooklets, and much much more.

Python Community Intro

2015-09-22T07:00:00Z

The PSF just created a new mailing list, "PSF-Community", then autosubscribed a bunch of people and solicited introductions. At first I was surprised, but I was quickly charmed by the response and joined in on the action. Here's what I wrote:

If Alex Martelli is doing it, then brace yourselves because the floodgates are open.

I first used Python as a junior in a South Dakota high school, off a Knoppix CD because "Live CDs" were all the rage then. It was a good fad because I didn’t have a computer, and the Windows machines at school weren’t writable and didn’t have Python (2.2 at the time). I read a bit of the tutorial and wrote a really bad prime number sieve.

After a professional loop through Java, C++, C#, and finally PHP, I resumed Python development in 2009 as a full-stack web developer at PayPal. I wrote the tool that (still) manages all the pricing arrangements.

From there I hired my first teammate and we wrote a couple other business-critical components before standardizing out PayPal’s first grassroots alternative stack. That was early 2011 and since then we’ve had a lot of fun and come so far. Now we’re focusing on PayPal’s security offerings: putting Python at the very heart of PayPal’s availability model, handling billions of requests per day. And believe me when I say that’s it’s the best thing that’s happened to PayPal’s security in a long time! The details will have to wait for a future blog post (and upcoming O’Reilly project). Or, if you’re remotely as excited as I am, you can email me directly. :)

On the side, I really enjoy working on Wikipedia-based projects > under the banner of Hatnote, all Python. Most recently, we did the official Wikipedia IFTTT channel (handling 1.3 million requests per day). And because I can’t get enough, a bunch of open-source stuff, most notably Boltons, where I’ve been particularly busy lately.

If you’re in the Bay Area, do not hesitate to reach out to talk about Python, Wikipedia, security, federated and open systems (like BBS stuff), or even PayPal!

Specifically, this is sort of odd, but October 14th at 1pm, I'm doing an overview of Python usage at PayPal, and would like to invite anyone senior and curious to be my guest and come to PayPal in San Jose to check it out. Guido came in 2012 and he loved it. And stuff now is waaaay cooler!

Anyways, I just wanted to end by saying thanks to you all. If you hadn't been so numerous and out there, I probably would have gotten myself fired long before any of this bore fruit. ;)

THANKS!

Mahmoud

There were a lot of autosubscribed folks deploring the spammish inquisition and threatening unsubscription, so here's hoping my straw didn't break too many camels backs.

10 Myths of Enterprise Python

2015-08-25T07:00:00Z

(Originally posted on the PayPal Engineering blog, reproduced here with minor updates, link fixes, etc.)

PayPal enjoys a remarkable amount of linguistic pluralism in its programming culture. In addition to the long-standing popularity of C++ and Java, an increasing number of teams are choosing JavaScript and Scala, and Braintree's acquisition has introduced a sophisticated Ruby community.

One language in particular has both a long history at eBay and PayPal and a growing mindshare among developers: Python.

Python has enjoyed many years of grassroots usage and support from developers across eBay. Even before official support from management, technologists of all walks went the extra mile to reap the rewards of developing in Python. I joined PayPal a few years ago, and chose Python to work on internal applications, but I've personally found production PayPal Python code from nearly 15 years ago.

Today, Python powers over 50 projects, including:

Features and products, such as eBay Now and RedLaser
Operations and infrastructure, both OpenStack and proprietary
Mid-tier services and applications, like the one used to set PayPal's prices and check customer feature eligibility
Monitoring agents and interfaces, used for several deployment and security use cases
Batch jobs for data import, price adjustment, and more
And far too many developer tools to count

In the coming series of posts I'll detail the initiatives and technologies that led the eBay/PayPal Python community to grow from just under 25 engineers in 2011 to over 260 in 2014. For this introductory post, I'll be focusing on the 10 myths I've had to debunk the most in eBay and PayPal's enterprise environments.

Myth #1: Python is a new language

What with all the startups using it and kids learning it these days, it's easy to see how this myth still persists. Python is actually over 23 years old, originally released in 1991, 4 years before Java. A now-famous early usage of Python was in 1996: Google's first successful web crawler.

If you're curious about the long history of Python, Guido van Rossum, Python's creator, has taken the care to tell the whole story.

Myth #2: Python is not compiled

While not requiring a separate compiler toolchain like C++, Python is in fact compiled to bytecode, much like Java and many other compiled languages. Further compilation steps, if any, are at the discretion of the runtime, be it CPython, PyPy, Jython/JVM, IronPython/CLR, or some other process virtual machine. See Myth #6 for more info.

The general principle at PayPal and elsewhere is that the compilation status of code should not be relied on for security. It is much more important to secure the runtime environment, as virtually every language has a decompiler, or can be intercepted to dump protected state. See the next myth for even more Python security implications.

Myth #3: Python is not secure

Python's affinity for the lightweight may not make it seem formidable, but the intuition here can be misleading. One central tenet of security is to present as small a target as possible. Big systems are anti-secure, as they tend to overly centralize behaviors, as well as undercut developer comprehension. Python keeps these demons at bay by encouraging simplicity. Furthermore, CPython[cypython] addresses these issues by being a simple, stable, and easily-auditable virtual machine. In fact, a recent analysis by Coverity Software resulted in CPython receiving their highest quality rating.

Python also features an extensive array of open-source, industry-standard security libraries. At PayPal, where we take security and trust very seriously, we find that a combination of hashlib, PyCrypto, and OpenSSL, via PyOpenSSL and our own custom bindings, cover all of PayPal's diverse security and performance needs.

For these reasons and more, Python has seen some of its fastest adoption at PayPal (and eBay) within the application security group. Here are just a few security-based applications utilizing Python for PayPal's security-first environment:

Creating security agents for facilitating key rotation and consolidating cryptographic implementations
Integrating with industry-leading HSM technologies
Constructing TLS-secured wrapper proxies for less-compliant stacks
Generating keys and certificates for our internal mutual-authentication schemes
Developing active vulnerability scanners

Plus, myriad Python-built operations-oriented systems with security implications, such as firewall and connection management. In the future we'll definitely try to put together a deep dive on PayPal Python security particulars.

Myth #4: Python is a scripting language

Python can indeed be used for scripting, and is one of the forerunners of the domain due to its simple syntax, cross-platform support, and ubiquity among Linux, Macs, and other Unix machines.

In fact, Python may be one of the most flexible technologies among general-use programming languages. To list just a few:

Telephony infrastructure (Twilio)
Payments systems (PayPal, [Venmo][venmo])
Neuroscience and psychology (citation)
Numerical analysis and engineering (numpy, numba, and many more)
Animation (LucasArts, Disney, Dreamworks)
Gaming backends (Eve Online, Second Life, Battlefield, and so many others)
Email infrastructure (Mailman, Mailgun)
Media storage and processing (YouTube, Instagram, Dropbox)
Operations and systems management (Rackspace, OpenStack)
Natural language processing (NLTK)
Machine learning and computer vision (scikit-learn, Orange, SimpleCV)
Security and penetration testing (so many)
Big Data (Disco, Hadoop support)
Internet infrastructure (DNS) (BIND 10)

Not to mention websites and web services aplenty. In fact, PayPal engineers seem to have a penchant for going on to start Python-based web properties. YouTube and Yelp, for instance.

Myth #5: Python is weakly-typed

Python's type system is characterized by strong, dynamic typing. Wikipedia can explain more.

Not that it is a competition, but as a fun fact, Python is more strongly-typed than Java. Java has a split type system for primitives and objects, with null lying in a sort of gray area. On the other hand, modern Python has a unified strong type system, where the type of None is well-specified. Furthermore, the JVM itself is also dynamically-typed, as it traces its roots back to an implemention of a Smalltalk VM acquired by Sun.

Python's type system is very nice, but for enterprise use there are much bigger concerns at hand.

Myth #6: Python is slow

First, a critical distinction: Python is a programming language, not a runtime. There are several Python implementations:

CPython is the reference implementation, and also the most widely distributed and used.
Jython is a mature implementation of Python for usage with the JVM.
IronPython is Microsoft's Python for the Common Language Runtime, aka .NET.
PyPy is an up-and-coming implementation of Python, with advanced features such as JIT compilation, incremental garbage collection, and more.

Each runtime has its own performance characteristics, and none of them are slow per se. The more important point here is that it is a mistake to assign performance assessments to a programming languages. Always assess an application runtime, most preferably against a particular use case.

Having cleared that up, here is a small selection of cases where Python has offered significant performance advantages:

Using NumPy as an interface to Intel's MKL SIMD
PyPy's JIT compilation achieves faster-than-C performance
Disqus scales from 250 to 500 million users on the same 100 boxes

Admittedly these are not the newest examples, just my favorites. It would be easy to get side-tracked into the wide world of high-performance Python and the unique offerings of runtimes. Instead of addressing individual special cases, attention should be drawn to the generalizable impact of developer productivity on end-product performance, especially in an enterprise setting.

Given enough time, a disciplined developer can execute the only proven approach to achieving accurate and performant software:

Engineer for correct behavior, including the development of respective tests
Profile and measure performance, identifying bottlenecks
Optimize, paying proper respect to the test suite and Amdahl's Law, and taking advantage of Python's strong roots in C.

It might sound simple, but even for seasoned engineers, this can be a very time-consuming process. Python was designed from the ground up with developer timelines in mind. In our experience, it's not uncommon for Python projects to undergo three or more iterations in the time it C++ and Java to do just one. Today, PayPal and eBay have seen multiple success stories wherein Python projects outperformed their C++ and Java counterparts, all thanks to fast development times enabling careful tailoring and optimization. You know, the fun stuff.

Myth #7: Python does not scale

Scale has many definitions, but by any definition, YouTube is a web site at scale. More than 1 billion unique visitors per month, over 100 hours of uploaded video per minute, and going on 20% of peak Internet bandwidth, all with Python as a core technology. Dropbox, Disqus, Eventbrite, Reddit, Twilio, Instagram, Yelp, EVE Online, Second Life, and, yes, eBay and PayPal all have Python scaling stories that prove scale is more than just possible: it's a pattern.

The key to success is simplicity and consistency. CPython, the primary Python virtual machine, maximizes these characteristics, which in turn makes for a very predictable runtime. One would be hard pressed to find Python programmers concerned about garbage collection pauses or application startup time. With strong platform and networking support, Python naturally lends itself to smart horizontal scalability, as manifested in systems like BitTorrent.

Additionally, scaling is all about measurement and iteration. Python is built with profiling and optimization in mind. See Myth #6 for more details on how to vertically scale Python.

Myth #8: Python lacks good concurrency support

Occasionally debunking performance and scaling myths, and someone tries to get technical, "Python lacks concurrency," or, "What about the GIL?" If dozens of counterexamples are insufficient to bolster one's confidence in Python's ability to scale vertically and horizontally, then an extended explanation of a CPython implementation detail probably won't help, so I'll keep it brief.

Python has great concurrency primitives, including [generators][gen_concurrency], greenlets, Deferreds, and futures. Python has great concurrency frameworks, including eventlet, gevent, and Twisted. Python has had some amazing work put into customizing runtimes for concurrency, including Stackless and PyPy. All of these and more show that there is no shortage of engineers effectively and unapologetically using Python for concurrent programming. Also, all of these are officially support and/or used in enterprise-level production environments. For examples, refer to Myth #7.

The Global Interpreter Lock, or GIL, is a performance optimization for most use cases of Python, and a development ease optimization for virtually all CPython code. The GIL makes it much easier to use OS threads or green threads (greenlets usually), and does not affect using multiple processes. For more information, see this great Q&A on the topic and this overview from the Python docs.

Here at PayPal, a typical service deployment entails multiple machines, with multiple processes, multiple threads, and a very large number of greenlets, amounting to a very robust and scalable concurrent environment. In most enterprise environments, parties tends to prefer a fairly high degree of overprovisioning, for general prudence and disaster recovery. Nevertheless, in some cases Python services still see millions of requests per machine per day, handled with ease.

Myth #9: Python programmers are scarce

There is some truth to this myth. There are not as many Python web developers as PHP or Java web developers. This is probably mostly due to a combined interaction of industry demand and education, though trends in education suggest that this may change.

That said, Python developers are far from scarce. There are millions worldwide, as evidenced by the dozens of Python conferences, tens of thousands of StackOverflow questions, and companies like YouTube, Bank of America, and LucasArts/Dreamworks employing Python developers by the hundreds and thousands. At eBay and PayPal we have hundreds of developers who use Python on a regular basis, so what's the trick?

Well, why scavenge when one can create? Python is exceptionally easy to learn, and is a first programming language for children, university students, and professionals alike. At eBay, it only takes one week to show real results for a new Python programmer, and they often really start to shine as quickly as 2-3 months, all made possible by the Internet's rich cache of interactive tutorials, books, documentation, and open-source codebases.

Another important factor to consider is that projects using Python simply do not require as many developers as other projects. As mentioned in Myth #7, lean, effective teams like Instagram are a common trope in Python projects, and this has certainly been our experience at eBay and PayPal.

Myth #10: Python is not for big projects

Myth #7 discussed running Python projects at scale, but what about developing Python projects at scale? As mentioned in Myth #9, most Python projects tend not to be people-hungry. while Instagram reached hundreds of millions of hits a day at the time of their billion dollar acquisition, the whole company was still only a group of a dozen or so people. Dropbox in 2011 only had 70 engineers, and other teams were similarly lean. So, can Python scale to large teams?

Bank of America actually has over 5,000 Python developers, with over 10 million lines of Python in one project alone. JP Morgan underwent a similar transformation. YouTube also has engineers in the thousands and lines of code in the millions. Big products and big teams use Python every day, and while it has excellent modularity and packaging characteristics, beyond a certain point much of the general development scaling advice stays the same. Tooling, strong conventions, and code review are what make big projects a manageable reality.

Luckily, Python starts with a good baseline on those fronts as well. We use PyFlakes and other tools to perform static analysis of Python code before it gets checked in, as well as adhering to PEP8, Python's language-wide base style guide.

Finally, it should be noted that, in addition to the scheduling speedups mentioned in Myth #6 and #7, projects using Python generally require fewer developers, as well. Our most common success story starts with a Java or C++ project slated to take a team of 3-5 developers somewhere between 2-6 months, and ends with a single motivated developer completing the project in 2-6 weeks. It's not unheard of for some projects to take hours instead of weeks, as well.

A miracle for some, but a fact of modern development, and often a necessity for a competitive business.

A clean slate

Mythology can be a fun pastime. Discussions around these myths remain some of the most active and educational, both internally and externally, because implied in every myth is a recognition of Python's strengths. Also, remember that the appearance of these seemingly tedious and troublesome concerns is a sign of steadily growing interest, and with steady influx of interested parties comes the constant job of education. Here's hoping that this post manages to extinguish a flame war and enable a project or two to talk about the real work that can be achieved with Python.

Keep an eye out for future posts where I'll dive deeper into the details touched on in this overview. If you absolutely must have details before then, shoot me an email at mahmoud@paypal.com. Until then, happy coding!

Designing a fast

2015-07-16T07:00:00Z

I wake up with a jolt, spilling most of my breakfast cereal onto a thirsty couch. My eyes find the clock. Cleaning will have to wait. I'm downing water like there's no tomorrow, but really tomorrow starts in one minute. Still drinking. All work is thirsty work if the day is long enough, and engineering is no exception. Time's up.

From the literal break of dawn to sunset, no food, drink, or other respite. It's Ramadan. What does this mean, practically? Well, summertime here in Silicon Valley, it means from 4am to 9pm, I battle human nature while writing emails and software. But, far from an antiquated ritual, I see Ramadan as an exercise in lifestyle design.

As we near the end of Ramadan 1436, this year has proven that even in modern and diverse environs, every year brings the same reactions and questions as 1435. Mostly boiling down to:

What? Not even water?

A bit facetious, but this really is the most common question I get. So just to be clear, traditional interpretation calls for no food, drink (including water), or drugs. From the crack of dawn to sunset. Or in the technical terms, the beginning of sunrise's astronomical twilight to the beginning of sunset's civil twilight.

Individuals adjust according to limitations. If you're not healthy enough to fast, you don't fast. If you feel like you can't complete a fast, you don't. If the sun doesn't set, just do something reasonable. Your intentions are your own, and self-harm does not enter into the purposes of Ramadan.

Why?

Everyone has their reasons, but first off Ramadan is not some sort of collective diet. Yes, Ramadan is used by many as a springboard to stymie smoking, overeating, and other unhealthy physical habits. But for me, fasting is about building four virtues:

Empathy
Reflection
Discipline
Confidence

Not exactly the stuff of classrooms and annual compliance trainings. And yet people are expected to just find these characteristics within themselves, even in environments most antithetical. Countless well-compensated designers and engineers know about the limits of limitless life. We almost immediately pine for constraints. Negative liberty only goes so far, then real freedom becomes about the ability to formulate and follow the orders you give yourself. Design grants creative autonomy, but design tools offer a hundred possibilities draped in a thousand distractions.

Empathy is the most obvious trait built by fasting, and the one promoted most when I was younger. There are poor people in the world, and all should experience their hunger and thirst to understand. Fasting puts you on the path closest to the one they walk, building a visceral empathy that simple imagination can't match. When was the last time you were hungry like the wolf? One month of senses too sharp for civil society. One month of feeling the natural appetites object and interrupt your every thought. But it keeps one connected to so many people, from the most intense protesters to as many as a fifth of American students.

Reflection is critical to the Ramadan fast. Take away food and water, and within a few hours you're transported to the banks of a personal Walden Pond. In much the same way that exercise burns off dirty, anxious energy, fasting stops it from being produced in the first place. It quiets the shores of one's psyche and in the stillness, all is clear. This is the part of Ramadan I look forward to most: a staycation from my usual self-imposed obligations. The line between essential and unnecessary is bright. I don't know much about meditation, but most days of the month, around sunset, I find a certain peaceful state, every thought sorted away in its right place.

Midday is another story. Shouldering a normal workload with the added constraint of a fast is the definition of a stress test. Except unlike software and other commonly-tested constructs, the systems at work here involved grow and strengthen naturally. During Ramadan, I stockpile this discipline to burn over the next 11 months. Discipline complements motivation, especially with creative work like software and architecture. Whereas frustration obviates motivation, discipline rises to the occasion, grateful for the opportunity to push through and grow.

All of the above pours into the last attribute. Confidence is deeply linked to feelings of sufficiency: the ability to say, "What I have is enough to do what I want to do." I'm a big fan of water myself, but even something as essential as hydration isn't as big a deal as we make it. My adolescent fascination with basketball was rooted in Hakeem Olajuwon playing whole NBA games against the Chicago Bulls, 12 hours into a fast. More recently, a fasting Algeria played a strong World Cup game against winners-to-be Germany. People thirst for confidence, not water. Ramadan is a reminder that personal excess breeds anxiety. Consumerism's advertising immerses us in false dependence. Ramadan is the gentle reaffirmation you send yourself that, yes, you can do more with less.

How?

At this point, the how is more of a logistical appendix, but this year's approach was particularly successful. Each year, Ramadan's approach gets me nervous. No matter how many times I fast, despite having survived and thrived not one year ago, I still get skittish at the thought of it. I focus in on the circumstances new to the year, and can't help tweaking my design.

Everyone has different lives and schedules, but my Ramadan unfolds in three phases:

Phase 1: Just make it through in one piece. The first 4-5 days.
Phase 2: Requires a conscious and concerted effort. The middle twenty days or so.
Phase 3: The fast is the new normal. Usually just the last few days of the month.

My Ramadan technique goes into effect from day 1. It can be a rough transition, involving some falling asleep while eating cereal, but the long-day summer technique has been perfected over years. Granted, its design leans on the unique schedule afforded a young software engineer. Not everyone can switch away from a standard work-a-day-sleep-at-night schedule. The median practicing Western Muslim probably approaches Ramadan like this:

Get to work at 9am.
Work til 5pm.
Get home at 6pm. Cook, clean, tend to kids.
Eat at 9pm.
Sleep around midnight.
Wake up before 4am, eat again.
Sleep until 6-8am.

Straightforward enough, but far from optimal. There's no period of sleep longer than 4 hours, which leaves my energy on a different valence altogether. For the last three years, I've improved on the naïve solution, by switching to a bimodal sleep schedule:

Get to work around 11am.
Skip lunch, hit the books til 5-6pm.
Get home, take a long nap at 7pm. This last bit would just be clockwatching anyways.
Wake up at 9pm. Dinner for breakfast!
Read, write, and code for the next 6 hours.
3:45am. Eat breakfast, taking care not to fall asleep.
Sleep through til 10am and repeat.

It's a fun change of pace. If the workday seems short, keep in mind that there are no meal or snack breaks, so it evens out. Similarly, there's a lot of new time discovered in these quiet, contemplative nights. Overall my energy, while restricted, stays predictable and manageable. I'm no Hakeem Olajuwon or Algerian footballist, but this year I managed to continue to bike everywhere, several times riding 6 to 15 miles per day. Other innovations this year have included playing violin to stay awake and just eating a small bowl of raisin bran for breakfast. Eating less is unintuitive, but I wake up less thirsty than trying to cram in more calories, and hunger is easier to manage than thirst. Oh, and bubble water.

Sometimes during the day I'd find myself impatient, checking the calendar to see how many days are left. But just as many times at night I've caught myself lamenting the quickness with which my split days have slid past. With Eid-ul-Fitr right around the corner, I must admit I am pleased with the special satisfaction brought by another year, another fast well designed.

Colophon

2015-05-01T07:00:00Z

Most blogs, like this one, are reverse-chronological, causing the first post to appear last in the archive. This convention makes a colophon the King's Pawn Game of web authorship; there's no better place to showcase certain implementation details than the first post of a blog.

This site is generated with Chert¹, an open-source static site generator built with Python, Markdown², ashes, pygments, and YAML. Chert is named for a very common fine-grained sedimentary rock, often referred to as flint, which has been of critical use to firestarters through the ages.

(Keep an eye out for a forthcoming, longer entry on why I built Chert and what makes it different.)

English pronunciation rhymes with dirt, maintainer/Farsi pronunciation: chair with a t at the end. ↩
Enhanced Markdown, including support for footnotes, definition lists, and tables of contents. ↩