Open-Source Projects

I've had a lot of ideas over the years. Lucky as I am to be a programmer, I've been able to realize many of them in the form of open-source projects. Looking back, I'm pleased with the conversion ratio, and I'm more sure than ever that the best is yet to come.

In the meantime, here is a summary and status for each of my current projects, ordered by some combination of popularity, age, and personal utility. While the most up-to-date information is on my GitHub repositories page, included here are details about the background and current state of each project.

Don't mind the odd names; you don't need to be a geologist to appreciate all this free software!

Contents

glom

Nested data exists everywhere in the programming world these days, from document databases to web APIs. I needed a way of working with this structured and semi-structured data that was not a hack. I wrote about it here, and gave a lightning talk about it at PyCon 2018. I have never had a library go from scratch to production to release so quickly, nor have I ever been quite this happy, post-release to be one of my own users.

ZeroVer

Continuing my research into one of my favorite topics, software versioning, ZeroVer (aka 0ver) is a tongue-in-cheek versioning scheme where projects stay version 0.x. The joke was pretty well-received, generating some laughs and discussion, with only minimal Poe's Law collateral damage.

Particularly good domains on this one (zerover.org and 0ver.org).

Face

After working on URLs and Hyperlink, I had a pretty major realization. Despite all our best intentions and standards, the software world is mostly held together by conventions. Most practical implementations out there pick and choose which conventions they want to support, and drop the rest, even if they are part of a standard.

Well, software does not get more conventional than command-line interfaces. Sick and tired of problems with argparse and click, I threw my hat into the ring with this entry. I have used it for Face and Pocket Protector, and am very excited to extend it and use it for more.

Pocket Protector

Security is a tough topic for teams of all sizes, and secrets management can present a challenge for projects, big and small. Drawing on my team's experiences developing similar solutions at PayPal, Pocket Protector provides a scalable, serverless secrets management system for small-to-medium-sized teams. We use it extensively and successfully at my office, to the point that I open-sourced it, and even gave a (poorly-recorded-by-me) talk on it.

Hyperlink provides a pure-Python implementation of immutable URLs. Based on RFC 3986 and 3987, the Hyperlink URL makes working with both URIs and IRIs easy.

Hyperlink is now part of the Hyper project, but derives from Boltons and Twisted, which now depends on Hyperlink for its URL functionality. Hyperlink is tested against Python 2.7, 3.4, 3.5, 3.6, and PyPy.

Full documentation is available on Read the Docs.

CalVer (Calendar Versioning)

Partly an essay on software practices, partly a response to SemVer, and most importantly a landing page for its users, calver.org documents the pattern of including calendar dates in a project's version. I think a lot about software versioning, and have become a strong proponent of CalVer in many if not most contexts.

Boltons

Boltons is an expansive collection of 160+ Python types and functions I've badly missed from the Python standard library. It is also my most popular project, and certainly the most mature in terms of being documented, tested, and officially released.

I actively maintain Boltons and encourage you to take a look at the README and CHANGELOG and let me know if you have any questions or feature requests.

Ashes

Ashes is a small and scrutable HTML templating library with a practical, language-agnostic featureset. Ashes achieves this by implementing virtually all of the Dust templating language in a single, easily-embeddable Python file. It is tested on Python 2 and 3 and used in production by countless projects, including Clastic. Documentation is underway, and I usually refer people to the documentation of the complementary frontend implementation.

Clastic

Clastic is a web site and service microframework built for high development productivity and low runtime overhead. By eschewing global state, Clastic is easy to reason about and run with all concurrency models. It also uses Werkzeug's web primitives, so it's well-documented and widely understood.

I created Clastic in response to several shortcomings I encountered with Django and Flask. I used it to build ETAVTA and some Hatnote projects, and gradually developers at PayPal started using it, too. Billions of requests later it continues to fulfill its primary goals without issue and unlike any other service framework. There are still several TODO items, but seeing as it works so well, development is occasional, at least until I get the time to write up a public release announcement.

Chert

Chert is the static site generation system designed to fit the workflow of modern digital written media. In addition to the standard Markdown/YAML syntax and RSS/Atom feeds, Chert supports automatic table of contents generation and structured list entry formats. Eat your heart out, Buzzfeed.

Unlike most othe projects on this page, I did not intend Chert to be particularly innovative or generally useful. That said, a lot of people showed interest, maybe it was the colophon?

EspyMetrics

A Python runtime analytics server, written/released as a vehicle for examples used in my Enterprise Software with Python project, published by O'Reilly.

It includes a design doc, fully-functioning server (which my team ended up using for a little over a year at PayPal), and even a code review redressing some antipatterns in Python development.

SuPPort

SuPPort is PayPal's Python infrastructure minus all the PayPalisms. I founded PayPal's Python team in 2011 and open-sourced SuPPort in 2015. Over those 4 years, SuPPort's architecture evolved to achieve six nines reliability while serving billions of production-critical PayPal requests.

SuPPort's enterprise elements may make it heavyweight for many projects, but I still refer to and recommend it as a rich resource of codified networking and robustness knowledge. It combines and builds on several of the other projects listed on this page.

Many patterns and components from SuPPort ended up back in core gevent and used by other gevent users, achieving our original goal of releasing a reference framework.

Sky

Sky was a developer tool designed to enable devs to build modern Python applications and microservices, with built-in support for pip packages, Anaconda dependencies, Docker packaging, and DCOS/Kubernetes deployment.

Using sky, a small team of shopkick developers ported almost half of shopkick services off of a Python 2.6 monolithic development and release process in just a few months. I touched on sky usage in my packaging talk at AnacondaCon 2018.

Lithoxyl

Lithoxyl is a full-featured instrumentation framework encompassing structured logging and statistical profiling.

Too often application instrumentation is forgotten completely or relegated to the afterthought of adding a few hasty logging statements. Lithoxyl aims to fix that by being lightweight and useful from the first stages of development through to deployment.

Lithoxyl started after discovering major inefficiencies in Python's built-in logging module, a hasty port of Java's Log4J. Rather than engineering a better implementation of an outdated design, Lithoxyl is an inherently more powerful system that is lower-overhead and higher-performance by nature.

Lithoxyl is used by several of my projects and documentation is underway.

Strata

Strata is a holistic configuration framework. Strata came into existence to fill the need of complex applications that accept inputs from a variety of sources and need to resolve that into a consistent, predictable configuration. For example, even basic applications read: command line arguments, configuration file, and environment variables. On top of that, there's almost always a default value for a particular field.

Strata solves this with "layers", each one providing variables from that domain. This way precedence is always clear. For instance, a developer can decide that command line arguments always override configuration file values, and when neither exists, there is a fallback default value. Once Strata performs the configuration load procedure, the resulting configuration object keeps track of which values came from where.

I'm still very much in love with the idea of Strata and would love to pick up work on it again. If you'd like to participate, please get in touch, otherwise it will have to wait for some other projects to process through the queue.

Cronfed

Cronfed is a tool for monitoring basic batch jobs, or any other cron-based scheduled commands. It achieves this by parsing a given mailbox and turning it into an RSS feed, which can then be monitored with your browser, feedreader or other RSS-compatible service (such as IFTTT).

I use it literally every day to monitor my Hatnote projects. Special thanks to Mark Williams for all his help on this.

ETAVTA

ETAVTA is my answer to the Valley Transit Authority's (VTA) complicated light rail schedule and Google Maps' inaccurate estimates. It computes a full weeklong schedule for all trains and tells you the next five trains going north and south at any of the 62 VTA stations. It's pretty simple, and serves as the earliest example of how to use Clastic.

Hematite

Hematite is a set of HTTP primitives for server and client use. It's tough creating a symmetrical abstraction that is practical, so while I use it in some projects, it's not production-ready yet.

Tectonic

Tectonic is a process management and IPC framework built on nothing but pure Python, pure POSIX, and pure potential. It's in very early phases, pending work queue clearance.

PythonDoesBlog

Originally created as platform for Python Does What?! (PDW), PythonDoesBlog is a one-of-a-kind blogging engine, specifically for blogs about Python.

Posts are written as Python modules. Text is ReStructuredText in docstrings, and the remainder of the code is highlighted. There was also special docstring handling to make sure that the module executed as expected before publishing.

PythonDoesBlog still works, and its partial results are still visible at pythondoeswhat.com, but as the PDW topics became more complex, we were writing cases that would crash the interpreter and do other bad things that PythonDoesBlog could never support.

I still enjoy scoffing at Python static site generators (including this one), "Oh, your blog post is rendered with Python? That's cool. My blog posts are Python."

Other projects

Affiliated projects

These are projects close to my heart, projects I'm proud to have helped with.