John Feminella

Pivotal

Tue, 04 Aug 2015 06:00:00 -0400

I like to think of myself as a relatively curious person, and as a professional who builds software, that often has a lot of benefits. But one big downside is that I have a lot of itches to scratch, and I’ve never been able to satisfy them by doing just one thing. I love what our team has built at UpHex and the work we’re doing to make analytics monitoring easy for businesses everywhere. I wanted to continue using that expertise in a way that was bigger than any one company, but I wasn’t sure how to do it.

I recently caught up with a friend and colleague at Pivotal over lunch, and was astounded to hear about some of the things they’re working on. While I can’t disclose all of the internal details of what’s happening under the covers, there’s already a lot of public evidence that Pivotal is an engineering powerhouse that’s building incredible things. They are primary contributors to some of the biggest open-source enterprise projects, like Spring, Cloud Foundry, and Geode. And they provide one of the core infrastructure pieces of many modern software projects: Pivotal Tracker.

Overall, they’re a startup that’s serious about a business strategy with open source at its core. That’s something almost no other company can say, and I was impressed by the depth of the commitment there. Even more importantly, Pivotal’s jam-packed full of people smarter than I am. Everyone learns differently, but I do it primarily by immersing myself in folks that are more experienced than I am, and then immersing myself in a difficult problem while surrounded by those folks. Pivotal looked like a great fit from this perspective.

After learning more about each other and explaining our goals, Pivotal asked if I’d consider becoming an advisory field engineer. I loved the idea of combining strategic, high-level exploration of business needs with practical, real-world engineering solutions, and it was an instant yes for me.

So, effective August 1, 2015, I’ve joined Pivotal. Of course, I’ll continue to be a principal in UpHex with my wonderful cofounder Bradley Kipp, who is both a marvelous human being and a whip-smart businessperson. I’m looking forward to more new and exciting adventures with both companies!

Comprehensive Technologist's Guide to Charlottesville for Prospective Citizens

Wed, 01 Jul 2015 06:00:00 -0400

I frequently get questions from my other technologist friends in SF, Boulder, NY, and elsewhere about what it’s like in Charlottesville. I’ve lived here for a little over a decade now, and while New York will always be my home, Charlottesville has many things to recommend it.

The technology profession has the great advantage of not requiring us to be tethered to a specific physical location; if you’ve got a laptop and Internet, chances are good you can work from anywhere. So if you can work anywhere, does it make sense to put this city on your shortlist? I think the answer is yes.

It’s difficult to describe the advantages and disadvantages of Charlottesville succinctly, but in this post I’d like to give as good a summary as I can. What follows are some of the notable qualitative and quantitative highlights of this interesting city from the perspective of a technologist.

Brief local history

Serpentine walls at the University of Virginia. Source.

Charlottesville was founded in 1762 to serve as the seat of Albemarle County and named after Queen Charlotte, the wife of King George III, the monarch whose various taxes and slights against colonial Americans eventually provoked the Revolutionary War. It sat along a major highway, Three Notch’d Road, that led from the state capitol of Richmond to the Great Appalachian Valley; today, the cross-country Interstate 64 highway follows in its footsteps.

In 1819, Thomas Jefferson founded what is now one of the highest-rated national public universities, the University of Virginia. UVA is a major nexus of activity in the regional startup and business market, and it’s the largest employer the area. Among many other interesting people, UVA has matriculated the poet Edgar Allen Poe, seven NASA astronauts, artist Georgia O’Keefe, writer and comedian Tina Fey, journalist Katie Couric, and U.S. President Woodrow Wilson.

In the 20th century, the area’s culture and commerce gradually began to rise around three big areas: agriculture and food, technology, and healthcare. The rise of the University of Virginia as an affordable, public, high-quality research institution also helped attract more citizens. During the 1970s through 1990s, the City undertook a series of ambitious public-works renovations to build the Downtown Mall, which today is one of the largest pedestrian malls in the US.

What it’s like living here

Charlottesville is somewhere in between a large town and a small city, with a population of about 50,000 people spread over 10 square miles. Despite the modest scale, it seems to have the influence, amenities, and prestige of cities many times its size.

Charlottesville consistently wins accolades like “best place to live in America”, “happiest city in America”, and “best college town for people who aren’t in college”. It’s pretty nice here!

People

It’s a challenge to generalize broadly about Charlottesville residents. They tend to be from all walks of life, status, and birthplace. While there’s a sizable contingent of people who lived and grew up nearby, the majority were attracted by jobs or the University. The median income, wealth, and education for Charlottesville residents is above the national average in all cases, and Virginia as a state scores well on these metrics relative to other states. Similarly, most people who live in Virginia were not born in Virginia, and that’s especially true in Charlottesville.

Charlottesville is a predominantly liberal city situated in a broader rural region, Virginia’s Fifth Congressional District, that skews conservative. This is evident in a lot of election results, for example, where Charlottesville will tend to vote for the more progressive candidate in a contested race, while the broader district votes for the more conservative candidate.

Climate and location

Blue Ridge Mountains in the fall. The scenic Skyline Drive winds through the trees.

Charlottesville forms a triangle with Washington, DC two hours away to the northeast and Richmond, VA one hour away to the east. Outside the city, the scenery rapidly becomes more rural and less developed, with mountains to the west and forests, farms, and vineyards to the south and east. The Blue Ridge Mountains and skiing at Wintergreen and the surrounding area are thirty minutes by car; Atlantic beaches are three hours.

Weather is very pleasant. There’s about 220 days of sunshine per year, so it’s much more likely to be sunny than not, and there’s usually a light west-to-east wind coming from the mountains.

property	San Francisco, CA	New York, NY	Washington, DC	Charlottesville, VA
annual total precipitation (in)	20.7	49.9	39.7	46.7
annual snowfall (in)	< 1	25.1	14.5	16.3
annual sunshine (days)	259	212	203	219
mean February humidity (%)	84	60	61	66
mean February temperature (°F)	60	42	47	51
mean August humidity (%)	75	66	69	72
mean August temperature (°F)	68	80	87	77
annual days above 85°F highs	3	15	37	42
annual days below 30°F lows	0	78	84	113

Summers are warm and sometimes on the humid side, albeit milder than Virginia as a whole. Typical summer temperatures are in the high 70s. Very brief summer thunderstorms are common, and then it’s quickly back to sunshine again.

Winters are milder, with infrequent snows. Over the last 20 years, snowfall has varied considerably; a typical year sees zero to three snowfalls of 6” or more each. Occasionally there is zero or minimal snowfall in a year.

Extreme weather is rare. Occasionally Atlantic hurricanes make landfall near Charlottesville, but it’s far enough inland that the brunt of the storms are mitigated. On very rare occasions, Charlottesville has been subjected to powerful storms known as derechos that inflict substantial damage.

Things to do

Food: Charlottesville is chock-full of highly-rated excellent restaurants and eateries that place a premium on high-quality local ingredients. Local restaurants run the range from 3 AM burger joints to world-class eateries, and have routinely garnered high praise from national reviewers. A few local venues and their chefs have won awards at the national or international level, including Hamilton’s, Peter Chang’s China Grill, Ivy Inn, Citizen Burger, Fossett’s, and Bodo’s Bagels.

Every Saturday morning is the City Market, a pop-up bazaar that lasts a few hours and frequented by street vendors, farmers, potters, artisans, and so on. It’s a great way to sample excellent local food. Charlottesville has a large faction of specialty coffee enthusiasts; Mudhouse, Java Java, Grit Cafe, and several other locations all make top-quality brews.
Shopping: Charlottesville is home to three large shopping districts: Fashion Square Mall, Barracks Road Shopping Center, and the Shops at Stonefield. All have large anchor department-store tenants and chain eateries. Grocery needs are supplied by Whole Foods, Trader Joe’s, Harris Teeter, Food Lion, Kroger, and a number of other chains. A Costco will open at the end of 2015 in Stonefield.
Entertainment: Charlottesville is home to a wide range of venues; John Paul Jones Arena (15,000 people), the Paramount Theater (1,000 people), and the Pavilion (5,000 people) are the biggest, and they consistently draw performers both large and small. During the off season, the University of Virginia’s Scott Stadium also hosts the bigger events that are part of large national tours (e.g., U2, Rolling Stones). There’s also a fourteen-screen IMAX cineplex, a 20,000 square foot indoor trampoline park, a renovated drive-in movie theater, and various other interesting spots.
Culture: The Charlottesville area has a number of museums, public classes, and festivals. The nationally popular Virginia Film Festival and Virginia Festival of the Book are both hosted in Charlottesville, as is the Tom Tom Founders Fest. The museums and historic sites of Thomas Jefferson’s Monticello, Ash Lawn-Highland, the Virginia Museum of Fine Arts, Fan Mountain Astronomical Observatory, and Kluge-Ruhe Aboriginal Art Museum are all within an hour’s drive. Public adult-education and general-interest classes are offered by Piedmont Virginia Community College, the University of Virginia, and the McGuffey Art Center.
Breweries: The craft brewing scene is in full swing in and around Charlottesville; a few breweries have national acclaim. Because of a quirk in the alcohol regulations of Virginia, many breweries that don’t serve food let you bring your own food in (as long as you buy their beer).

This results in a number of such breweries being interesting local meetup spots for friends or groups; you bring your own pizza and the beer’s already waiting for you. The most convenient ones to Charlottesville are South Street Brewery, Three Notch’d, and Champion Brewery. Farther out are Devil’s Backbone, Wild Wolf, and Blue Mountain Brewery.
Wineries: There are several dozen wineries with tastes and portfolios to suit almost anyone. Charlottesville also has two area cideries, Albemarle Cider Works and Castle Hill Cider, that put out consistently great offerings.
Nightlife: There’s a very lively bar scene on the Downtown Mall, particularly when the free live concert series Fridays After Five is in full swing. During the summer, vineyards often have summer concerts and picnic nights. For a different sort of nightlife, the University’s Observatory offers starlight viewings every other Monday. Options for dance clubs are minimal; R2 and Escafe (LGBTQ-friendly) are the biggest choices.
Nature: Charlottesville is next door to Shenandoah National Park, a sprawling rectangle of over 300 square miles that stretches along the Blue Ridge Mountains. There are dozens of great trails, but set aside time for the strenuous hike and rock scrambles of an Old Rag Mountain day hike. Charlottesville also the Rivanna Trail, an extensive network of hiking and biking paths that almost-loops around the city’s perimeter.
Recreational sports: Baseball, kickball, and soccer are popular sports in the city’s social leagues, which range from very competitive to laid-back. There are also sports leagues of a less traditional kind, including kickball, ultimate frisbee, and flip-cup.

Internet

Perhaps the most important reference point for prospective technological citizens: Internet service is pretty good. It’s primarily provided by three mutual competitors: Ting Gigabit (1,000/1,000 Mb/s, no hard caps), Comcast (105/10 Mb/s, no hard caps), and Embarq DSL (10/1 Mb/s, no hard caps). Ting has a limited rollout to about 3,000 homes at the moment, but is expected to reach about 10,000 homes by the end of 2015, mostly in the North Downtown, Martha Jefferson, Locust Grove, and Belmont districts.

Cost of living

Charlottesville’s Downtown Mall on a quiet winter evening.

Charlottesville has an extremely competitive cost of living compared to a number of major cities. This is especially true when measured against the nearest metropolitan counterpart, the Northern Virginia pan-urban area, but holds up for several different metrics.

property (national = 1×)	San Francisco, CA	New York, NY	Washington, DC	Charlottesville, VA
goods and services	1.1×	1.5×	1.0×	1.1×
groceries	1.2×	1.5×	1.1×	1.0×
health care	1.1×	1.1×	1.0×	1.0×
housing	3.4×	2.3×	2.1×	1.4×
transportation	1.1×	1.3×	1.0×	1.0×
utilities	1.0×	1.4×	1.0×	0.9×
overall cost of living	1.8×	1.7×	1.3×	1.1×

Sources: AreaVibes cost of living summaries rounded to two significant figures, US Census Bureau ACS PUMS.

Quality of life

Charlottesville’s general quality of life is excellent, especially given the low cost of living. About 90% of Charlottesville residents rated it an “excellent” or “good” place to live, according to the 2014 National Community Survey.

property	San Francisco, CA	New York, NY	Washington, DC	Charlottesville, VA
weekly commute time (hours)	4:57	6:18	4:49	1:26
monthly rent, 1 BR apartment	$3,500	$3,000	$2,000	$1,100
marginal state income tax rate at $100,000	9.3%	6.7%	8.5%	5.8%
general sales tax rate	8.8%	8.9%	5.8%	5.3%
walkability score at median rent	71	89	96	95

Major concerns

While most residents give Charlottesville high marks across the board relative to similar cities, it’s not without complaints. When Charlottesville citizens express concerns, the majority of them fall into one of three big areas:

Traffic and parking. Peak traffic spikes very high during specific weekends – football games, the University of Virginia graduation and move-in weeks, and major John Paul Jones Arena or Pavilion concerts. While Charlottesville has a rush hour, it doesn’t last very long, usually about 30 to 45 minutes in the mornings and evenings. Free parking is in very limited supply near the Downtown Mall, especially during big events, when you’ll almost always have to use one of the nearby parking garages.
Affordable housing and homelessness. Although Charlottesville has low overall cost of living, it’s much higher than the surrounding rural areas, and there aren’t a lot of entry-level or mid-level jobs for people outside of white-collar positions. This has created steady upward pressure on rents, which makes it difficult for younger, less-skilled, or lower-paid workers to establish a foothold. This, in turn, exacerbates an existing homelessness problem. Additionally, the city also prefers a “slow burn” approach to growth, and actively tries to encourage dense construction that favors family-size units within city limits – this makes single-person housing more expensive, and/or forces them to live with roommates in larger units.
Safety. Charlottesville’s crime rate is about 10% below than the national average on a per-capita basis. However, a number of high-profile crime cases have hit Charlottesville in the past few years, drawing unwanted national media coverage. Some of these come from reported incidents at the University of Virginia, which generates a higher than average quantity of nonviolent crimes like theft, vandalism, and public drunkenness. The overall crime rate is also on the decline, having steadily trended downwards every year in the last two decades. 96% of Charlottesville residents say they feel safe in their neighborhoods.

Sources: Zumper, Tax Foundation, US Census Bureau ACS PUMS, National Community Survey.

Challenges

There are also some secondary problems that may be of particular interest to younger professionals moving here:

Dating is tough. A significant fraction of the twentysomething dating pool is at the University. That means that there may be an implicit time bomb attached to any relationships you strike up if you date a student: once your prospective partner’s graduation date approaches, there’s a choice to make.
No coherent vision from the City. A secondary complaint from a lot of professionals is that the city doesn’t offer any kind of idea of what the future is like for technology. The city doesn’t directly fund any major technology initiatives, doesn’t particularly encourage economic development of technology-industry firms any more than others, and so on. The absence of any signals about the local government’s intent, coupled with limited opportunities for advancement in tech fields that aren’t directly related to or supporting software development, can make staying in Charlottesville a career-hurting move compared to a city more favorably inclined to technologists.

Technology jobs and community

Charlottesville is a regional hub for technology jobs, and employs approximately 13,000 people in the technical, software, and hardware domains. These positions across a range of different companies, from multinational conglomerates like SRA and Northrop Grumman to small, newly-funded startup outfits.

Major technology employers

The most visible Charlottesville employer is the University of Virginia and the associated University of Virginia Health System. Together they employ over 15,000 people.

Other interesting technology employers of note include (but are certainly not limited to):

ArcheMedX, healthcare informatics and e-learning provider for clinical professionals.
Borrowed and Blue, wedding marketplace startup connecting vendors with couples.
CCRi, software engineering and data science company.
CFA Institute, global nonprofit for investment professionals that provides the CFA accreditation.
Crutchfield, national audio electronics retailer that was one of the first Internet retailers.
Elder Research, a data science company whose CEO wrote one of the definitive texts on data mining.
FireEye, public company offering network threat forensics and malware protection.
General Electric Intelligent Platforms, industrial engineering venture of General Electric.
Open Source Connections, search relevance
PsiKick, designs ultra-low-power wireless chips with widespread industrial and commercial applications.
Quantitative Investment Management, one of the world’s largest managed-futures hedge funds.
Relay Foods, online grocery delivery and shopping service.
RoomKey, hotel search engine operated as a joint venture among several hotel chains.
SNL Financial, business intelligence and financial information provider.
Ting, nascent gigabit Internet and mobile provider deploying fiber into Charlottesville.
UpHex, analytics monitoring and reporting startup for businesses.
VividCortex, SaaS provider of performance management tools purpose-built for databases.
WillowTree Apps, small mobile-apps development firm and winner of multiple Webby awards.

Job diversity

Here’s a list of statistically-significant phrases for the Charlottesville job market – phrases which are more likely to appear in a Charlottesville technology job posting than the nationwide average.

technology phrase/keyword	likelihood above national mean
Epic	█████████████ 13 ×
robotics	███████████ 11 ×
sensors	██████████ 10 ×
biomedical	████████ 7 ×
Ruby	████▌ 4.5 ×
Go	███▌ 3.5 ×
Python	██▌ 2.5 ×
.NET	██ 2 ×

Notice it’s top-heavy on sensors and medical applications; the top-ranking term, Epic, is a healthcare software provider. Charlottesville has an outsize funding presence in biomedical startups and applications, owing to the influence of the University of Virginia in this domain.

Startups

The startup community in Charlottesville is still very nascent but steadily growing over the last decade or so. There have been some outsize superstars, like Reddit; Steve Huffman and Alexis Ohanian met at UVA and built the site that would eventually become Reddit here in Charlottesville. There have also been some spectacular failures from the 1999 dotcom bubble, like Value America.

Official statistics aren’t kept by the city on this, but my estimate is that there are at least 100 startup businesses employing a total of about 1,000 people, where “startup” means “a company whose primary goal is to search for a business model”. There are a number of demonstrably successful startups with good traction and broader publicity outside the Virginia area, including Relay Foods, Borrowed and Blue, and VividCortex.

Capital sources

While there is some capital for startups available, Charlottesville doesn’t have a large capital base. Still, it’s possible to fund a compelling business idea without needing to extend a hand to VCs that might be further afield, but the bar is somewhat higher than it might be in Silicon Valley or another hub. Startup funding comes from one of four primary sources:

Angel/seed funds. There are a handful of local seed-stage funds that provide capital to launch businesses, but the quality and expertise they can bring to the table vary substantially.
Individual angels. A small number of high-net-worth individuals who are also sophisticated investors play a significant role in startup funding in the area. Warm introductions, a reputation that precedes you, or an idea with strong traction are prerequisites for getting in the door.
Family offices. Assets controlled by family wealth-management offices are occasionally a local source for new startup capital. One office in particular has done a number of technology deals.
Government entities. The CIT GAP Funds, which are funded with state income tax revenue and run as a quasi-private institution, serve as a de facto form of seed capital. As of 2014, CIT has invested about $13M across about 100 portfolio companies, of which about 15 have been acquired or gone public. Student entrepreneurs have some additional options in the form of seed funds at the University, like the Virginia Venture Fund.

Meetups

Charlottesville has a relatively diverse technical community. Local or regional meetups are hosted for a number of major languages – Ruby, Python, .NET, and so on. Charlottesville technologists seem to be polyglots more often than not, perhaps motivated by the diversity of the technical jobs in the area. A general software-technologist group focusing on software craftspersonship, beCraft, is well-attended. There are also less structured and more social meetups like First Wednesdays and Charlottesville Women in Tech.

Charlottesville doesn’t have a code school or a boot camp, but there are small group-learning classes in various contexts, like that offered by Rails School or the frequent meetups hosted by the Center for Open Science. Our neighboring city, Richmond, is larger and offers many more options for meetups within a reasonable drive.

Conferences

Twice a year there are two larger software conferences: beSwarm, a day-long informal unconference focused on showing things you’ve built or learned, and beCamp, a more general two-day technology unconference that covers broader topics. Other technology conferences also take place in Charlottesville, like edUi, which targets web professionals working in educational environments.

Transportation

Charlottesville residents enjoy some of the best commutes in the country; the mean is below 20 minutes, and 28% of Charlottesville’s labor force walks to work. Getting around town is pretty easy.

Traffic

Routes in blue include Charlottesville’s highest-volume roads.

The major arteries through and around the city are West Main Street, Route 250, Interstate 64, and Route 29. Traffic in Charlottesville is light on average, but most problematic on these arteries. If you live and work somewhere in the boundaries of the City, you can expect your commute to be 15 minutes or less on a typical day if you drive, or under 40 minutes if you walk and use public transport. Rush hours are brief but intense, usually dissipating in under an hour.

Walking

Charlottesville is very pedestrian-friendly; crosswalks are frequent, clearly marked, and well-maintained. An eight-block district of the city, the Downtown Mall, is pedestrians-only; vehicles are not permitted except at two crossings. In general, police are not shy about ticketing cars for failing to yield to pedestrians, which carries a fine of up to $500 and is a moving violation.

Charlottesville is one of a small handful of cities in the US to receive a “Gold Level” award from Walk Friendly Communities, which rates cities according to their walkability.

In the city proper, you can expect sidewalks or mixed-use paths on most major roads and in suburban neighborhoods, with the exception of Route 250 and Interstate 64, which are both high-traffic corridors.

Public transportation

Charlottesville’s public transport consists of a network of ten paid bus lines and a free trolley that travels between the University and the Downtown Mall districts. Fares for the bus lines are usually cheap:

$0.75 per trip
$1.50 day pass, unlimited rides for the day
$20.00 monthly pass, unlimited rides for the month

Coverage is generally excellent; most of the city is within five blocks of a bus station. The downside is that some routes are very long and it can be inconvenient if you get on at a point which is far away from your destination. Some routes have a loop which takes 90 minutes or more to complete, meaning it could take you that long to get where you want to go.

Private transportation

Uber is available in Charlottesville, especially on weekend nights and for traveling to and from the airport. Surge pricing generally doesn’t kick in except during right around 2:00 AM, the latest that bars can legally close in Virginia, or during big events like football games and UVA’s graduation. Uber’s Charlottesville fares are:

base fare: $2.75 per trip
time fare: $0.25 per minute
distance fare: $1.50 per mile
plus a $1 Safe Rides fee

Charlottesville has a number of taxi providers. While taxi drivers must be licensed drivers by state law, there is no state or local taxi authority, and taxis are free to set their own rates.

Biking

Most of Charlottesville’s major roads (but not the highways or Interstate 64) have dedicated bike lanes. Cycling advocacy organizations have in general been pushing for more bike space, and Charlottesville scores well on various bicycle-friendly metrics. Charlottesville is rated “Silver Level” by the League of American Bicyclists.

If you prefer to stay off-road, recreational cycling on trails is widely available within a short distance from the city at Walnut Creek, Mint Springs, and Preddy Creek. Inside the city, the best options are McIntire Park and Observatory Hill.

Trains

Amtrak provides northbound service between Charlottesville (CVS) and Union Station (WAS) in Washington, DC, and southbound service towards Lynchburg, VA (LYH). Travel time is about 2.5 hours, roughly 15% faster than traveling by car without any traffic. From Union Station, Amtrak’s Acela train stops at Baltimore, Philadelphia, New York, New Haven, and Boston.

Planes

Charlottesville has a regional airport with about 200 flights per day to various destinations, of which approximately 70% are general commercial aviation (the rest are private air taxi services or military flights).

If you’re going anywhere else in the US, you’ll probably take a short flight to a nearby international airport, like Washington Dulles (IAD) or Atlanta (ATL) first. Otherwise, the airports with direct flights from Charlottesville are:

Charlotte, NC (CLT) via US Airways
Atlanta, GA (ATL) via Delta
Washington, DC (IAD) via United
Philadelphia, PA (PHL) via US Airways
Chicago, IL (ORD) via American Eagle
New York, NY (LGA) via Delta, American Eagle

You too, perhaps?

Making the choice about where to live is both difficult and deeply personal. While there are many objectively great reasons to live in Charlottesville, perhaps the biggest one for me is completely subjective: it just feels like home.

I’ve loved living here, and would be hard-pressed to trade the experience for anywhere else. I hope you’ll visit and feel the same way, too!

Acknowledgements. Thank you to:

Kyle Redinger for his excellent summary comment on the startup environment; and

Waldo Jaquith and Spencer Ingram for looking over my shoulder on this article

Any errors or omissions are entirely my own.

Finding Out What Changed During Debian/Ubuntu Package Upgrades

Fri, 22 May 2015 04:00:00 -0400

Like many Linux distributions, Debian comes with a package management and distribution system called apt (Advanced Package Tool). While apt is a very useful feature of Debian systems, upgrading packages can be a little daunting. Here’s what a recent apt-get upgrade looked like for me:

ψ sudo apt-get upgrade
[sudo] password for jxf:
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  linux-headers-generic linux-signed-generic
  linux-signed-image-generic lxc-docker postgresql-client-common
  postgresql-common
The following packages will be upgraded:
  atom libdrm-intel1 libdrm-intel1:i386 libdrm-nouveau2
  libdrm-nouveau2:i386 libdrm-radeon1 libdrm-radeon1:i386 libdrm2
  libdrm2:i386 libicu52 libtasn1-6 libtasn1-6:i386 linux-libc-dev
  oxideqt-codecs-extra steam-launcher
15 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.
Need to get 61.3 MB of archives.
Do you want to continue? [Y/n]

Notice that a number of important-looking packages changed: graphics driver middleware, libc, and the text editor I use to write this blog. But it’s hard to know whether we should install these changes or not, and what the urgency on them is.

What if the updates are critical security fixes in libc? We might want to install them immediately. Conversely, what if the new graphics driver introduced features that disrupted the framerate on our favorite games? We might want to hold off.

With the apt-get interface, we know whether something has been updated, but we don’t know what the updates are. Can we do better?

What changed?

Enter apt-listchanges, a wonderful tool by Sandro Tosi and Thadeu Lima de Souza Cascardo. Each apt package usually ships with two files that describe interesting human-readable details about the package:

a debian/changelog file that describes recent changes made to the package, usually of a technical nature
a debian/NEWS file that describes user-friendly updates about the package

For example, if we download a recent Firefox nightly package:

apt-get download firefox

and run apt-listchanges on it:

apt-listchanges --which=changelogs firefox

we’ll get this output:

firefox (40.0~a2~hg20150520r262219-0ubuntu1~umd1~trusty) trusty; urgency=medium

  * Refresh patches
    - update debian/patches/unity-menubar.patch
    - update debian/patches/dont-include-hyphenation-patterns.patch
    - update debian/patches/allow-lockPref-everywhere.patch

 -- Chris Coulson <chris.coulson@canonical.com>  Wed, 20 May 2015 17:17:16 +0100

firefox (39.0~a2~hg20150429r255253-0ubuntu1) vivid; urgency=medium

  * Refresh patches
    - update debian/patches/unity-menubar.patch
    - update debian/patches/ubuntu-ua-string-changes.patch
  * Drop patches fixed upstream
    - remove debian/patches/add-non-skia-fallback.patch
    - update debian/patches/series
  * Bundle our checkout of compare-locales in a different location, given
    that the Mozilla repo now contains a different version of it in the
    location we used previously
    - update debian/build/rules.mk
    - update debian/build/create-tarball.py

 -- Chris Coulson <chris.coulson@canonical.com>  Tue, 31 Mar 2015 19:32:34 +0100

Using apt-listchanges

It would be nice if we could get this list whenever we installed a package. Fortunately, apt-listchanges does just that for us, too!

First we’ll install the package:

sudo apt-get install apt-listchanges

After installation, there’s a configuration step which asks you how you’d like to view the changes and make some other choices; I find the pager mode is most convenient for viewing. If you want to revisit your configuration choices, you can run dpkg-reconfigure apt-listchanges.

After the configuration is done your preferences will be saved in /etc/apt/listchanges.conf.

Now that we’ve got that, let’s install a package!

sudo apt-get install firefox
[...]
Do you want to continue? [Y/n]
Reading changelogs... Done

firefox (40.0~a2~hg20150520r262219-0ubuntu1~umd1~trusty) trusty; urgency=medium
  * Refresh patches
    - update debian/patches/unity-menubar.patch
    - update debian/patches/dont-include-hyphenation-patterns.patch
    - update debian/patches/allow-lockPref-everywhere.patch

 -- Chris Coulson <chris.coulson@canonical.com>  Wed, 20 May 2015 17:17:16 +0100
apt-listchanges: Do you want to continue? [Y/n] y

Notice the second prompt for apt-listchanges; this is one of the configuration settings you can choose. Here, you’ll get a chance to review what changed before you decide to pull the trigger.

In short, apt-listchanges is an excellent addition to your Debian package-management workflow and highly recommended.

The Infinity Zoo: What is Infinity?

Fri, 24 Apr 2015 04:00:00 -0400

To see a world in a grain of sand, And Heaven in a wild flower, Hold Infinity in the palm of your hand, And Eternity in an hour.

William Blake, “Auguries of Innocence”

Infinity is an idea that has fascinated humans for a long time. To some it can seem inscrutable. For others, it symbolizes the mystical or spiritual. For mathematicians, it’s a tool that’s essential to working with and understanding the behavior of many things – although it’s been a point of strong historical controversy.

But what is infinity, exactly? It turns out there are many kinds and instances of infinity – a veritable zoo of infinities! In this post, we’ll try to do a survey of the different kinds of infinity, and explore some of the possibilities that are enabled by having it.

Actual and potential infinity

A tower in Toronto’s financial district stretches high into the evening sky. Source.

People have been grappling with the philosophy of infinity for centuries, and mathematics is no exception. The ancients, especially Indian, Greek, and Arab mathematicians, readily entertained the notion of two kinds of infinity. One kind was called potential infinity; the Greeks called it apeiron (ἄπειρον, “without end”). The other kind was called actual infinity.

Potential infinity allows a process to be repeated as many times as you would like – for example, adding a new number to a list – but no matter when you stop, the result is always finite.

Essentially, a potential infinity describes two things: a bag that holds items, and a process that adds items to the bag which can be repeated indefinitely.

You may keep adding items to the bag as long you like via the process. But you’ll never exhaust the supply of elements you can add – it has no end. If you do stop at some point, you will always have a finite number of items in the bag. In other words, the potential infinity is never realized into true infinity, so a crisis is averted.

But true, actual infinity eluded their grasp somewhat. An actual infinity, in their view, was like a bag that contains not merely many items, but infinitely many – an unlimited, unbounded number. This was a somewhat radical idea. Plato, for example, objected strongly to actual infinities – how could something be so big that it contained everything, and yet fit inside the Universe?

Actual infinity was difficult to grasp, and controversial from the beginning. It took two millenia before the notion of actual infinity as a real, useful property began to take root in mainstream mathematics.

9999 $\cdots$ problems

To see why it was so controversial, let’s reflect for a moment on the meanings of infinity.

Sometimes we use infinity to mean something like “an incomprehensibly big quantity”, like the number of stars in the sky, grains of sand on a beach, or atoms in the Universe. These are indeed very big numbers, in the sense that it would take you a long time to count them out, but they aren’t infinite. They are large – for example, the number of atoms in the Universe is about $10^{80}$ – but nevertheless they are finite. For mathematicians, the unsatisfactory imprecision of phrases like “big number” or “a long time” renders them useless for making rigorous statements about the world. This use of the word “infinite” is a meaningless one.

Another problem is that there seem to be many kinds of infinities. The single word “infinity” fails to capture the diversity and depth of these concepts, much like the word “number” fails to capture the many different kinds of numbers. Even the ancients already distinguished at least two kinds of infinities, the actual and potential infinities. There are certainly many more numbers than the kind that can be written down using only the digits zero through nine, as we’ll see later. And if infinity can be said to be a number at all, it probably a number with a conventional representation.

Finally, the abstract nature of infinity can also be difficult to connect to everyday experiences. There are a multitude of reasons why this might be so, but one candidate hypothesis is that our evolution has not required us to regularly confront infinities in order to survive. The things we interact with are typically very finite – scarce, small in quantity, usually able to be touched and held and seen. So it is perhaps not surprising that humans don’t deal with the notion of infinity well.

Yet, while problematic, these issues aren’t wholly intractable. It’s not only possible to talk about infinity, it’s a necessity for some parts of mathematics. But where can we begin?

Infinity means a lot of things

One place to start the discussion is to think about some flavors of infinity. Let’s imagine the following quantities:

the number of digits after the decimal point in $.123123123\cdots$
the value of $\sqrt{2+\sqrt{2+\sqrt{2+\cdots\strut}}}$
the length of the sequence $0, 1, 2, 3, \cdots$

Almost infinities

These are all, at a cursory glance, things that might seem to be infinite. But a closer look will reveal distinctions between them.

“Infinitely” repeating digits: $.\overline{123}$

There certainly appears to be something infinite about a string of digits that repeats forever. But notice that the number $.\overline{123}$ is equal to $\frac{123}{999}$ , which in turn can be reduced to $\frac{123}{999} = \frac{41}{333}$ .

So this apparent infinity is really just a quirk of how we represented the number, and not anything intrinsic to the number itself. For example, in base 10, $\frac{1}{3}$ is the repeating decimal $.\overline{3}_{10}$ . But in base 12 it becomes $(\frac{1}{3})_{10} = 0.4_{12}$ .

The fact that we can write a number down at all and pinpoint it precisely on the number line means that we know its exact value, so this doesn’t seem like a very interesting kind of infinity.

“Infinitely” nested structures: $ \sqrt{2+\sqrt{2+\sqrt{2+\cdots}}} $

What about $ \sqrt{2+\sqrt{2+\sqrt{2+\cdots\strut}}} $? An infinitely recursing expression sure sounds like it’s infinite. This cannot be waved away as a trick of representation, as we could with the previous example. There is a structure which must be untangled.

Can this expression have a defined value? Let’s suppose for a moment that it can, and we’ll call this value $S$ . Then

$S = \sqrt{2+\sqrt{2+\sqrt{2+\cdots\strut}}}$

When we square both sides, we get:

$S^2 = 2 + \sqrt{2+\sqrt{2+\cdots\strut}}$

But notice that this includes $S$ too:

$\begin{eqnarray} S^2 & = & 2 + \underbrace{\sqrt{2+\sqrt{2+\cdots\strut}}}_{S} \\ S^2 & = & 2 + S \\ S^2 - S - 2 & = & 0 \end{eqnarray}$

This is a quadratic equation with respect to $S$ and can be factored:

$0 = (S + 1)(S - 2)$

yielding the roots $S = \{2, -1\}$ ; if we assume that square roots can only be positive, then $S = 2$ , so

$2 = \sqrt{2+\sqrt{2+\sqrt{2+\cdots\strut}}}$

Again, our attempt at infinity has yielded a very finite number. It’s clear that merely repeating something indefinitely or recursing indefinitely doesn’t necessarily mean we’ll get something that looks like infinity.

“Infinitely” long sequences: the length of $0, 1, 2, 3, \cdots$

Our third example, the length of $0, 1, 2, 3, \cdots$ , looks more promising. Unlike our repeating-decimal example, there doesn’t seem to be an obvious way we can simply change the problem’s representation to wind up with something finite. This sequence of natural numbers increases slowly but surely, upwards forever without bound.

Is this infinity, at last? And if so, is it just the integer that goes at the end of the natural number line, as shown below?

A little thinking will show this idea can’t work. Suppose we assume that $\infty$ can be defined as “the largest natural number” and we can place it on the number line somewhere.

We know that we can always add one to a natural number $n$ to get the next natural number – its successor, $\mathcal{S}(n)$ . For example, $\mathcal{S}(496) = 496 + 1 = 497$ .
Since $\infty$ is a natural number, we can take its successor $\mathcal{S}(\infty) = \infty + 1$ .
That would create a new number $\infty' = \infty + 1$ , so $\infty' > \infty$ . But we just said $\infty$ was the largest natural number!
That’s a contradiction. So our assumption was false – an object representing “the point on the number line representing the largest natural number” is, in some sense, incompatible with the idea of a number line which can be extended indefinitely.

By the same token, we can see that infinity can’t be the largest integer, the largest rational number, the largest real number, and so on.

So the definition of infinity as “the largest natural number” doesn’t work very well. $\infty$ here seems more like it’s not on the number line at all. It’s just an extreme, unattainable limit:

Infinity and numbers: properties

Our earlier attempts at trying to wrangle infinity into a conventional number failed; infinity is too different from those numbers to be represented this way. But there’s another way to think about the infinite: we can talk about the properties we want to describe.

One property of infinity that seems worth capturing is the idea of size: infinity seems like it should be indefinitely large, larger than anything else. Another is the idea of ordering: infinity seems like it should come last, after everything else.

We can represent these two properties by using two special kinds of numbers: cardinal and ordinal numbers.

Cardinal numbers

Imagine we have a set, a mathematical collection of unique elements. One property of sets we’re often interested in measuring is their size, or cardinality. So we’d like to ask the question, “how many objects are there in this set?”. The answer to that question is always a cardinal number.

The cardinality $\vert S \vert$ of a finite set $S$ is the number of items in the set. For instance, if $S = \{\mathtt{blue, sunshine, puppy}\}$ then its cardinality $\vert S \vert = 3$ . Likewise the cardinality of the empty set $\vert \{\} \vert = 0$ .

When we count things, we’re labeling each item with a specific name from the set of cardinal numbers: “one”, “two”, “three”, and so on. This matches an item in the set we’re counting with the name of a cardinal number. The cardinal number we stop at after matching all the items is the size of the set.

When we ask for the cardinality of an infinite set, like the natural numbers $\mathbb{N}$ , the same rules apply: we get the cardinality by matching all the items in the size of the set. The answer is also a cardinal number.

Cardinality of infinite sets

What happens when we ask about the cardinality of an infinite set – say, the natural numbers $\mathbb{N}$ ? Since this set has an infinite number of elements, the cardinality we want to assign to it can’t be a natural number. (If it were, then we’d just wind up back in the number-line problem we saw before.) Instead, it has to be a different, special kind of number.

The special numbers that mathematicians use to refer to the cardinalities of infinite sets are called the aleph numbers, and they belong to the set $\aleph = \{\aleph_{0}, \aleph_{1}, \aleph_{2}, \cdots\}$ . The first and smallest member of this set is $\aleph_{0} = \vert\mathbb{N}\vert$ : this is, in effect, the smallest kind of infinity.

The way to determine cardinality of a set relative to another is to match the two sets up against each other. If we are able to pair each element of one set with another, with no unpaired elements, then we know that the sets have the same cardinality. This kind of pairing relationship is called a bijection. For example, if $S = \{\mathtt{blue, sunshine, puppy}\}$ and $T = \{\mathtt{apple, orange, banana}\}$ , then we can show $\vert S \vert = \vert T \vert$ by noticing that

$S \times T = \{(\mathtt{blue}, \mathtt{apple}), (\mathtt{sunshine}, \mathtt{orange}), (\mathtt{banana}, \mathtt{puppy})\}$

A counterintuitive result of this is that infinite sets which seem completely different in size can have the same cardinality, because we can often find a bijection between them. For example, the even natural numbers $\{0, 2, 4, 6, \cdots\}$ have the same cardinality as all natural numbers $\{0, 1, 2, 3, \cdots\}$ .

At first, this may be impossible to believe: how can a set with some of its elements missing be the same size as the original set? Such are the consequences of being infinite: things no longer behave the way you might expect! Objects certainly don’t behave this way in the real world.

To see why it’s true that the even natural numbers $2\mathbb{N}$ have the same cardinality as the natural numbers $\mathbb{N}$ , imagine that our mapping $\mathbb{N} \overset{f}{\mapsto} \mathbb{N}$ is the function $f(n) = 2n$ . Now we’ve defined a relationship between every natural number and every even number:

If we can define a bijection between any two sets, like we do here, we know they must be the same cardinality.

Since we were able to pair each element of one set with the other with no left-over elements, the sets must have the same cardinality. And since $\vert \mathbb{N} \vert = \aleph_{0}$ , then it must be true that $\vert 2\mathbb{N} \vert = \aleph_{0}$ .

A similar approach applies for almost any other infinite subset of the natural numbers you can name. In general we can just set $f(n)$ to be “the $n$ ^th element of the other set”, and we can get a bijection.

Ordinal numbers

Whereas cardinality and cardinal numbers describe the size of sets, we can also talk about the order of sets, too. Ordinal numbers describe a particular kind of ordering of the elements in sets, called a well-ordering.

For an ordering to be a well-ordering, it must have two properties:

Orderable: For any two elements $a, b \in S; a \neq b$ , exactly one of the following is true: $a < b$ or $b < a$ .
Least element: There is a least element in $S$ .

Just because a set satisfies the first property doesn’t mean it will satisfy the second. For example, there is no smallest real number in $\mathbb{R}$ , so the real numbers are orderable, but not well-ordered. Nevertheless, well-orderings are a very useful way to think about many kinds of sets.

Building the ordinals

Conveniently, if you order the natural numbers, you get a well-ordering. Mathematicians usually like to start from simple building blocks to construct more interesting ideas, so this seems like a good place to start building the ordinals. The most common way to construct the ordinals is to build them one at a time, starting from zero.

Each element in the sequence of ordinals corresponds to an order type: a description of how the items in the set are ordered. The first ordinal is zero, the order type of the empty set:

$0_{\mathtt{ord}} \equiv \emptyset$

The second ordinal is constructed by taking the previous ordinal and adding it to a set that contains that ordinal. In this case, that means taking the union of the ordinal $0_{\mathtt{ord}}$ and a set containing it, or $\{0_{\mathtt{ord}}\}$ . And since $0_{\mathtt{ord}} = \emptyset$ , the empty set, that means we have $\emptyset \cup \{\emptyset\}$ , so we wind up with just $\{\emptyset\}$ :

$1_{\mathtt{ord}} \equiv 0_{\mathtt{ord}} \cup \{0_{\mathtt{ord}}\} = \{0_{\mathtt{ord}}\} = \{\emptyset\}$

All the subsequent finite ordinals are constructed in a similar way:

$\begin{eqnarray} 2_{\mathtt{ord}} & \equiv & 1_{\mathtt{ord}} \cup \{1_{\mathtt{ord}}\} = \{0_{\mathtt{ord}}, 1_{\mathtt{ord}}\} & = & \{\emptyset, \{\emptyset\}\} \\ 3_{\mathtt{ord}} & \equiv & 2_{\mathtt{ord}} \cup \{2_{\mathtt{ord}}\} = \{0_{\mathtt{ord}}, 1_{\mathtt{ord}}, 2_{\mathtt{ord}}\} & = & \{\emptyset, \{\emptyset\}, \{\{\emptyset\}\}\} \\ 4_{\mathtt{ord}} & \equiv & 3_{\mathtt{ord}} \cup \{3_{\mathtt{ord}}\} = \{0_{\mathtt{ord}}, 1_{\mathtt{ord}}, 2_{\mathtt{ord}}, 3_{\mathtt{ord}}\} & = & \cdots \\ 5_{\mathtt{ord}} & \equiv & 4_{\mathtt{ord}} \cup \{4_{\mathtt{ord}}\} = \{0_{\mathtt{ord}}, 1_{\mathtt{ord}}, 2_{\mathtt{ord}}, 3_{\mathtt{ord}}, 4_{\mathtt{ord}}\} & = & \cdots \\ \end{eqnarray}$

Just as cardinal numbers are the possible answers to the question, “How many elements are in this set?”, the ordinal numbers are the possible answers to “What is the order type of this set?” The order type describes the arrangement of the items in the set as a sequence. An order type of 3, for example, means that we expect to see 3 elements in the set:

the smallest element
the second smallest element
the largest element

We could label those three elements as a sequence $(0, 1, 2)$ . The cardinality of the set for this sequence would be $\vert S \vert = 3$ , so the order type is the same as the set’s size. In other words, the ordinality and cardinality of finite sets are always the same. That’s not very interesting. It’s only when we consider infinite sets that they become useful.

Ordering the ordinals

We’re beginning to get the tools we need to wrangle the infinities into our grasp. With one more piece we’ll have everything we need: understanding the infinite ordinals.

We can order the ordinals by saying that $i_{\mathtt{ord}} < j_{\mathtt{ord}}$ if and only if $i_{\mathtt{ord}} \in j_{\mathtt{ord}}\$ . Remember, before we saw that ordinals are actually sets comprising all the previous ordinals; for instance, $3_{\mathtt{ord}} = \{0_{\mathtt{ord}}, 1_{\mathtt{ord}}, 2_{\mathtt{ord}}\}$ . So our ordering is obtained by looking to see whether an ordinal is already contained in another ordinal; if it is, then it’s a lesser ordinal.

The first infinite ordinal is what we get when we consider the set of all finite ordinals. This is the order type of a sequence which is well-ordered but doesn’t have a final element:

$(0, 1, 2, \cdots)$

We call this ordinal $\omega$ . The next ordinal is, perhaps not surprisingly, $\omega + 1$ , and we define it just like we did all the previous ordinals:

$\omega + 1 \equiv \omega \cup \{\omega\} = \{0_{\mathtt{ord}}, 1_{\mathtt{ord}}, 2_{\mathtt{ord}}, \cdots, \omega\}$

This corresponds to an ordering that looks like $(0, 1, 2, \cdots, \omega)$ , where an infinite number of elements are in the $\cdots$ . Notice that because $\cdots$ is infinitely long, there is no one ordinal that comes just before $\omega$ . But every finite ordinal $k_{\mathtt{ord}}$ comes before $\omega$ , no matter what finite value you pick for $k$ !

In effect, $\omega$ comes at the end of an infinite sequence of elements which lead up to it, which then ends that sequence. As a result, we call it a limit ordinal. The subsequent limit ordinals can be likewise defined:

$\begin{eqnarray} \omega + 1 & \equiv & \omega \cup \{\omega\} & = & \{0_{\mathtt{ord}}, 1_{\mathtt{ord}}, 2_{\mathtt{ord}}, \cdots, \omega\} \\ \omega + 2 & \equiv & (\omega + 1) \cup \{\omega + 1\} & = & \{0_{\mathtt{ord}}, 1_{\mathtt{ord}}, 2_{\mathtt{ord}}, \cdots, \omega, \omega + 1\} \\ \omega + 3 & \equiv & (\omega + 2) \cup \{\omega + 2\} & = & \{0_{\mathtt{ord}}, 1_{\mathtt{ord}}, 2_{\mathtt{ord}}, \cdots, \omega, \omega + 2\} \\ \end{eqnarray}$

If we extend this infinitely, we can get another limit ordinal: $2\omega$ . This is all the ordinals up to $\omega$ , and then $\omega$ more limit ordinals!

$2\omega \equiv \omega + \omega = \{\underbrace{0_{\mathtt{ord}}, 1_{\mathtt{ord}}, 2_{\mathtt{ord}}, \cdots}_{\omega \text{ finite ordinals}}, \underbrace{\omega, \omega + 1, \omega + 2, \cdots}_{\omega \text{ limit ordinals}}\}$

Then we can likewise define $2\omega + 1$ , then $2\omega + 2$ , and so on:

$\begin{eqnarray} 2\omega + 1 & \equiv & \{0_{\mathtt{ord}}, 1_{\mathtt{ord}}, 2_{\mathtt{ord}}, \cdots, \omega, \omega + 1, \omega + 2, \cdots, 2\omega\} \\ 2\omega + 2 & \equiv & \{0_{\mathtt{ord}}, 1_{\mathtt{ord}}, 2_{\mathtt{ord}}, \cdots, \omega, \omega + 1, \omega + 2, \cdots, 2\omega, 2\omega + 1\} \\ 2\omega + 3 & \equiv & \{0_{\mathtt{ord}}, 1_{\mathtt{ord}}, 2_{\mathtt{ord}}, \cdots, \omega, \omega + 1, \omega + 2, \cdots, 2\omega, 2\omega + 1, 2\omega + 2\} \\ \end{eqnarray}$

This again continues until we get $3\omega$ , then again for $4\omega$ , and so on:

$\begin{eqnarray} 3\omega \equiv 2\omega + \omega & = & \{\underbrace{0_{\mathtt{ord}}, 1_{\mathtt{ord}}, 2_{\mathtt{ord}}, \cdots}_{2\omega \text{ ordinals}}, \underbrace{2\omega, 2\omega + 1, 2\omega + 2, \cdots}_{\omega \text{ ordinals}}\} \\ \\ 4\omega \equiv 3\omega + \omega & = & \{\underbrace{0_{\mathtt{ord}}, 1_{\mathtt{ord}}, 2_{\mathtt{ord}}, \cdots}_{3\omega \text{ ordinals}}, \underbrace{3\omega, 3\omega + 1, 3\omega + 2, \cdots}_{\omega \text{ ordinals}}\} \end{eqnarray}$

If we keep going, we can also define another limit ordinal, $\omega^2$ :

$\omega^2 = \omega \cdot \omega = \underbrace{\omega + \omega + \cdots}_{\omega \text{ additions}}$

This is starting to get a little dizzying, but we can continue this even further with $\omega^\omega$ :

$\omega^\omega = \underbrace{\omega \cdot \omega \cdot \omega \cdots}_{\omega \text{ products}}$

The chain continues indefinitely, forever and ever, on without end. Each successive limit ordinal describes a new structure – a new way of ordering all the elements in the set you’re considering.

And here’s the most important part: the cardinality of all the limit ordinals, from $\omega$ to $\omega^\omega$ and beyond, is exactly the same as the cardinality of the natural numbers, $\aleph_0$ . Given an infinite number of elements, there are an infinite number of ways to structure and order them!

A infinite zoo of infinities

The choice of infinite structure we make – the choice of what to consider important when arranging elements, and what rules we choose to follow and break – is the source of a lot of counterintuitive ideas about infinity. Our notion of “size”, as represented in cardinality, is only one notion out of the infinite possible ways to define it. And cardinality ignores many properties of structure that we consider important in our everyday experiences.

The cardinality of any line and any plane are the same, since we can construct a bijection between any point on the line and any point on the plane.

For example, the number of points on a line is the same as the number of points on a plane. This an idea that seems impossible at first: the plane has more dimensions than the line. Not only does the line fit into the plane, it does so infinitely many times. But this is exactly the kind of structure that doesn’t need to be preserved in comparing cardinalities. The bijection between a line and a point cares only about whether the mapping is possible at all, and not whether it comports with our notions of how reality works. And the bijection does, in fact, exist; see space-filling curves.

So there isn’t just “an infinity”, $\infty$ ; this is a very restricted view of what infinity can be. Infinity is $\omega$ , and $\aleph_0$ , and their infinite continuations – each with their own special properties, each containing an infinity unto themselves. For each and every way we can imagine that there might be an endless sequence of something, there is an infinity associated with that idea.

Infinity is really, really, really, $\cdots$ , really big.

Solving the "Cheryl's Birthday" Problem with Prolog

Fri, 17 Apr 2015 04:00:00 -0400

The “Cheryl’s Birthday” problem has attracted a lot of press, with CNN, the Washington Post, Slate, and the Telegraph and countless others weighing in.

There’s plenty of explanations that give a good overview of the problem and show you how to work out the solution to the problem via a series of logical deductions. But what if we could just ask a computer to find the answer for us?

The problem

To briefly recap, the problem statement is as follows:

Albert and Bernard have just met Cheryl. “When is your birthday?” Albert asked Cheryl. Cheryl thought for a moment and said, “I won’t tell you, but I’ll give you some clues”. She wrote down a list of ten dates:

May 15, May 16, May 19

June 17, June 18

July 14, July 16

August 14, August 15, August 17

“One of these is my birthday,” she said.

Cheryl whispered in Albert’s ear the month, and only the month, of her birthday. To Bernard, she whispered the day, and only the day. “Can you figure it out now?” she asked Albert.

Albert: “I don’t know when your birthday is, but I know Bernard doesn’t know, either.”

Bernard: “I didn’t know originally, but now I do.”

Albert: “Well, now I know, too!”

When is Cheryl’s birthday?

The heart of the problem are the statements exchanged by Albert and Bernard. Initially, neither Albert and Bernard know when Cheryl’s birthday is, but through deduction they mutually arrive at a unique, correct answer.

This must mean that, somehow, the set of possibilities offered by Cheryl satisfies some set of properties that make it possible for Albert and Bernard to arrive at the answer. If we can write these properties down into a form that a computer can understand, we can have these deductions done for us. Wouldn’t that be nice?

Enter Prolog! It’s pretty rare that I ever pick it up, but this seemed like a great excuse to try it out again. Let’s see where we get!

Prolog

If you want to install Prolog yourself and follow along, follow the directions here. You can load Prolog programs with prolog -e x.pl.

Prolog belongs to a relatively rare breed of logic programming languages, in contrast to imperative programming languages. Broadly, imperative languages require that your program take the form of a series of explicit instructions that describe what to do. For example, an imperative program to find the smallest element in a list might take a form like this:

smallest(list):
  * Declare `result`.
  * Set `result` to the first element of `list`.
  * For each element `e` in `list`:
    * Set `result` to be `e`, if `e` is smaller than `result`.
  * Answer: `result`.

With logic languages, however, you describe a set of facts and ask whether there is a consistent conclusion that can be reached from those facts. The logic-programming version of the same program might be something like:

smallest(list):
  * Answer: the element `e` in `list` such that for all other
      elements `E`, `e` <= `E`.

This is an interesting inversion of responsibilities relative to imperative programs: instead of telling a computer what to do, we say what the answer has to look like. Then the computer is free to decide how to arrive at the answer.

; is disjunction in Prolog, similar to the or keyword elsewhere; , is conjunction, like and.

All Prolog programs adopt this approach. We describe a series of true statements, then ask Prolog to evaluate a question to see if there is an answer that is consistent with everything we described. Those statements can take the form of either facts or rules.

Facts are true statements in the universe of the program, like “grass is green”, “Bob is a person”, or “Alice is Bob’s sister”. In Prolog, we might express these as green(grass), person(bob), and sibling(alice, bob), respectively.
Rules tell you how to evaluate a set of circumstances to decide if they’re true or not. For example, we might define a rule like parent(X, Y) :- mother(X, Y); father(X, Y).; this could be used to express the rule “X is Y’s parent if (a) X is Y’s mother, or (b) X is Y’s father”. Facts are like rules whose body is just the predicate true.

That’s about it. Let’s see if we can use Prolog to help us solve Cheryl’s birthday.

Getting to an answer

Doing anything useful in Prolog is about constructing the correct set of facts and rules to describe the universe you want to talk about. Our task is to represent the problem statements as facts or rules so that we can arrive at the correct answer.

Cheryl’s list of birthdays

First, Cheryl says that her birthday is one of the previously mentioned ten distinct possibilities. To represent this in Prolog, we’ll just enumerate them as individual facts:

candidate_birthday('May', 15).
candidate_birthday('May', 16).
candidate_birthday('May', 19).
/* ... */
candidate_birthday('August', 17).

If we wanted to cut down on the repetition, we can also write this as a rule instead of a series of facts. The rule is that a month and a day are a candidate birthday if they belong to the set of candidate birthdays:

candidate_birthday(Month, Day) :-
  member(Month/Day,
    [
      'May'/15, 'May'/16, 'May'/19,
      'June'/17, 'June'/18,
      'July'/14, 'July'/16,
      'August'/14, 'August'/15, 'August'/17
    ]
  ).

The rule form is nicer if we have a lot of birthdays to check, since we can just add an element to the list.

Next, we have the dialogue between Albert and Bernard:

Albert: “I don’t know when your birthday is, but I know Bernard doesn’t know, either.”
Bernard: “I didn’t know originally, but now I do.”
Albert: “Well, now I know, too!”

We’ll need to transform each of these three statements into Prolog to find the answer.

First statement: Albert’s stumped, but knows Bernard is too

We’re looking for a <month, day> combination that makes it possible for each of the three statements to be true, so we should use some rules to represent that. We’ll start with Albert’s first statement.

We know that the final answer must be one of the days Cheryl named, so let’s start with that constraint:

s1(Month, Day):-
  candidate_birthday(Month, Day),
  /* ... */

Next, we know that Albert knows the month but not the day of Cheryl’s birthday, so he only has one half of the answer. However, even if Albert didn’t know the day, he could know the answer if his month only had one possible day. But since every month has at least two possible choices for the day, he doesn’t know the answer. So it must be the case that Albert’s month has two or more choices for the day:

s1(Month, Day):-
  candidate_birthday(Month, Day),
  findall(X, candidate_birthday(Month, X), [_, _ | _]),
  /* ... */

The findall predicate produces a list that satisfies all the constraints of the goal, the second parameter. In this case the predicate is true if the list of candidate birthdays has two or more elements. The [...] syntax denotes a list, while underscores mean “an arbitrary result”, so [_ , _] means “a list with exactly two results”.

Adding the list constructor | here means “the rest of the list”, so [_, _ | _] means “a list with exactly two results, followed by zero or more results”. So the overall effect is what we want: only candidate birthday months that have at least two elements.

This accounts for the first half of Albert’s statement – that he didn’t know what Cheryl’s birthday was, because there is not yet a unique answer. We also need to account for the second half of Albert’s statement: Albert knows Bernard doesn’t know the answer, either. The only way Albert could know that is if Bernard had a unique answer. And Bernard could only have a unique answer if his day uniquely determined his month.

There are two unique dates: the 19th only appears in May, and the 18th only appears in June. Albert therefore doesn’t have May or June; if he did, Bernard could be holding either 18 or 19, and thus have a unique answer. But since Albert doesn’t have May or June, he knows Bernard can’t deduce the answer yet. So Albert knows that he holds a month which does not decide the day. We can write a rule that represents this idea:

month_has_deciding_day(Month):-
    candidate_birthday(Month, Day),
    findall(M, candidate_birthday(M, Day), [_]).

and then we can invoke the negation of the rule to finish off Albert’s first statement:

s1(Month, Day):-
  candidate_birthday(Month, Day),
  findall(X, candidate_birthday(Month, X), [_, _ | _]),
  not(month_has_deciding_day(Month)).

This leaves us with five possible answers – the dates in July and August. We can ask Prolog to verify this for us at the console:

$ prolog -e cheryl.pl

?- s1(Month, Day).
Month = 'July',
Day = 14 ;
Month = 'July',
Day = 16 ;
Month = 'August',
Day = 14 ;
Month = 'August',
Day = 15 ;
Month = 'August',
Day = 17.

Second statement: Bernard was stumped, but now he isn’t

Bernard now says he knows the answer. He knows that Albert must hold a month which doesn’t decide the day – so he knows Albert is holding July or August. Since he knows the answer, it must be because every date uniquely decides a month at this point. Therefore, Bernard can’t be holding 14, because that could be either July or August.

So our rule must therefore be able to determine the month at this point:

s2(Month, Day):-
  candidate_birthday(Month, Day),
  findall(X, s1(X, Day), [Month]).

This rule returns true if the month and day of a given birthday are in the candidate list and if they uniquely determine the month after taking into account the first statement. Only three possible dates remain at this point: July 16, August 15, and August 17 – the remaining dates which uniquely determine their month. We can verify this at the console:

$ prolog -e cheryl.pl

?- s2(Month, Day).
Month = 'July',
Day = 16 ;
Month = 'August',
Day = 15 ;
Month = 'August',
Day = 17.

Third statement: Albert’s got it too

We’re getting close! We just need to show that Albert can deduce the answer too. Since Albert holds the month, he could only know the answer for sure if his month uniquely determines the day:

s3(Month, Day):-
  candidate_birthday(Month, Day),
  findall(X, s2(Month, X), [Day]).

This rule returns true if the month and day of a given birthday are in the candidate list, and if they uniquely determine the day after taking into account the second statement.

Albert can’t be holding August, therefore, because that wouldn’t uniquely determine the day – it could be either August 15 or August 17. So he must be holding July, which in turn means that Cheryl’s birthday must be July 16. And, indeed, that’s what we get:

?- s3(Month, Day).
Month = 'July',
Day = 16 .

Fantastic! Logic programming isn’t suitable for everything, but it unquestionably shines at solving constraint-based problems like this one.

Additional resources

You can see the GitHub repository for our script here if you’d like to experiment with things yourself. It is borrowed almost completely from Charles Perng’s version.

And finally, if you’d like to read more about Prolog, there are a lot of great resources around the web. The one that helped me the most was Learn Prolog Now! by Patrick Blackburn, Johan Bos, and Kristina Striegnitz. You can also check out the very excellent IRC channel, which is ##prolog on Freenode.

Happy hacking, friends.

Fully-Featured Custom Collections in Jekyll

Fri, 10 Apr 2015 03:00:00 -0400

Jekyll is a popular static site generator written in Ruby and in wide use. A major selling point for Jekyll adoption is that it provides some excellent defaults out of the box, heavily geared towards the assumption that you’re writing a blog. It does this largely by taking a convention-over-configuration approach; the defaults are designed to be sensible choices for the majority of its users. However, like many convention-over-configuration systems, it can be tough to make it behave the way you want as soon as you step outside of the prescribed boundaries.

Because it’s so easy to get started with Jekyll, it feels frustrating when you’re eventually hamstrung by framework limitations. Unfortunately, while knee-deep in writing this site, I found myself in exactly that situation with respect to Jekyll’s collections. I wanted to make my own custom collections, but very few of the features available to the default collections are available to the custom collections.

Jekyll’s default collections

Jekyll ships with two types of collections by default:

Posts are static files that live in your _posts directory by default and have the form <YYYY>-<MM>-<DD>-<title>.<extension>. Jekyll steers people towards the use of posts in blog environments thanks to comfortable defaults, and for that reason, most people write up their blog’s entries as posts since it’s the best fit. In your Liquid templates, the list of posts is available via site.posts, which is sorted in reverse chronological order.
Pages are static input files that get processed and mapped to correspondingly named output files. For example, if you have contact.markdown as an input file, it’ll get mapped to contact.html in the output folder. In your Liquid templates, you can get the list of pages by iterating over the site.pages collection.

Limitations of default collections

Pages and posts suffice for most use cases. However, these have some limitations.

Inflexible post filename formats

First, every post file must be named according to the format string <YYYY>-<MM>-<DD>-<title>.<extension>. Any post which doesn’t have this form is invalid and throws an error.

I found this to be an obnoxious limitation because I don’t usually finish a post on the same day I start it. That means that the filename I make the post with reflects the day I started the post, not the day I finished the post. You can override the date that Jekyll actually outputs by using date in your YAML front matter, but then there’s no point in also having a date in the filename, since it’s redundant and never used.

Inconsistent domain model of posts and pages

Pages don’t have access to most of the Liquid attributes that posts do. This means that your templates must special-case pages and posts to a degree that requires a lot of repetition. In my layouts, a post is merely a page that has some extra metadata shown, so I wanted to avoid special-casing things if I could.

The crux of the issue can be found in the Jekyll code for posts and pages. Posts get fourteen attributes to work with:

ATTRIBUTES_FOR_LIQUID = %w[
  title url dir date id
  categories tags
  next previous
  path

  content
  excerpt
  excerpt_separator
  draft?
]

But pages only get a small subset of these, plus a confusing name attribute that is distinct from the title found on posts.

ATTRIBUTES_FOR_LIQUID = %w[
  content
  dir
  name
  path
  url
]

There is an open issue for Jekyll 3 discussing the possibility of unifying collections of different kinds, but it’s not yet resolved, so the status quo is still a vast gulf of difference.

Inconvenient sorting defaults

While most of Jekyll’s defaults are sensible, the sorting defaults can be undesirable. The sort key on a given post is the date of the post, which is either generated from the filename or from the date attribute. Jekyll sorts posts in ascending date order, so in a list of posts, a post dated January 15 comes after, e.g., one dated January 2, even though January 2 is lexicographically later.

But for pages, the sort key is the filename, so january-2.html is sorted after january-15.html, which is the reverse of how posts behave. So if you want to get away from the special-casing of posts and have everything be a page, you’d have to accept some behavior changes.

The way this behavior is implemented isn’t very convenient, however. Jekyll recommends using the sort filter in your Liquid templates. If you want pages to be sorted by, say, their foo property, you can do:

{% for p in site.pages | sort: 'foo' %}

This requires you to make sure you have a foo property in your YAML front matter for each page, which can be tedious. If you forget to add this for even a single post, there won’t be any hints or warnings – your site’s ordering will just silently break.

Anything that’s not in the _posts collection doesn’t get next or previous attributes to it, as we saw before. That means there’s no way to get an ordering over the items in your collection, which is obnoxious if your collection has a natural ordering over some field. For example, you almost always want blog posts to be ordered by their date field, but you won’t get this ability on your pages.

Custom collections to the rescue

What do you do if the thing you want to add is neither a page nor a post? You can add custom collections – Jekyll’s term for a set of things of the same kind, like “blog entries”, “soccer matches”, or “team members”. While Jekyll’s documentation is usually very good, it doesn’t cover how to do this in a manner that retained most of the features of the default collections without having to rewrite it myself.

To make a custom collection, first decide on a name for your collection. Then make a folder named _<name> in your source directory. For instance, if you want to make a collection representing the authors on your site, you might have an authors collection, so you would want a directory named _authors.

Second, configure the collection in your project’s _config.yml under a hash named collections. Each key in this hash is the name of a collection you’ll use in Jekyll. For example, a minimal authors collection would look like this in your _config.yml:

collections:
  authors

If you have other collections, you can add them here too, one collection per key.

collections:
  authors:
  entries:
  recipes:

If the files in a collection should generate corresponding output files, then add output: true to each custom collection for which that’s true. For example, if you want to generate a page for each author, your collection can be:

collections:
  authors:
    output: true

At this point, you’ve got a custom collection, but it’s not very useful – it has none of the nice default behavior from posts that we mentioned before, and it lacks even basic things like being able to sort your collection by date.

Making custom collections useful

To fix this, I wrote the SpicyJekyll plugin: this augments your custom collection with five useful behaviors to make them behave more like you might expect:

Orders your collections by date (or a custom sort key of your own choosing).
Adds next and previous references to your collection’s documents so you can use them in lists or archives.
Checks for a title property on each document.
Adds stable, filename-based permalinks that won’t change if you change the title.
Logs what it’s doing to diagnose collection problems if there are any issues.

To use it, just drop it into your plugins directory. Give it a spin on your own custom collections!

Time Passes

Sun, 05 Apr 2015 03:00:00 -0400

Every once in a while, I’ll think about a somewhat morbid and yet universal question: how long do I have left to live? While no one knows for sure when their time will eventually run out, everyone confronts the reality of their ultimate fate sooner or later, willingly or not.

Thanks to its continuous presence in all human lives, people have thought about death quite a bit. One of my favorite ruminations on the subject is Longfellow’s Psalm of Life:

Art is long, and Time is fleeting, And our hearts, though stout and brave, Still, like muffled drums, are beating Funeral marches to the grave.

This excerpt, at first glance, sounds rather depressing and hopeless – it suggests that we’re inevitably and inextricably on a track towards doom. But on reading the full poem, we can see that it’s actually an exhortation to make the most of the time we do have, no matter how long or short:

Lives of great men all remind us We can make our lives sublime, And, departing, leave behind us Footprints on the sands of time.

I started reflecting about some other events that have been “footprints in the sands of time”, and like any developer, decided to write a little code to investigate things. Below are the approximate dates of some historical and predicted future events, and how far away we are from them.

event	Δ time (in sec · days · yr)
first humans evolve from primates	·
first mechanical clock	·
first movable-type printing press	·
first e-mail over ARPANET	·
since I was born	·
HTTP 1.0	·
September 11th attacks	·
Marty McFly’s arrival in the future	·
2016 U.S. presidential election	·
Ray Kurzweil’s prediction for the Singularity	·
Halley’s comet returns in 21^st century	·
world population reaches 10 billion	·

The table is an interesting starting point for getting our bearings, but big stretches of time are tough to internalize for many people. We can usually recall what we were doing a day ago with some fidelity. We can give the general outline of what happened ten days ago, maybe by consulting our calendar.

But it would be difficult to remember what was happening one hundred days ago. And going back one thousand days – about 2.7 years – is likely to exceed any normal person’s capabilities. You can probably remember big ongoing events, like where you lived and your major relationships and friendships, but little else. These days are lost forever in the mists of time and memory.

To put that in better perspective, citizens in the top one-third of countries by life expectancy can expect a mean lifespan of about eighty years. That’s about 30,000 days. By the time you hit your 27th birthday, you’re down about 10,000 of those days – and we can barely remember the overwhelming majority of them.

In that milieu, what does 10,000 days mean against your total lifespan? It’s hard to get the same perspective there that we have for a single day or a week. What if we ask a more direct question: how much time do we have left, and how can we visualize that?

Visualizing your lifespan

I made this small <canvas> widget to help examine that question. We initially assume you’ll live to the ripe old age of eighty and that you were born on January 1, 1980. Make adjustments below as necessary for your situation.

Each colored horizontal line represents one year of your life.
Red years are your past: you can never get them back.
Green lines are your future, as yet unwritten.
The pulsing blue dot shows where you are right now at this very moment.

Your birthday is .

You expect to live for years.

Making the most of it

A series of lines feels like too simple a representation for something as complicated and messy as a human life. But I can’t deny that this depiction also mirrors a simple truth: everyone has a finite amount of time alive before they become a husk of inert matter. Everything we will do or have done falls somewhere on one of those thin lines. It’s ultimately up to us to make the most of it.

And yet, this doesn’t need to be a discouraging idea. Longfellow had something to say about that, too:

Let us, then, be up and doing, With a heart for any fate; Still achieving, still pursuing, Learn to labor and to wait.

In the Beginning

Wed, 01 Apr 2015 04:00:00 -0400

It’s probably unusual to start the first post for a blog by talking about one’s past failures, and yet few things seem more appropriate right now.

My failure to start and commit to any kind of blog has been a recurring thorn in my side. There’s always been a litany of perfectly plausible excuses to draw on:

I don’t have enough time right now
Other things in my life are more important
Building my startup should be my sole and top priority

The most pernicious lies we tell ourselves always seem to be the most comforting ones.

After reflecting on why the past attempts failed, I noticed a common theme: I kept starting things off not by writing text, but by writing software. The fun part for me was always writing the blog engine, the build pipeline, the database mappings. Each attempt rarely made it to the point where I was actually putting words down instead of code. Inevitably, I’d be sidetracked by self-imposed programming obstacles. The software, it turned out, was the thing I really wanted to build; the blog was just the convenient excuse to do it.

With the fresh perspective of a new year ahead, I decided it was finally time to take the drastic measures necessary to get myself writing again. One word at a time, one keystroke in front of another – the relentless yet somehow enjoyable slog of putting words on a page that even rank novices like me manage to derive satisfaction from.

To avoid falling into the same trap as before, I did something that was really hard for me: I didn’t write the blog engine from scratch. I allowed myself a few small fixes, but anything that didn’t get me closer to a single-command publish-to-the-Internet build pipeline was immediately scrapped. Nothing improves productivity so much as eliminating stuff you don’t really need to do.

The result: this blog, and more importantly, this post! We’ll see how it goes.