Creative destruction: The fragmentation and consolidation of Canadian political parties (part 1)

Note — After sharing an earlier version of this post with my friend JJ McCullough, he noted that many historical sources on Canadian election data differ considerably: you need to be careful about what sources you pull from.  He has produced a wonderful website on Canada (The Canada Guide), which also covers historical elections.  You should check it out!

Mergers and acquisitions

The creation of new parties and the mergers of old ones is a perfectly normal part of the Parliamentary system and one that is sometimes overlooked:  in discussions on electoral reform, it's tempting  to focus on which party today would benefit from a proposed change (after a quick Google, here is an example).  Although these articles can be fun speculation, there is a problem with this sort of analysis: it assumes today's political parties will remain static even after some fundamental change to the voting system.  

Over the course of two short posts, I want to show you a longer time series of Canadian elections and a couple quick cuts of the data to show how parties rise and fall.  Their emergence (and destruction) is a crucial part of the "first past the post" (FPTP) system — in fact, a crucial part of any democratic political system!  Ideological (or other) movements must iron out their difference and coalesce into a well-organized party so as to present the country with a coherent and unified option for Government.  These "broad churches" are themselves smaller deliberative bodies, where different factions must compromise.

When factions cannot find reasonable compromise within an existing party, they can start their own.  This is what happened when Lucien Bouchard left the PCs to form the Bloc Quebecois or when Preston Manning (and others) founded Reform.  The converse is also true: Stephen Harper and Peter MacKay merged the Canadian Alliance and the Progressive Conservatives, which created the Conservative Party in 2003.

Even over the last decade there has been plenty of speculation on whether the Liberals and NDP should merge (here is but one example).  

Federal votes and election outcomes

I've pulled the historical Federal election outcomes since Confederation from a variety of sources (although mainly from the Library of Parliament).  Below are two plots: the top one shows the distribution of seats from every election since 1867, while the lower one shows the popular vote percentages.

I have ordered the parties (roughly) so that the Conservative / Right-wing parties are near the top of the chart and the Liberals are at the bottom.   Since these are the only two parties that have held government, this should show the back-and-forth between them over time (in particular, whenever one crosses the white 50% line).

This first chart (above) shows the distribution of seats in the House of Commons after every Federal vote coloured by political party. It is easy to see the Liberal and Conservative majorities as they cross the white horizontal 50% line.

This second chart (above) shows the distribution of votes coloured by political party). There are few cases where any political party gets over 50% of the vote (these cases are mostly when there were only two major parties from 1867-1917)

A few quick reflections from these charts:

  • After the 19th century, majority governments have been common but majority popular votes are not: since the rise of >2 major political parties, there has only been 2 elections where a party received the majority of the popular vote

  • The distribution of seats is more volatile than the distribution votes: this shows the convexity that emerges from our voting system - small changes in the vote proportions lead to large changes in the seats (I have another post on this topic.)

  • There appear to be two major "fragmentation moments" in Canadian politics over the last 150 years:

    • ~1917 (onward): The rise of the Progressives, Social Credit and other parties (CCF, New Democracy)

    • ~1990: The fragmentation of the political right and the rise of separatism

  • Political fragmentations take years to resolve: after the break-up of the Right in the 19080s / 90s, the Liberals achieved four mandates (three majority and one minority) before a Conservative party took government

In a future post, I'll show some more detail on how the the “number” of political parties has increased over time.

The magic number: What’s with the fixation on 40% for a majority?

A common trope from the Canadian commentariat is that 40% national support should result in a majority government (see here from Nanos Research).  I’ve always been skeptical whenever a journalist trots out "rules" like this since they often ignore the complexities and subtleties of our system.

But 40% does seem roughly right: the last two majority governments received just above 39% of the vote. 

To test this, I've taken all the elections since 1945 to see the relationship between the popular vote received by each party and the proportion of seats they won in the House of Commons:

Each dot in this plot represents a single party's outcome in an election (e.g., in 2015, the Conservative Party won 32% of the national vote and got 29% of the seats in the HoC).  This plot shows that the relationship between the national vote and seats in the HoC is convex: smaller parties are under-represented, while larger ones are over-represented.  This attribute makes it easier to form majority governments since a party can win 50% of the seats with fewer than 50% of the votes. 

Looking at the data and fitting a quadratic line** shows that when a party gets 40% of the national vote, they should expect to get roughly 50% of the seats in the House of Commons.  The "40% for a majority" rule-of-thumb is pretty good! 

But there is another interesting pattern in the data: the relationship is quite clearly convex. That is, if you get below 30% of the national vote, you should expect to be "under-represented" in the House of Commons.  If you score over 30% of the national vote, then you should expect to get a disproportionate number of the seats in the House of Commons.  (This is shown on the chart as being above or below the 45 degree line - if you are on the 45 degree line, your national vote percent and representation in the HoC are the same)

This demonstrates that larger parties have an advantage in the FPTP system: two smaller parties that each get 20% of the national vote would expect to elect fewer MPs than a united party that gets 40% of the vote.  This provides a strong incentive for parties to unite and create broad coalitions (instead of smaller fringe parties).

Another implication is that relatively small differences in vote can result in large differences in the House of Commons.  Consider the current situation in Canada: the Liberals got just shy of 40% of the vote and received 54% of the seats in the House.  The opposition Conservatives got 32% of the vote and 29% of the seats.  An eight percentage point difference in vote results in a 25 percentage point gap in representation in the House of Commons. 

Put another way, changes in the House of Commons should be much more volatile than changes to the national vote.  This is very clear in the data, but that will be covered in a future post.

** An older version of this analysis on this site used a Loess regression.  I couldn't be bothered running the code again so just did this analysis in Excel using a polynomial regression.  The result is very similar and the conclusion doesn't change.

A week is a long time in politics: 14 years of Canadian polling data

This post’s title is a saying often attributed to Harold Wilson, the British Labour Prime Minister who served in the 1960’s and ‘70s.  Although the origins of the quote are unclear -- Wilson did not recall ever saying it -- the meaning is not: fortunes in politics are subject to rapid rises and falls.  This is, no doubt, very true in Canada.

In this first post, I want to show you a straightforward time series of Canadian political party support over the last 14 years.  To my knowledge, it is the longest single time series of Canadian political polls. 

This period coincides with the creation of a united Conservative party, the secular decline of Quebec separatism, the “coalition crisis” of 2008, the “Orange crush” collapse of the Liberal vote in 2011, and the subsequent Liberal majority in 2015.  It spans four Liberal leaders (Martin, Dion, Ignatieff, and Trudeau), three NDP leaders (Layton, Mulcair, and Singh), and two Conservatives (Harper and Scheer). 

With all that laid out, here are some quick high-level points that emerge from this national picture:

  • Changes in national political support have “fat tails”: party support can appear relatively stable for long periods of time before suddenly shifting.  To put this another way, political support is subject to “Black Swan events” where the magnitude of shifting support can dwarf any smaller variations previously observed.  A clear example of this is the rise of the NDP in the 2011 election: if you look at the data leading up to the 2011 election, it would be nigh impossible to predict the NDP surge, which is well above any historic highs they ever achieved.  In fact, it appears as if the Liberals and NDP quite literally swap 15 points of support in a single week!  Which leads to the second point…
  • Election campaigns matter: major shifts in the political environment coincide with election campaigns.  Whether it’s the Conservatives overtaking the Liberals (2006), the NDP and Liberals swapping (2011), or the Liberals overtaking both the Conservatives and NDP (2016), there are many examples where support for the parties changes radically from the start to end of a campaign.
  • Party leadership conventions move polls: Just eye-balling the chart shows that new leaders often get a bounce when they’re nominated.  Dion saw a ~5ppt bump, Ignatieff was probably closer to 10ppt, Mulcair got ~5ppt, and Trudeau saw a sustained ~10ppt step-up in support.  Although it might be too soon to tell, Scheer and Singh appear to have much smaller bumps in comparison.

With all that being said, anyone familiar with Canadian politics knows that the national support for a party is actually a rather meaningless number.  The actual outcome of the election is determined by the individual electoral districts (like the US electoral college).  For instance, the Bloc Quebecois has little support nationally yet they play a disproportionate role in our politics due to their regional concentration.  Similarly, the Green party is a national irrelevance because their support is so dispersed. 

In short, what’s probably more interesting is what this picture doesn’t show: our political system is highly regional and this can have profound effects on electoral outcomes.   As a continuing series of posts on this blog, I hope to lay out some of my thoughts on how our election system can be modeled.

How to read the chart:

  • Each party is represented by a colour: Liberals are red, Conservatives are blue, the NDP is orange, the Greens are green (surprise!), and the Bloc is purple
  • Each of the lightly coloured dots represents a single poll published by a polling agency
  • The coloured lines and bands represent a smoothed average of these polls over time (using a Loess regression.  I’ll probably explain how this works in a future post.  For now, I want to avoid too much maths)

Sources: Over the years, I have manually tracked publicly reported polls.  In recent years, I have started also relying on Wikipedia to make sure I have not missed any polls.  You can easily access the Wikipedia data here.