In 2018 the 6.7 million NZETS emission units allocated to 76 emitters were worth at least 243 million NZ dollars

All this data tidying and charting. It can lead to not seeing the forest for the trees. In this post I estimate the value of the 2018 free industrial allocation of emissions units to emitting industries. The number is 243 million New Zealand dollars. I find that gob-smacking

Following up from my last post, I wondered what was the market value of the 6.7 million emission units given to eligible emitters under the New Zealand Emissions Trading Scheme 2018 industrial allocation?

We will need market prices for emissions units. There is an online 'open data' Github repository of New Zealand Unit (NZU) prices going back to May 2010.

The NZU repository has it's own citation and DOI:

Theecanmole. (2016). New Zealand emission unit (NZU) monthly prices 2010 to 2016: V1.0.01 [Data set]. Zenodo. http://doi.org/10.5281/zenodo.221328

Under the Section 86 of the Climate Change Response Act 2002, eligible emitters or 'participants' in the ETS, as they are defined, may apply to the Environmental Protection Authority for a 'provisional' or estimated quantity of units for a future compliance year and for a 'final' or actual quantity of units for a past year.

We recall that one of National's 2009 amendments to the NZETS was to make unit allocation proportional to actual production. So that requires an provisional estimate and an actual 'wash-up' calculation once actual production from the regulated 'activity' is known.

The emitters must apply for both provisional and final allocations between 1 January and 30 April of each year. I am assuming that the EPA checks the applications and then transfers the initial allocation to accounts in the NZ Emissions Trading Register in May of each year.

The provisional allocation does not have to be gazetted or published. The final allocation must be gazetted or publicised on the EPA's website under Section 86B(5). But it can't be known until the EPA has received and processed all the historic wash-up applications for the just finished calendar year. That's why in September 2019 the EPA have only the 2018 unit allocations on their website. And not the 2018 final allocations.

It may be the accountant within me, but I think the emitters will want to get their initial application as close to 100% correct as possible. Or to exceed it. Because free units now will always be better than free units in 12 months time.

So let's assume the initial industrial allocation is transferred to emitters' accounts in May. At that point we can make a calculation of market value. We can check our NZU price data for a mid-May price as it's expressed in average monthly prices.

On our graph of of NZU prices, we add a vertical line for the 15th of May and then where that line intersects with the price line we add a horizontal line across to the prices on the Y axis. Or we could have looked up the .csv file of the prices. We get $NZ 21.38 per unit.

Multiplying the 6,743,573 units by $21.38 equals 143,503,233 NZ dollars!

I am gob-smacked by that! $143.5 million! Just gifted to emitters! Deliberately to reduce the effect of the carbon price on these privileged emitters. And as Idiot/Savant noted, the allocations will slowly and incrementally 'phase out' by the minutest of percentages until 2015!

Of the big three, New Zealand Steel received units worth $NZ 37.9 million, New Zealand Aluminium Smelters received units worth $NZ 28 million and Methanex received units worth $NZ 20 million. Here's the pie chart denominated in dollars.

Here is a bar chart which are usually easier to read.

Here is the tidied data of the name of the emitter, the amount of final units and the value at the mid May price of $NZ21.38. The data file is also at Google Drive sheets.

EPA Industrial Allocation Units Value 2018

Name	Allocation	Value
New Zealand Steel Development Limited	1782366	37928748.48
New Zealand Aluminium Smelters Limited	1324556	28186551.68
Methanex New Zealand Ltd	945210	20114068.8
Fletcher Concrete and Infrastructure Limited	584032	12428200.96
Oji Fibre Solutions (NZ) Limited	484322	10306372.16
Ballance Agri-Nutrients (Kapuni) Limited	325594	6928640.32
Pan Pac Forest Products Limited	210652	4482674.56
Norske Skog Tasman Ltd	200556	4267831.68
Winstone Pulp International Limited	151546	3224898.88
Graymont (NZ) Limited	144405	3072938.4
Whakatane Mill Limited	139690	2972603.2
ACI OPERATIONS NZ LIMITED	59945	1275629.6
Fonterra Limited	50664	1078129.92
Asaleo Care New Zealand Limited	29419	626036.32
Nelson Pine Industries Limited	26569	565388.32
Wallace Group Limited Partnership	26539	564749.92
Pacific Steel (NZ) Limited	19550	416024
EVONIK PEROXIDE LIMITED	18443	392467.04
Daiken New Zealand Limited	17770	378145.6
Dongwha New Zealand Limited	16854	358653.12
Status Produce Limited	15496	329754.88
Taranaki By-Products Ltd	14197	302112.16
Exception Limited	11618	247231.04
Tuakau Proteins Ltd	11393	242443.04
Anchor Ethanol Limited	10784	229483.52
Southern Paprika Limited	10406	221439.68
Alliance Group Limited	10012	213055.36
Affco New Zealand Limited	9465	201415.2
Under Glass (Karaka) Limited	7574	161174.72
Gourmet Mokai Limited	7006	149087.68
Under Glass (Bombay) Ltd	6194	131808.32
Websters Hydrated Lime Company Limited	5999	127658.72
J.S.Ewers Ltd	5853	124551.84
Hawkes Bay Protein Limited	5740	122147.2
CMP Canterbury Limited	5470	116401.6
Juken New Zealand Ltd	5304	112869.12
Gourmet Paprika Limited	4837	102931.36
PVL Proteins Limited	3632	77288.96
Fletcher Building Products Limited	3344	71160.32
Sharma Produce Limited	2677	56966.56
Gourmet Waiuku Limited	2190	46603.2
Kakariki Proteins Limited	2037	43347.36
Shipherd Nurseries Limited	1900	40432
Island Horticulture Limited	1717	36537.76
Tegel Foods Limited	1632	34728.96
Value Proteins Ltd	1581	33643.68
Whakatane Growers Limited	1393	29643.04
P H Kinzett Ltd	1344	28600.32
Moffatts Flower Company Limited	1182	25152.96
Karaka Park Produce Limited	1169	24876.32
Van Lier Nurseries Ltd	1123	23897.44
Taylor Preston Limited	1104	23493.12
Meenakshi Devi Sharma, Raj Kumar Sharma	1080	22982.4
Vege Fresh Growers Limited	1075	22876
Jai Shankar Growers Limited	928	19747.84
Prime Range Meats Limited	928	19747.84
Homestead Produce Ltd	881	18747.68
Sinai Hort Limited	604	12853.12
J.S. Mahey Limited	599	12746.72
Castle Rock Orchard Ltd	564	12001.92
Karamea Tomatoes Limited	526	11193.28
Poppas Peppers 2009 Limited	351	7469.28
Taaza Green Limited	337	7171.36
Harbour Head Growers Ltd	261	5554.08
Ting-Yuan Robert Wu	239	5085.92
Parkgard Growers 2000 Limited	222	4724.16
Antone James Ivicevich, Joanne Elizabeth Gould Ivicevich	210	4468.8
Graeme Lowe Protein Limited	198	4213.44
Mary Jane Fausett, Peter James Fausett	143	3043.04
Pomoana Gardens Limited	100	2128
John Hamilton Charles Falloon, Paul Gregory Whitehead	79	1681.12
Royal Roses Limited	66	1404.48
Kingbridge Ltd	61	1298.08
Eseta Kovati, Reupena Kovati	37	787.36
GELITA NZ Ltd	29	617.12
Wallace Corporation Limited	0	0

Ten NZ companies were given 6.7 million free emission units in 2018

Have open tidy data; will graph it. I whip up a pie chart of the top ten New Zealand companies rorting the New Zealand Emissions Trading Scheme via free allocation of emissions units.

Of 6.7 million NZ Emissions Trading Scheme emission units allocated by the Environmental Protection Authority (given for free instead of being sold by auction) to industries in 2018, 6.2 million or 91% went to ten well-known New Zealand companies.

Here is the R script.

Here is the data of the emissions units gifted for free to industrial emitters in 2018.

Windfall gifting of emissions units to industry in 2018

Name	Allocation
New Zealand Steel Development Limited	1,782,366
New Zealand Aluminium Smelters Limited	1,324,556
Methanex New Zealand Ltd	945,210
Fletcher Concrete and Infrastructure Limited	584,032
Oji Fibre Solutions (NZ) Limited	484,322
Ballance Agri-Nutrients (Kapuni) Limited	325,594
Pan Pac Forest Products Limited	210,652
Norske Skog Tasman Ltd	200,556
Winstone Pulp International Limited	151,546
Graymont (NZ) Limited	144,405
Whakatane Mill Limited	139,690
ACI OPERATIONS NZ LIMITED	59,945
Fonterra Limited	50,664
Asaleo Care New Zealand Limited	29,419
Nelson Pine Industries Limited	26,569
Wallace Group Limited Partnership	26,539
Pacific Steel (NZ) Limited	19,550
EVONIK PEROXIDE LIMITED	18,443
Daiken New Zealand Limited	17,770
Dongwha New Zealand Limited	16,854
Status Produce Limited	15,496
Taranaki By-Products Ltd	14,197
Exception Limited	11,618
Tuakau Proteins Ltd	11,393
Anchor Ethanol Limited	10,784
Southern Paprika Limited	10,406
Alliance Group Limited	10,012
Affco New Zealand Limited	9,465
Under Glass (Karaka) Limited	7,574
Gourmet Mokai Limited	7,006
Under Glass (Bombay) Ltd	6,194
Websters Hydrated Lime Company Limited	5,999
J.S.Ewers Ltd	5,853
Hawkes Bay Protein Limited	5,740
CMP Canterbury Limited	5,470
Juken New Zealand Ltd	5,304
Gourmet Paprika Limited	4,837
PVL Proteins Limited	3,632
Fletcher Building Products Limited	3,344
Sharma Produce Limited	2,677
Gourmet Waiuku Limited	2,190
Kakariki Proteins Limited	2,037
Shipherd Nurseries Limited	1,900
Island Horticulture Limited	1,717
Tegel Foods Limited	1,632
Value Proteins Ltd	1,581
Whakatane Growers Limited	1,393
P H Kinzett Ltd	1,344
Moffatts Flower Company Limited	1,182
Karaka Park Produce Limited	1,169
Van Lier Nurseries Ltd	1,123
Taylor Preston Limited	1,104
Meenakshi Devi Sharma, Raj Kumar Sharma	1,080
Vege Fresh Growers Limited	1,075
Jai Shankar Growers Limited	928
Prime Range Meats Limited	928
Homestead Produce Ltd	881
Sinai Hort Limited	604
J.S. Mahey Limited	599
Castle Rock Orchard Ltd	564
Karamea Tomatoes Limited	526
Poppas Peppers 2009 Limited	351
Taaza Green Limited	337
Harbour Head Growers Ltd	261
Ting-Yuan Robert Wu	239
Parkgard Growers 2000 Limited	222
Antone James Ivicevich, Joanne Elizabeth Gould Ivicevich	210
Graeme Lowe Protein Limited	198
Mary Jane Fausett, Peter James Fausett	143
Pomoana Gardens Limited	100
John Hamilton Charles Falloon, Paul Gregory Whitehead	79
Royal Roses Limited	66
Kingbridge Ltd	61
Eseta Kovati, Reupena Kovati	37
GELITA NZ Ltd	29
Wallace Corporation Limited	0

New Zealands biogenic methane emissions and the not zero target

In the recently unveiled Zero Carbon Bill, or more correctly, the Climate Change Response (Zero Carbon) Amendment Bill, biogenic methane emissions get a separate and distinctly non-zero emissions reduction target.

What are these methane emissions? I thought I would do some graphical analysis.

First of all - definitions. The draft bill in Section 4 amended (Interpretation) defines biogenic methane as:

"...all methane greenhouse gases produced from the agriculture and waste sectors (as those sectors are defined in the New Zealand Greenhouse Gas Inventory)".

The New Zealand annual greenhouse gas inventory records three sub-categories of methane; methane from enteric rumination, methane from agricultural soils and methane from the waste sector. Waste sector emissions in 2017 were 95% methane, 4.12475 million tonnes and 5% of total gross emissions. That's a significant chunk of New Zealand's GHG emissions.

Why is the waste sector methane included? It's not sourced from agriculture. None of the advocates of the split-gas target, like David Frame or Simon Upton, made any arguments about waste sector methane. Also, the Productivity Commission in it's 2018 report Low Emissions Economy thought waste sector emissions could be feasibly reduced. (See page 451 of their final report; "Waste also represents a major mitigation opportunity.")

The methane from enteric rumination, from agricultural soils and from the waste sector and the total biogenic methane look like this.

The total biogenic methane emissions in 2017 was 40.34134 million tonnes. It's a crude point so here's a crude pie graph to make it, 40 million tonnes of biogenic methane is half of gross emissions (excluding landuse and forestry).

40 million tonnes of biogenic methane is more than twice the quantity of net emissions (including landuse and forestry). Net emissions are 41% of biogenic methane emissions.

This line graph shows how the biogenic methane emissions compare to gross and net emissions over the 1990 to 2017 period. In the early 1990s, biogenic methane emissions exceeded net emissions.

The amount of biogenic methane emissions exceeds all emissions attributed to the agricultural sector. So the 'split' gross target, in carbon dioxide equivalents, is less than a counterfactual target where all agricultural sector emissions are excluded.

For context, here is my preferred graph of all New Zealand's greenhouse gas emissions by sector.

This next graph below shows actual emissions and the split gas targets; biogenic methane to 10% less by 2030, and to either 24% to 47% less than 2017 by 2050.

Net emissions in carbon dioxide equivalents, after taking out methane, go to...well..zero of course in 2050. I have assumed that carbon sequestration from the land use and forestry sector is the same in 2050 as it was in 2017.

Note that from 1990 to 1995, the net emissions were less than the 2050 net zero carbon dioxide target. In other words, under the net zero split gas carbon dioxide/methane targets definition, New Zealand's greenhouse gas emissions were negative. Can anyone credibly argue that New Zealand's greenhouse gas emissions in the early 1990s were 'net zero'?

Now this final graph shows the targets scaled up to include gross emissions as well as net. Gross emissions must decline at the same rate as net emissions. As the difference between them is the carbon sequestration from land use and forests.

The key point here is that if the sequestration in 2050 is about 23 million tonnes (the same as in 2017) then the New Zealand 'net zero' target for gross emissions, is 50 million tonnes. That's still a heck of a lot of emissions.

The net zero/split gas target is not nearly as stringent as all the pundits seem to think. It is hard not to conclude that the waste sector methane has been moved to the less stringent methane target in order to make the carbon dioxide target larger in 2050 and therefore easier to achieve. This, unfortunately, seems to be another example of New Zealand's favourite climate change practice; achieving targets by creative accounting instead of reducing emissions.

Charting New Zealand Greenhouse Gas Emissions by Sector 1990 to 2015

I have created a revised chart of New Zealand's greenhouse gas emissions analysed by economic sector for the years from 1990 to 2015. Something I have done before. Before that I made a chart of just the gross and net emissions.

This first image is an uploaded .png file at actual size (560 pixels wide) which is the width of the text container in the blog's template.

For a comparison, this second image uses Flickr's embed code to show a larger file (1280 pixels wide) I uploaded to Flickr. It's not a very large file; 128 kilobytes. That seems minute, when I am uploading 4MB or larger photographs to Flickr. If I wanted a larger file, I could output the chart from R as a .tiff format file.

The first smaller image uploaded to Blogger seems slightly clearer.

The data source is of course;

"New Zealand's Greenhouse Gas Inventory 1990–2015", Publication date: May 2017, Publication reference number: ME 1309, Full report - New Zealand’s Greenhouse Gas Inventory 1990-2015, and supporting tables and files. CRF summary data [Excel file, 45.6 KB]

The key difference from previous charts is that I have omitted gross emissions or as the Ministry for the Environment calls them "Gross emissions without Land Use, Land Use Change and Forestry (LULUCF)". Gross emissions frequently if not mostly seem to be the focus of analysis of trends and achievement of targets. Land Use, Land Use Change and Forestry emissions frequently get omitted.

I started with the sector emissions, then I added net emissions. Net emissions are the sum of the sectoral emissions obviously. Net emissions (with a qualification I may address in another post) are what end up in the atmosphere. So from a science-informed viewpoint, analysis of trends should be based on net emissions.

What struck me is that this format highlights a different interpretation of trends. Look how much of NZ's 1990 emissions were 'counter-balanced' by Land Use, Land Use Change and Forestry. In 1990, the land use and forestry sector sequestration (removal) of greenhouse gases was equivalent to the sum of the other sectors excluding agriculture. The 1990 net emissions were the same as the emissions from the agriculture sector. In other words, if you excluded agriculture emissions, NZ's emissions in 1990 would have been 'net zero'.

Since 1990, the land use and forestry sector sequestration has declined by 21%. In 1991, the land use and forestry sector sequestration 'counterbalanced' 100% of non-agriculture emissions. In 2015, the land use and forestry sector sequestration only counterbalanced only 57% of non-agriculture emissions. As long as land use and forestry sequestration is measured consistently over time, this trend can only get worse. A lot of commercial forest planting happened in the 1990s. These forests will soon be due for harvesting. That's why I want to scream each time I hear some pundit say forestry will be a 'get out of jail card' for growing emissions in other sectors, notably agriculture.

Here is the R script (with a couple of Linux Xterminal commands) for obtaining and preparing the data and for creating the chart.

The slow road to getting open data from the Government's Clean Water 2017 water quality monitoring sites

Who remembers the National Government's consultation over it's proposed Clean Water package 2017?

Who remembers the headline announcement of the proposal? - that there would be a 'target', that 90% of rivers and lakes would be swimmable by 2040?

The environmental NGOs were very critical of the target (and the proposal as a whole).

The Green Party said the new swimmable standard was just shifting the goalposts.

Forest & Bird's Kevin Hague described the proposal as a reduced swimmability standard.

Marnie Prickett of the Choose Clean Water group described the proposal as "fraud" as it intended to change the definition of swimmable to meet a lower standard.

The environmental NGO's argument was that the new proposed 'risk' standard for swimming (expressed in E Coli as an indicator of faecal matter and pathogens) allowed a one in a twenty probability of getting sick when the old standard was a much more precautionary one in a hundred probability of getting sick.

Dr Siouxsie Wiles and Dr Jonathan Marshall explained that the change in risk wasn't quite as simple as that. As did University of Auckland Professor of Biostatistics Thomas Lumley.

However, I thought there was something wrong with that 90 percent number. I seemed to recall Green MP Eugenie Sage saying in 2014 that more than 60 percent of the monitored river swimming sites were unfit for swimming.

The Clean Water package 2017 included this barchart which shows that the 90% 'swimmable' target (and five new swimming quality categories from 'excellent' to 'poor') are actually expressed in a different variable: length of river measured in kilometres (not in number of monitoring sites).

It also shows, in the left-most bar, that the use of the use of the 'length of river' variable in place of numbers of river monitoring sites, results in a very different result.

On the basis of recent data, 72 percent of kilometres of rivers currently meet the 'swimmable' standard (the sum of the 'Fair', 'Good' and 'Excellent' quality categories. Expressing the results in kilometres of river lengths and not in numbers of sampling sites immediately enables a more positive spin to be put on the results.

The underlying data must be water quality sampling results from NIWA's National Rivers Water Quality Network (NRWQN) and sites operated by regional councils.

So, way back on 15 March 2017, I asked for the underlying sampling data from the water quality monitoring sites.

I felt I had expressed my official information request sufficiently clearly to get a reply in a reasonable time.

On your website on the page "Clean Water package 2017" there is a bar chart explaining the target of 90% of rivers and lakes swimmable by 2040 included in the report "Clean Water, ME 1293". The bar chart is also on page 11 of report "Clean Water, ME 1293". The bar chart shows kilometres (which I assume are lengths of segments of rivers) in each of the five 'quality' categories (Poor, Intermittent, etc) with a time variable which has three bars; "Current", "2030" and "2040".

Will you please provide me with the underlying data; which I assume must be water quality monitoring site results (and future predictions for 2030 and 2040) analysed by the five quality categories and the three time categories "Current", "2030" and "2040". Will you also please include the name or number of each monitoring site, its region and for the "Current" selection, the sampling period for the actual E Coli counts. Please provide this data either in comma separated values or Excel 2007 format via the FYI website.

However, I had to lodge a complaint with the Office of the Ombudsmen to eventually obtain the data. That only happened after the investigator from the Office of the Ombudsmen brokered a deal with the Ministry for the Environment. He rang me and said that the Ministry didn't want to give me the data in either .csv or .xls format as I'd requested as the data was in a special binary format; .rdata, specific to a certain statistical programming language named after the letter 'R'.

In other words, it appeared to me the Ministry were claiming that a 'technical' problem in providing me the data I had requested, and not a problem of intent to frustrate the information request.

Sure, it's fair enough to take the Ministry at their word that they didn't intend to delay and frustrate my request. However, whatever the intention, it was still a delay from my point of view as the requester.

I told the investigator I would be happy to get the data in .rdata format. I also expressed the view that it would have only been a very short line of 'R' script to convert the .rdata formatted file into .csv format. And that it was a weak reason for the delay and for not providing me the data in .csv format. I observed that the Ministry's response was pretty unsatisfactory from an open data perspective. The investigator said he couldn't comment on open data issues, as we were in an official information space.

I was finally emailed the data in .rdata format by the Manager, Executive Relations, on 5 July 2017.

I used this R script;

to write the .rdata file to a .csv file.

The .rdata file is WQdailymeansEcoli.rdata at Google Drive.

The .csv format file is WQdailymeansEcoli.csv at Google Drive.

Now I just need to find the time to analyse the sampling sites data.

New Zealand greenhouse gases by sector from the inventory

I have made another chart from the New Zealand's Greenhouse Gas Inventory 1990–2015 released the other week by the Ministry for the Environment.

It shows the greenhouse gas emissions by sectors of the economy. It includes 'negative' emissions, more properly called carbon removals, or carbon sequestration or simply carbon sinks. This is the sum of all the carbon dioxide taken out of the atmosphere by the sector of the economy called Land use, Land use change and Forestry.

Here is the chart.

This time I took a more traditional R approach to getting the the data into R from the Excel file CRF summary data.xlsx.

First, I used opened an X terminal window, and used the Linux wget command to download the spreadsheet "2017 CRF Summary data.xlsx" to a folder called "/nzghg2015".

I then used ssconvert (which is part of Gnumeric) to split the Excel (.xlsx) spreadsheet into comma-separated values files.

The Excel spreadsheet had 3 work sheets, 2 with data, and 1 that was empty. So there's now a .csv file for each sheet, even the empty sheet. And we read in the .csv file for emissions by sector.

The final step is to make the chart.

The latest inventory of New Zealand's greenhouse gases

On the Friday before last Friday, the 26 of May 2017, Minister for Climate Change Issues, Paula Bennett and the Ministry for the Environment released the latest inventory of New Zealand's greenhouse gases.

Minister Bennett and the Ministry have as their headline Greenhouse gas emissions decline.

I thought would I whip up a quick chart from the new data with R.

I pretty much doubted that there was any discernible decline in New Zealand's greenhouse gas emissions to justify Bennett's statement. We should always look at the data. Here is the chart of emissions from 1990 to 2015.

Although gross emissions (emissions excluding the carbon removals from Land Use Land Use Change and Forestry (LULUCF)) show a plateauing since the mid 2000s, with the actual gross emissions for the last few years sitting just below the linear trend line.

Gross 2015 emissions are still 24% greater than gross 1990 emissions.

For net emissions (emissions including the carbon removals from Land Use Land Use Change and Forestry) the data points for the years since 2012 sit exactly on the linear trend line. Net 2015 emissions are still 64% greater than net 1990 emissions.

There was of course more data wrangling and cleaning than I remembered from when I last made a chart of emissions!

The Ministry for the Environment's webpage for the Greenhouse Gas Inventory 2015 includes a link to a summary Excel spreadsheet. The Excel file includes two work-sheets.

One method of data-cleaning would be to save the two work sheets as two comma-separated values files after removing any formatting. I also like to reformat column headings by either adding double-speech marks or by concatenating the text into one text string with no spaces or by having a one-word header, say 'Gross' or 'Net'.

Of course, that's not what I did in the first instance!

Instead, I copied columns of data from the summary Excel sheet and pasted them into Convert Town's column to comma-separated list online tool. I then pasted the comma-separated lists into my R script file for the very simple step of assigning them into numeric vectors in R. Which looks like this.

Then the script for the chart is:

The result is that the two pieces of R script meet a standard of reproducible research, they contain all the data and code necessary to replicate the chart. Same data + Same script = Same results.

I also uploaded the chart to Wikimedia Commons and included the R script. Wikimedia Commons facilitates the use of R script by providing templates for syntax highlighting. So with the script included, the Wikimedia page for the chart is also reproducible. Here is the Wikimedia Commons version of the same chart.

For comparison, here is my equivalent chart of greenhouse gas emissions for 1990 to 2010. Gross emissions up 20% and net emissions up 59%.

What can I say to sum up - other than Plus ça change, plus c'est la même chose.

Graph of atmospheric carbon dioxide concentrations from another cool data package

I feature another cool self-updating data package, this time of concentrations of atmospheric carbon dioxide recorded from the well-known Mauna Loa Observatory, in Hawaii. Graphs of this data are perhaps the most iconic images of anthropogenic climate change.

This post features the atmospheric carbon dioxide data package. Again, it is one of the Open Knowledge International (OKFN) Frictionless Data core data packages, that is to say it is one of the

"Important, commonly-used datasets in high quality, easy-to-use & open form".

The data is known as the Keeling Curve after the American chemist and oceanographer Charles Keeling. It is an iconic image for anthropogenic climate change.

Like the global temperature data package, the atmospheric carbon dioxide data package is open and tidy and self-updating and resides in an underlying Github data package .

Similarly, the data package can be downloaded as a zip file and unzipped into a folder. That will include the data files in .csv format, an open data licence, a read-me file, a json file and a Bash script that updates the data from source.

I can run the Bash script file on my laptop in an X-terminal window and it goes off and gets the latest data and formats it into 'tidy' csv format files.

Here is a screenshot of the script file updating and formatting the data.

Here is my chart.

Here is the R code for the chart.

2016 the warmest year on record via a cool self-updating data package of global temperature

Radio New Zealand reports that 2016 was the new record warmest year in the instrumental record, so I will pitch in too. But with an extra touch of open data and reproducible research.

It's been a while since I uploaded a chart of global temperature data. Not since I made this graph in 2011 and then before that was this graph from 2010. So it's about time for some graphs. Especially since 2016 was the world's warmest year as well as New Zealand's warmest year.

When I made those charts, I had to do some 'data cleaning' to convert the raw data to tidy data (Wickham, H. 2014 Sept 12. Tidy Data. Journal of Statistical Software. [Online] 59:10), where each variable is a column, each observation is a row, and each type of observational unit is a table. And to convert that table from text format to comma separated values format.

I would have used a spreadsheet program to manually edit and 'tidy' the data files so I could easily use them with the R language. As Roger Peng says, if there is one rule of reproducible research it is "Don't do things by hand! Editing data manually with a spreadsheet is not reproducible".

There is no 'audit trail' left of how I manipulated the data and created the chart. So after a few years even I can't remember the steps I made back then to clean the data! That then can be a disincentive to update and improve the charts.

However, I have found a couple of cool open and 'tidy' data packages of global temperatures that solve the reproducibility problem. The non-profit Open Knowledge International provides these packages as as part of their core data sets.

One package is the Global Temperature Time Series. From it's web page you can download two temperature data series at monthly or annual intervals in 'tidy' csv format. It's almost up to date with October 2016 the most recent data point. So that's a pretty good head start for my R charts.

But it is better than that. The data is held in a Github repository. From there the data package can be downloaded as a zip file. After unzipping, this includes the csv data files, an open data licence, a read-me file, a .json file and a cool Python script that updates the data from source! I can run the script file on my laptop and it goes off by itself and gets the latest data to November 2016 and formats it into 'tidy' csv format files. This just seems like magic at first! Very cool! No manual data cleaning! Very reproducible!

Here is a screen shot of the Python script running in a an X-terminal window on my Debian Jessie MX-16 operating system on my Dell Inspiron 6000 laptop.

The file "monthly.csv" includes two data series; the NOAA National Climatic Data Center (NCDC), global component of Climate at a Glance (GCAG) and the perhaps more well-known NASA Goddard Institute for Space Studies (GISS) Surface Temperature Analysis, Global Land-Ocean Temperature Index.

I just want to use the NASA GISTEMP data, so there is some R code to separate it out into its own dataframe. The annual data stops at 2015, so I am going to make a new annual data vector with 2016 as the mean of the eleven months to November 2016. And 2016 is surprise surprise the warmest year.

Here is a simple line chart of the annual means.

Here is a another line chart of the annual means with an additional data series, an eleven-year lowess-smoothed data series.

Here is the R code for the two graphs.

Opening up the Ministry for the Environment data and webscrape the 2015 free allocation of emission units

Let's look at the latest data on the very generous free give-aways of emissions units to emitters made by the New Zealand Ministry for the Environment
N.B. Update on 10 December 2016. The allocation decisions have moved to the web page of the Environmental Protection Authority.

The Environmental Protection Authority now hosts the 2015 Industrial Allocation Decisions that show the final free allocation of emission units to emitters for 2015 under the New Zealand Emissions Trading Scheme.

The New Zealand Ministry for the Environment no longer hosts the unit allocation data and the old link returns an Acess Denied page.

I looked at the 2010 to 2014 data in my post Opening up the data on emissions units in the NZ emissions trading scheme. So in this post I am will repeat my steps in web-scraping the freebie emissions unit data into a sensible open-data format (but with the links updated to the EPA).

The url of the old Ministry for the Environment web page is http://www.mfe.govt.nz/climate-change/reducing-greenhouse-gas-emissions/new-zealand-emissions-trading-scheme/participatin-4

The url of the EPA web page is http://www.epa.govt.nz/e-m-t/taking-part/Industrial-allocations/allocations-decisions/Pages/decisions-2010.aspx. And unfortunately, the Google sheet 'scrape the table' script does not seem to work with the EPA page.

Go to Google and open a new Google sheet.

Following the tip from the School of Data Liberating HTML Data Tables, enter this text in cell A1 of the Google sheet.

=importHTML("","table",1)

Add the url of the Ministry for the Environment's free allocation web-page between the double speech marks so you have this exact text in cell A1.

=importHTML("http://www.mfe.govt.nz/climate-change/reducing-greenhouse-gas-emissions/new-zealand-emissions-trading-scheme/participatin-4","table",1)

It was good thing that I kept a screen shot to show that it worked perfectly! We now have a Google sheet of the 2015 free unit allocation to NZ emissions trading scheme emitters.

I have saved it as NZETS-2015-final-allocations-for-eligible-activities.

However, the data does not have a "tidy" structure, where each variable is a column and each observation is a row (Wickham, Hadley . "Tidy Data" Journal of Statistical Software [Online], Volume 59, Issue 10 (12 September 2014)).

The first column includes both industry names and types of industries classified by the type of emissions the industry produces. And lots of asterisks. A tidy format would have these attributes (or variables) as separate columns so that each company/emitter would have a row each.

I used a programme called Open Refine (which is also at Github) to data-wrangle the data into tidy format and to save it as a comma-separated values file which is this Google sheet NZETS-2015-final-allocations-for-eligible-activities. Its a bit fiddly using Open Refine, and I have not documented the steps. I won't describe how I did it. Yes, I know, from the point of view of reproducing the tidied data I should have done the tidying with a script or code. Next time I will.

As usual, the big emitters get the most emission units! Of 4.417 million units allocated to industries, 90% went to 11 large companies. New Zealand Steel Development Limited, of arbitrage profits fame, gets 1,067,501 free units. New Zealand Aluminium Smelters Limited gets 772,706 free units.

This is the updated free emission unit allocation data from 2010 to 2015.

I did a bit of data visualising with the 2015 data and created this pie-chart in R programming language.

The R script for that is:

Did I not get the End the Rainbow memo? So I picked a better colour scale from Colour Brewer.

The R script for this non-rainbow pie chart is: