Anya, Jianhan, young Robert and bits of borrowed Ian have spent much of the past week investigating how we might ingest petition data into systems that allow people to search for them in a structured fashion, and our team of crack librarians to subject index and link them to pertinent parliamentary business. There has been a long standing desire from the Petitions Committee to have the petitions better described in data and this is one step on the road to that.
Whilst the petitions website is served from the Parliament domain, it’s built and maintained by our friends at the Gubmint Computational Section. So the mechanics were something of a mystery to us. A long while back, in a time far away, we began the process of ingesting the website JSON feed into the data platform. So Jianhan and Robert have been picking their way through data and code, writing a couple of applications and trying to make the numbers add up. It seems that there are petitions that appear on the website and which have been shunted into the data platform that are no longer present in the feeds. Archived or otherwise. A meeting with GDS has been pencilled in to explore why that might be. Jianhan has also taken the opportunity to fix a number of issues he found in the data platform orchestration process. Bugs that existed in the code and failed to deal with corner cases in the JSON data have now been squashed. And wonky data in the data platform resulting from said bugs, and mismatches between the source data and the data platform have been patched over as best possible. So we think we now have a pretty complete, well-modelled and fairly well-populated set of petition data. If not perfect.
The next step is slurping it back out of the data platform and into search and indexing, where our users can get at it. Jianhan spent much of last weekend building an Atom feed that picks data out of the platform and sends it toward the search and indexing harvester application. This is probably beyond the call of duty, but, let’s face it, there’s not much else to do. With welcome support from Sadia and Kunal, he’s now configured the harvester to turn the feed back into triples and insert them into triple store number two. So far we’ve only pushed five test petitions down that pipe, but they all seem to be turning up as expected. More progress next week we hope.
Following last week’s revelation from JO Jane that statutory instruments pertaining to money matters accidentally laid before the House of Lords, are not in fact withdrawn but expunged, Librarian Jayne and Michael had work to do. Encounters with clerks usually see us leaving the room with a feeling that everything procedural is more complicated than our tiny minds can cope with. So finding something that’s actually simpler than initial impressions makes for a very pleasant change. It is our usual practice, when mapping procedures, to start at a step and plot out what might happen next. But toward the end of any procedure, working backwards is the only reasonable response. One can almost hear Silver whispering about event storming at this point. We have a number of steps describing both the conclusion of a procedure and what might or must happen to the instrument next. And picking apart how you might get there is like knitting a jumper by first unravelling a scarf.
All of this led to Jayne and Michael spending a number of hours unpicking maps they’d mapped and, for once, simplifying the frankly horrendous knots of NOT steps, AND steps, OR steps and routes which, at the start of the week, looked like a cat had attempted knitting. On Thursday, they dragged young Robert into the fray. He was generally positive but did point out a couple of optimisations they’d missed. Clever clogs. The logical draft negative, made negative and draft affirmative maps are now redrawn and the new data has entered the machines. We are confident that all our logically mapped procedures now conclude appropriately. At least for this week.
A wee while back, Michael was staring at the procedure editor database schema and was surprised to find a join table between procedures and routes. Now, usually, Michael is a big fan of join tables. A big fan. But on this occasion, he found himself confused at how this had ever got there. That a procedure might have many routes was never in question. That a route might be in many procedures was more puzzling. Checking our model comments, all became clear. We had, it appears, always intended to reuse routes across maps, thereby defining procedural components that could be described once and used repeatedly. Like reference data, if you will.
Monday saw Librarians Anya and Jayne and computational experts Robert and Michael sit down for a chat over shared pixels. Michael went into the meeting thinking they had perhaps overreached - that no one could be expected to get their heads around maps with shared routes. He left the meeting understanding that our past selves had in fact presented our present selves with a procedural present. There being a number of maps where we had been copying and pasting chunks of procedure wholesale, with the data for these chunks being managed four, five or even six times. The presence of that blessed join table allows us to split out these components, describe them once and reuse everywhere.
So by Thursday we had maps of component procedures for EVEL certification, the Select Committee on Statutory Instruments, the Joint Committee on Statutory Instruments and the Secondary Legislation Scrutiny Committee. And by Friday, Jayne and Michael ensured our data had a single map of the SLSC procedure reused across four different procedures. The machines even being kind enough to draw it for us. Thanks machines.
Librarian Jayne noticed that a couple of our procedural steps made mention of legislation without properly citing it. Which is not like us. Not like us at all. Jayne has now fixed this. So for steps actualised by instruments introducing one - or indeed more - criminal offences, we now properly cite Schedule 7 of the European Union (Withdrawal) Act 2018. And for instruments made under the Sanctions and Anti-money Laundering Act, we now properly cite Section 56(5) of the Sanctions and Anti-money Laundering Act 2018. Top work Jayne.
For those unfamiliar with statutory instrument procedure - and let’s face it, that’s probably most of us - there are two common types. In the affirmative procedure, the government must table an approval motion which must be approved before the instrument can either become or remain law. In the negative procedure, the instrument will automatically become or remain law when the egg timer of scrutiny runs dry of sand, unless either House agrees a motion rejecting it.
In the House of Commons, such rejection motions are tabled as Early Day Motions - more usually the preserve of Members congratulating football teams. There are two types of motion that might be tabled as EDMs. The intention of a fatal motion is to kill the instrument, whereas a non-fatal motion does not kill the instrument but presents an opportunity for Members to make their views known.
Occasionally EDMs are tabled as fatal motions but aren’t scheduled for debate until all the sand is at the bottom of the egg timer. At which point the opportunity to kill the instrument has passed. Which means a question on a fatal motion can no longer be put.
Anya and Librarian Emma had both noticed that sometimes we were capturing the tabling of a fatal EDM and the debate and decision on something that looked very like that EDM, but past the point where a fatal motion could have any effect. Which was puzzling. So on Monday, Anya, Jayne and Michael sat down with Mr Hennessey from the House of Commons Journal Office to try and figure out what they were missing. It turns out that EDMs tabled as fatals aren’t automatically withdrawn once the period of fatality has expired. And that Members might instead table a similarly worded - but non-fatal - EDM, or table a non-fatal - but not an EDM - motion which first pops up on the Order Paper. In the latter case, our librarians had not always been spotting these. We now hope to get a warning whenever a non-fatal motion hits the Order Paper which should lessen the confusion considerably. Thanks Mike.
All but one of our procedure maps involve some kind of scrutiny period, as defined in legislation. The legislation defines the type of day to start counting from, the kind of days to count and how many days of that type should be counted - weeknotes passim and then some. We’ve had a couple of long standing questions about these clocks, roughly summed up as does the clock not having started preclude Parliament from acting, and if and when we should ever reset the clock end step.
After consulting with lawyers and clerks far and wide, we have arrived at the conclusion that clocks not having kicked off does not preclude anything. So Jayne and Michael have now removed all clock start steps from all procedure maps and data - both logical and non-logical - and our Jianhan has also removed the code that actualised the clock start steps. Thereby reducing both noise and effort. Thanks for helping us to answer that question go to Alex, John and Luke. We cannot promise we will not have more questions in the future.
The question of resetting clock end steps is still a little up in the air, so we’ll save that as a treat for next week. Do tune in if you can.
A few weeks back we reached a frankly impressive milestone on matching IDs for our thesaurus terms to items in wikidata. Plans for phase two were made.
We have been quiet on this front but not not busy. Wikidata links to the public home of our thesuarus on data.parliament. Matching things means checking things … and the data feed from the application used by team Liz to manage our thesaurus to its public home on the web has long been troubled. So long and so troubled that we were no longer sure about what does and doesn’t happen, never mind what should and could happen. So Librarian Liz and Sadia popped on their overalls, polished their wrenches and took a long hard look at the plumbing. Together they cracked the first question: we think we can say we now have a scheduled update, which will run monthly, and completely. For some definition of completely: we’ve established that the update does not include terms deleted or demoted. Liz estimates the public version might have around 4,000 zombie terms. A meeting to grapple with the could and should question is in our diaries for next week.
Librarian Ned and Michael have continued to chip away at David’s peerage data in the hope of turning it into a shape that can also cope with subsequent holders of a title. To that end, Ned has started to compile lists of subsequent peerage holders which Michael has now ingested into the database. The Barony of Blythswood being one such delightful example. This week they’ve also been trying to generate ordered lists of letters patent. Results so far, being mixed. The ordering is still not quite right and the code is quite dreadfully slow. But they hope with a dab of downcasing they can fix the former and more efficient queries should definitely help with the latter. They’d both like to thank Grant from the College of Arms for answering peerage questions both broad and specific.