Our regular reader will be more than aware by now that a glitch in the system caused new Members to fail on their journey toward the data platform. Silver and friends at Data Language have been diving into the orchestration code to work out exactly what happened, how what happened might be patched over for now, and how it might be solved in the longer term. The suspicion had always been that a mismatch between the source model and our target model required both additional intervention and another source of data to ‘inflate’ the simplistic handling of House of Lords membership into something fit for assigning house seats to noble bottoms. And our Benjy had a repressed memory that SharePoint was somehow involved. This week we made real progress when Anya remembered that SharePoint had undergone a migration and it was more than likely that any side loaded data had moved house. Michael followed up with Liz who confirmed that the SharePoint lists had indeed been migrated and thankfully not deleted. At which point, Data Language Julian dived back in and, as if by magic, fifteen SharePoint lists emerged from the ether. They ranged from a list of Houses having the expected row count of two, to a list of Lords seat incumbencies with a grand total of 2629 rows. So now we know why, and when we run the orchestration into a clean triple store noble bottoms do indeed end up correctly seated.
We’re still less than clear on why the orchestration was turned off. Julian checked GitHub and found that Wojciech - ex of this parish - had commented out the line triggering the import of Member data on the 19th December 2018. Email archives have been peered at but nobody can remember much beyond a vague recollection that something had gone awry on the Lords side that may or may not have had something to do with names or maybe parties or possibly roles. If there’s anyone reading from the Lords with a better memory than us, please do get in touch.
In a better world, our source and target models would be aligned and no side loading of data would be necessary. This is our goal for the longer term. Business speak being business speak, we must of course be ‘going forward’ and taking ‘next steps’. For now then, we need to figure out if some combination of librarians and computational experts can somehow resurrect the interim solution and get the Member bit of the data platform back on its feet. More next week one hopes.
Our Jianhan has continued to make magnificent progress on his quest to add logic gates to our procedure maps. All steps currently existing have metamorphosed into business steps and it’s no longer possible to add a step without a type. With that, all the work required on the procedure editor database and application is now complete in both staging and production. Top work Jianhan.
Next week we should make a call on whether to plough on with getting the new model - and data - into the data platform. Or whether to instead concentrate on updating the procedure visualisations to take account of our new model. Actually, Jianhan should make that call. We’ll just go along with whatever he says. It’s safer that way.
Whilst Jianhan chipped away at code to support the new model, Anya, Jayne, young Robert and Michael screwed up eyes and scratched heads as they made their first attempt to transform an existing procedure map from one with route types to one with logic gates. First steps being tentative, they applied their brains to the simplest procedure we have - the proposed negative statutory instrument procedure as set out in the European Union (Withdrawal) Act 2018. To be fair to them, neither Robert nor Michael can be considered to be in the prime of their life. Mental faculties may well have faded with age. So we cannot pretend this was easy going for them - particularly as our procedure model took a turn toward ternary logic last week. They took things slowly, painstakingly sketching out arrows labelled TRUE or FALSE or indeed NULL, and pausing to point at the lines whilst muttering maths. Eventually patterns emerged as patterns often do and things became easier, or at least something like intuitive. And dare we say fun. One begins to understand why grown adults buy puzzle books and play games. Actually, there might be something in this. A ternary logic based procedure puzzle would make a delightful stocking filler for our clerkly colleagues. And if you doubt us, we doubt you’ve met a clerk. Anyway, our crack team have now completed almost one half of one procedure. Only six and a half to go. More next week.
In preparation for the transition to logic gates, Librarian Jayne and Michael spent a little time with the PNSI procedure map and questioned why they’d added requires routes from government statements to decisions being taken in both Houses. The appropriate legislation was checked and the routes found to be somewhere between unwarranted and misleading. So this week they took them out.
Last week we sent a couple of emails to CRAGy colleagues asking whether our treaty map should feature the opening of calls for evidence by the EU International Agreements Sub-Committee and the possibility of a government response to a decision by the European Union Select Committee. Dominique got back in the affirmative on the first question and Alex did likewise on the second. So Jayne and Michael have updated both map and data to allow for IAC evidence calls and more in the way of government responses.
On the subject of committee decisions, our team of crack librarians had a chat about why we’d used the words ‘decision made by committee X’ rather than ‘report by committee X’, given reports are where one finds decisions. ‘Report’ feels truer to the way people talk about such things but, as Librarian Emma pointed out, could sow confusion between a committee publishing a report and a committee reporting an instrument. So no change has been made. Though it’s possible JO Jane also has views here.
And on the subject of JO Jane, it turns out she actually works for the Legislation Office and not the Journal Office. Which, it also turns out, everyone else knew but had not seen fit to tell Michael. We’re sure our reader will agree that LO Jane does not have the same ring to it. Or any ring. For the purposes of weeknotes we intend to prioritise continuity and weak puns, and will continue to refer to JO Jane. Unless she complains.
Last week our friends in Oxford got in touch to say they’d released a new version of their House of Commons public Standing Order data. Which is of interest to us because it allows for applying identifiers to standing orders and fragments of standing orders irrespective of their past, current and future position in standing order publications. Which means we can cite such things without having to invoke their numbering system. Parliament has something of a habit of conflating the naming of things with the numbering of things with the identification of things which makes reliable linking harder than it should be. Please never sit us down with a drink and get us started on the citation for a session. We would bore your ears off.
Michael has taken the new data set - now formed of three sheets rather than one - and imported it into our database. There is now a mechanism to reliably link to standing order lists at the point of adoption, to standing orders and fragments in adoption lists by anchor links, and to standing orders and fragments in a fashion that’s completely independent of list positioning. Which means in the future we should be able to link to standing orders from procedure steps and routes without those links breaking every time the list is reordered. Young Robert chipped in by retuning our application to https, so no one gets to interfere with our orders. Thanks Robert.
The order view lists versions of orders and, for each version of an order, the fragment versions it contained or contains. Which means it’s all a little slow. Some denormalisation is probably called for but one never denormalises before first putting everything in its right place. As every fule kno. We would usually provide links as proof of work at this stage but unfortunately, the Heroku instance our application runs on is not one we’ve chosen to empty our purses for. Which means it’s run out of tokens again and refuses to work. It should be back up in time for Advent calendar season. At which point, we’ll link to it.
There is still work to do to reduce screen clutter and only list order versions where an edit has taken place. Michael got in touch with David, Paul and Matt to check whether an edit to change case - for example from ‘the Government’ to ‘The Government’ - would be considered substantive. It was thought not. So next week probably involves some down-casing and some text comparison. And hopefully some less cluttered lists.
Librarian Jayne and Michael attempted to stretch their limited brains round another mail from Mr Evans. This time on the question of questions not being decided. A couple of small matters were cleared up, again with help from David and Matt. They have reconciled questions not decided as a result of being talked out and questions not decided on the grounds of being inquorate into a single step. And made the talking out of motions into a thing that’s allowed from the moment the motion is moved. At least so long as another motion to exempt the first motion from not being decided past the moment of interruption has not been agreed. Quite remarkable how Parliament nudges you in the direction of the triple negative. Jayne and Michael both look forward to redrawing their motion map with logic gates. They really do.
Jayne has continued to add comments to our library of SPARQL queries. This week for all queries relating to debates on statutory instruments. We all hope Tony appreciates her efforts.
Shortly after luncheon on Thursday, Anya, Jayne, Robert and Michael met with JO Jane to chat next steps for the egg timer. They agreed to test further before advertising more widely, on the degree of improvisation that should take place in our calendars given the unknowability of the future, and that eventually calendar maintenance should move out of the Library and into the Journal Offices.
Michael had planned to talk Jane through the calculation code but realised just before the meeting that the comments were silent on the matter of the timer for made instruments being turned over at making rather than laying. This would most definitely have resulted in one of those moments where JO Jane asks, “have you taken any notice of anything I’ve ever said?” So he backed out gracefully and collared Robert to help him fix up comments and messaging to reflect a little more making and a little less laying.
Our only other change was a tweak by Robert to add IDs to calendar dates so anyone can now link to one. Nice work Bobs.