Despite it being four weeks since we last got in touch, our development Trello boards, unlike our information management boards, remain modestly silent on matters of progress. This is mostly because complex plans go out of the window when chaos strikes.
Our Jianhan was supposed to be concentrating all efforts on upgrading our taxonomy software. At least that was the plan that was planned at our last planning session. Unfortunately, events intervened when our internal version of data.parliament popped, banged and conked out. This would not have been such a big deal were it not a critical component in the publishing of briefings from our three research services. Which meant researchers in all three services pressed publish and nothing got published. This happened to coincide with a couple of days of Commons Library contingency planning, so that at least was fortuitous. Nothing makes wargaming quite so much fun as getting shot at.
The outage lasted three days before Jianhan took matters into hand. Memory has been now been increased to the maximum possible. Should that box pop or bang or conk out again, we have a much bigger problem. Jianhan continues to investigate, an upgrade to the antiquated triplestore looking not unlikely.
Sticking with the subject of slightly archaic software, our procedure editor application is also getting a little long in the tooth, being untouched by developer hand for longer than might be considered ideal. And I think we all know how software ages. For those unfamiliar, procedure editor is the application used to manage both our marvellous maps and the business items in work packages. Should it ever fall over, we’d be left with neither a statutory instrument service nor a treaty tracking service. Not to mention the all too early demise of our Procedure Browsable Space™.
For that reason, Librarian Jayne and her computational lapdog Michael have been hard at work investigating whether procedure editor could be replaced by an off the shelf solution. That solution going by the name of Data Graphs. Before any such decision can be made, testing needs to take place, and testing can only take place with data. Over the course of several dozen meetings, Jayne and Michael have been writing a whole bunch of SQL to export relationally-shaped data, in order to load it into its more graph-shaped proposed new home. SQL that will, in theory, make the whole process easy to replicate should the decision to move house be answered in the affirmative.
As of this week, they are pleased to report that the ‘reference’ side of the procedure model - so procedures, routes, steps, step types, step collections, houses, and legislatures (everything that Michael’s eyes believe to be green on our data loading diagram) is now populated. Just papers, work packages and business items to go. They promised they’d have the data fully loaded by Christmas, ready to test in the new year. At this point, that promise feels a little touch and go, but they might just make it.
In the meantime, librarian work continues. At some point earlier this year, the team began to record the reason why the Secondary Legislation Scrutiny Committee has drawn the attention of the House to a statutory instrument - previously noting only that attention had been drawn. This left something of a gap for instruments laid between late 2017 and early 2025. A gap that has now been plugged by Librarian Emma. Top work Emma.
In our final bit of Procedure Browsable Space™ news, that website is the latest to leave our corporate colours paint shop, and pretty impressive it looks too. Or at least Michael thinks it does. Librarian Jayne’s testing ruler awaits deployment.
Our crack team of librarians have also been cracking on with improvements to our Parliamentary Search service. Breaking out her indexing pencil, Librarian Emma has added tags - to augment the free text description - to proceedings introducing or discussing new clauses and / or new schedules for bills.
Librarian Emma has also been busy over in the world of the Committee of Selection, plugging gaps in our data reporting resolutions of, and discharging of members from, the Committee.
Not content with that, Emma has been busy indexing ministerial corrections made as written statements rather than, er, as ministerial corrections. This new-ish indexing policy was put in place sometime around November 2024 and covers all such statements made since then. Always one for consistency, Emma has combed through 19 sessions worth of written statements, marking those issued as corrections appropriately.
Still with written statements, and still, remarkably, with Librarian Emma - how does she find the time? - she’s been adapting our data to reflect a new policy on written statements made in response to Opposition Day resolutions. The new policy sees the team supplementing the link between the debate and the statement with a short note describing the link between the two, thus making them a little more discoverable. Not nothing.
Stage left, Librarian Martin has been attending to government responses to select committee reports, reviewing corporate authors added over the last couple of sessions. Perhaps more impressively, he’s also gone back over 31 sessions worth of contributions made during points of order proceedings, tagging those deemed to actually be points of order by the Speaker. As one might expect.
Finally, Librarian Anya stumbled across a handful of General Committee proceedings which had accidentally found themselves with a committee in the department field. Librarian Martin has also now fixed that.
Further back, in the backend of indexing, changes have also been happening in our taxonomy of terms. In the first place, a number of names which had been used by our crack team of indexers but not controlled as concepts have been checked, doubled checked, and turned into controlled concepts as appropriate. Any affected indexings also being updated. Fine work from Librarian Ned.
In the second place, team:taxonomy have also been pondering the preponderance of Member identifiers being present in the taxonomy, picked up by the ORefIS service and inserted into the Search and Indexing triplestore. For reasons that should be obvious, some taxonomic terms are in use for more than one Member. This being true for both bishops and hereditary peers. In such cases, our taxonomy might carry up to four MNIS IDs on one concept. The next Bishop of Durham bringing that up to a round five. Following investigations by our Jianhan, it turns out that ORefIS only wrote the value in the first MNIS field. Should it find a different value in this field, the old code would delete from the triplestore and replace with the new value. Jianhan has now adapted this code to make the process additive, rather than a like-for-like replacement. Thanks Jianhan.
Behind the scenes, a fair bit of work has been happening on our psephological efforts, all in preparation for the next general election. Last time around we published only when all the results in all of the constituencies had been verified by our crack team of researchers. The next time out, we’re aiming for a four state solution:
At dissolution, placeholder pages for every election in every constituency, with some kind of ‘coming soon’ message.
Once nominations close and all candidates are gathered, the same pages, additionally listing candidates.
As soon as possible following the election, the same pages, listing winners.
Once all the results in all of the constituencies are verified, vote numbers for all of the candidates in each of the constituencies. At this point, the many and varied aggregations will also be made available.
To that end - ably assisted by Data Analyst Rachel - Michael has been modifying the psephology code and generating four different databases to reflect the four different states. These efforts are currently with Statistician Carl for final signoff.
With an optimistic hat on, we’re half aiming for a 4.5 state solution, whereby the full results for a given election in a given constituency are published as soon as those particular results are verified. Between now and then though, we need to replace the application used to gather election results. Whether or not the partially verified state is possible depends on how well that goes.
Whilst managing the data describing current Members is quite hard, doing that in real time in response to breaking news is harder still. There are some events - be that defections, deaths, resignations or suspensions - that we really can’t afford to miss. Given team:Phil is one whole librarian down at the moment, he’s taken the time to set up a suite of Factiva alerts, in the hope we’ll never be caught on the hop again.
Last time out, we reported that our new tweaty twacking code was considered complete, awaiting the laying of a treaty for final testing. Since then, not one, not two, not three, but five treaties have been laid. All finding their way unimpeded to both Bluesky and Mastodon. Jianhan’s old code is now switched off, Michael’s code having taken over the task.
If you tuned in back in week 40, you’ll know we’ve made some stumbling attempts to establish a small community of linked data system thinking types. With apologies for the words, ‘systems thinking’. Amongst the cast of stars involved was TNA’s very own Jenny Bunn. Jenny is thinking about the path to turning their catalogue into a knowledge graph. How, Jenny asked Anya, did you transition a team of librarians more used to bibliographic records toward a more graph-like approach to description?
Which is how Wednesday found Jenny, Jayne, Anya and Michael occupying a meeting room in Tothill Street and going into some depth on making and maintaining models, maps and databases with conversation, collaboration, documentation and software. We hope it was not too much depth, Jenny. But then, you know us. Monomaniacal, even on our best days.
Thursday of this week saw a pub-based meeting with Samu, ex of this parish. Guinness was drunk, Baby Guinness was drunk. Rachel came away with new ideas for new things. Anya came away with new ideas for new things. We miss you man.
