Our dear reader will be aware that we are as reliable at abiding by our publication schedule as [insert name of well-known parcel delivery company]. That said, nine weeks with no weeknotes is remarkable even by our standards. As ever, we have our excuses.
Our dear reader will also be aware that we are not, in fact, civil servants. That we can order a pint without forming a queue giving that game away. Not being civil servants, we are not - officially speaking - subject to purdah. But, as yet another Parliament dissolved and the House of Commons Library went dark, it was felt best all round to go dark with them.
Which leaves us in the unfortunate position of trying to cram nine weeks of work into a single set of notes. So, we’re setting aside our style guide - ‘wittering on’ - for something more succinct. Even so, this remains something of a ‘long read’. Timings may be somewhat awry, events and attributions may be missing. It’s rather hard to remember everything.
It’s rare for our top librarian trophy to go to a non-librarian. The chief exception we can recall being our Jianhan, who’s managed to a score a brace. This time our prized pewter pot goes to Data Scientist Louie, for masterminding the Library’s general election data gathering efforts. From assembling a crack team of candidate data gatherers, to leading the team through election night, to organising the follow up operation to verify the numbers, to piping data into Research Briefings, election websites and procedural systems, he played an absolute blinder. Well done, Louie.
It goes without saying that our attentions have been firmly focussed on dissolutions, general elections and new Parliaments.
Back in week 22, Data Scientist Louie confirmed that new constituencies would not adversely impact the entry type in the Members Names’ Information Service (MNIS) for any returning Members. And that non-returning Members flagged as standing down would no longer be assigned to the ‘retired’ pile. Both nice things to know. Having confirmed the last bit, our crack team of librarians were free to finish fixing up our standing down data, which is now considered correct back to Parliament 53. Preparations complete, Librarian Phil stayed up extra late to click the ‘dissolution’ button in MNIS and Parliament 58 was closed. At least computationally speaking. The actual event happening slightly later in the day, behind closed doors and involving a seal or something. No ill-effects were noticed as a result of the computational dissolution, though - as ever - there was one thing we’d not thought of, Mr Speaker being accidentally stripped of his role - computationally speaking - for a few minutes, before the issue was noticed and resolved. One lives, one learns, one adds to the manual.
By week 24, Parliament 58 was but a distant memory, as preparations for the general election shifted up a gear. The election saw constituencies contested by candidates certified by parties we’d never seen before. And some we had, but with new names. By the end of the week, statements of nominations had been finely-combed and all new parties were in MNIS. There was merest hint of a kerfuffle around how candidates standing under the ‘Patriots Alliance’ should be filed, but Statistician Carl and Election Legislation-specialist Neil stepped in and solved that one. Not quite election related, but still with MNIS - all paths lead to MNIS - Librarian Anna finished up fixing end dates on constituency areas relationships with wider geographies. One more table slightly tidier. In amongst all of this, Data Scientist Louie’s crack team of volunteers were poring over the many and all too varied local authority websites - and dear old Democracy Club - perusing statements of persons nominated and entering the details into our election candidates database.
Week 27 was, of course, the big one. It all started quite gently, when some new party colours arrived from Statistician Carl for entry into both MNIS and our election results website. Then Thursday happened. And Thursday night. And Friday. 24 volunteers descended upon Dartmouth Street. Televisions were tuned, wireless sets fired up and websites dereferenced. First pass figures for vote counts, electorates and declaration times were entered and the all important winning candidate radio button clicked. The evening started off quiet enough, reaching the expected crescendo around 2am, when the pizzas arrived and data entry hell broke loose. And continued to break loose until dawn. As the normal working day began, their night’s labours ended, Librarians Anya, Ayesha and Phil and computational guesspert Michael did what they usually do at moments of high stress and headed off to a public house. Leaving poor Louie behind to pick up the last remaining pieces. Not that Librarian Phil is one to shirk. Pint sunk, he headed back to the office to lend Louie a helping hand with data upload tasks, not forgetting to flip the mission critical ‘party of government’ flag. Librarian Phil, we salute you.
Now you might think that week 28 would have proceeded at a more gentle pace, but, reader, it did not. Post election means new government and opposition positions to be added, photographs of Members of the new Parliament to be acquired and uploaded, and unsuccessful election records for defeated Members to be created. Library life being Library life, it also saw the minting of a slew of identifers for new Members, a single identifier never being quite enough. Team:Phil added entries for new Members to our taxonomy service, generated PIMS identifers for each of them and then added said identifiers to MNIS. PIMS identifiers being identifiers for a now defunct service, nobody being quite sure if we generate the IDs because we have to or because no one has told us to stop. It is, quite obviously, problematic to have this many identifiers floating around with no means to reconcile them. To that end, a whole bunch of rdfs:seeAlso statements were also added to our indexing system, nothing appearing to go wrong.
Away from the old computers for a second, Louie’s crack team of volunteers were also hard at it throughout weeks 28 and 29, contacting returning officers by email, telephone, and on one occasion visiting a town hall in person, to verify vote count and electorate figures. By Thursday of week 29, we finally had verified figures for all 650 elections. Data Scientist Louie popped them all into the candidates database, exported some XML, added a bunch of identifiers and passed them over to Librarians Anna and Emily. Who added a bunch more identifiers and passed them on to Carl and Michael. By the end of the day, Carl had pressed publish on his Research Briefing and Michael had uploaded to both our election results website and our lovely little Datasette offering.
All of which definitely deserves a pint. We think.
Psephologically speaking, week 21 started badly when the ONS released new geographic codes for a handful of Scottish constituencies which had been untouched - in name and shape - by the boundary changes. For similarly ‘unchanged’ constituencies in England, Wales and Northern Ireland, the ONS had minted new codes. In Scotland, for reasons we do not pretend to understand, they had not. Until they did. But only for five of them. You had one job, Michael was heard to mutter before issuing a quick update.
Week 22 was a big one, rake task runner Michael emerging crumpled from pillow, chugging on a breakfast cigarette and typing ‘rake dissolution’ into the first terminal he saw. Another Parliament 58 in another database was closed, four old boundary sets flickered out of existence and four new ones were born. For the first time in quite a while, our psephology schema gained a whole new table, this one describing the ordinality of a general election in a given boundary set. A table that has already been put to use in attempts to describe why some fields are based on notional results in some boundary sets, but not in others.
Week 24 saw the addition of more general election CSV downloads, this time covering general elections at country level. We hope they are of some use to someone. Week 24 also saw Young Robert emerge from under a cloud of ‘blasted browsers’ and finally fix the party table CSS rendering bug that had been plaguing desktop Chrome users everywhere. Top work, lad. Equally exciting, some fine work by Data Scientist Louie, built atop some equally fine work by Open Innovations, saw our general election pages get the hexmaps they deserve. Lovely they look too. Young Robert and Michael took the data that sits behind our election results website, converted it to SQLite and published the results as a Datasette. Not bad for an hour’s work.
Week 25 began with design tweaks to both banners and navigation, general election default views changing to party listings. Behind the scenes, 2019 notional results were tweaked a little to include the correct Yorkshire party and the English and Welsh Green Party parted company with the Scottish Green Party. Michael made a start on the 2024 general election results upload script. Testing the thing proved to be problematic. With new constituencies, a different number of constituencies in countries, less than reliable geographic codes, new parties and new Members to deal with, the generation of what the young people call ‘synthetic data’ was hard to imagine. We’ll have time to test between the election and the results verification, said Michael. Although, I’ll be on vacation, he added. Snatching victory from the jaws of defeat, Librarian Susannah managed to rope in Shedcode James to lend Young Robert a hand in Michael’s absence. A more competent hand than Michael’s, to be honest. Upload script in place, thoughts turned to how we might issue corrections after the fact. A meeting was convened, the schema picture was pored over and fields we think we may need to update made bold. A new field capturing Democracy Club person identifiers got added to our candidacy table which we hope might make the mission easier.
Tuesday of week 26 saw Young Robert and Michael meet with Computational Section colleague Mary about improving the design of our election results website. Until that point, we’d been operating under the slightly vague instruction to ‘make it look like a Parliament website’. No easy task with no formal design system in place. The meeting went well and a plan was hatched. Improvements to the main navigation, the sub-navigation, the placing of h1s and actual, functioning crumb-trails were all covered. Leading to Robert and Michael spending every waking hour - and one or two that probably should have been sleeping ones - implementing the new pixels before election day hit. A task they just about managed to complete, with only minutes to spare. Well done lads. And thanks to Librarian Ned for idiot checking. He really is the best tester we have. In amongst the markup and CSS tinkering, Robert and Michael also found time to tidy page titles, write some meta-descriptions and roll out rel-alternate links to pages with CSV equivalents. Because when one is playing with the web, it is always best to play nicely.
Gazing down the weeknotes column for new, old search it is difficult to do justice to all the work Developer Jon and our Jianhan have churned through. And all the testing Librarians Anya, Jayne and Ned have managed to tick off. We can but try.
Week 21 was mostly spent dotting Is and crossing Ts as preparations were made to finally go live with our Solr 9 upgrade. Delivery Manager Lydia once more taking to form filling duties prior to yet another engagement with CAB - which we think stands for the Change Advisory Board. Jianhan made some last minute code changes to prefer links to the internet rather than the intranet. Well why wouldn’t you? He also updated the piping used to pour statutory instrument and treaty data into the procedure editor database, which had been sourced from Solr 3, to instead flow from Solr 9.
Week 22 was Solr 9 go live week, which is a very quick sentence summing up months of work. What can we say? If you’re a Parliamentary Search user, we hope you enjoy the performance improvements and do not regret no longer having time to make and drink a cup of tea between pressing the search button and seeing some results. Thanks Jianhan. And thanks Librarian Ned.
Week 23 saw the focus turn back to the new frontend application, Librarians Anya, Ayesha, Claire and Jayne, alongside computational midfield sweeper Michael, getting stuck into some serious whiteboard action as they battled attribute layouts for our simple search result items. Developer Jon spent his time fettling what we’ve come to refer to as ‘result page furniture items’, being pagination widgets, query box design and a more than reasonable start on search facet integration.
In week 24 Anya and Michael finished speccing out detailed result design for each of our object types. And weeks 24 through 28 saw Developer Jon implementing the ‘time to return’ results functionality, researching then implementing the application code’s interface with faceted searching in Solr, and converting the code that calls our taxonomy service from single threads in series to multithreads in parallel. Which caused what had been ballooning performance times to converge back to quick. Or at least quick enough. That and a good deal of chat about where state should live - in the client or in the HTTP - a dilemma that continues to haunt us.
Moving on to week 29, Jon made a whole host of changes, not least the tidying of search result templates for oral answers and early day motions. He also set about plugging the number of results to return and and the result set ordering widgets into actual, functioning code. And made sure our handling of zero results returned did not look too wonky.
Work continues on attempts to liberate our taxonomy from the constraints of its software bondage. Week 23 saw a meeting with family friend Silver. Librarians Anya and Susannah and computational makeweights Robert and Michael all in attendance. Plans around which of our very many identifiers we hang the data off and how we’re going to handle deletes - hard right now, but softly in the future - were agreed. Business done, hands shook, we moved on.
Meanwhile, slightly offstage, Librarian Claire and computational reserve goalie Michael have continued to chip away at OaSIS - the Odds and Sods Information System - in an attempt to capture the data model as something roughly resembling an ontology. OaSIS being a system built to provide data for parliamentary business where source systems were commissioned and built with change event APIs so far down the backlog, they never made it to production. With a helping hand from Silver, they’ve now taken that work and plugged it into Data Graphs. Progress we all most pleased with.
Dissolution sees librarian eyes turn to the long list of things filed under ‘for when we have time’. Open calls are chased, long hard looks at our databases are taken, and new calls are logged. A by no means complete list of the things the team managed to move from the To do to the Done list over dissolution:
Public Accounts Committee responses to National Audit Office reports are now linked to the reports they respond to from 1992 to date.
Coming into force dates and coming into force notes have been reviewed and updated for all treaties back to 1992.
15,500 of our statutory instruments proved hypertext incompatible, carrying no legislation.gov.uk URIs. 16,000 have been reviewed, 15,500 have gained links.
Prompted by an equality of votes situation in the Lords back in March, we have a new term and all earlier instances have been winkled out and indexed.
Emergency debate applications and maiden speeches have been added to our list of content types, making them easier to find.
Way back in 2018, we worked with Librarian Andrew on a first pass attempt to mix-n-match our thesaurus terms with Wikidata IDs. Because other work came along - as other work so often does - that interlinking took something of a back seat. Not anymore. Thanks to more fantastic work by Librarians Deanne and Ned - who’ve been combing through several years of new and amended terms reports - that gap is now closed and the web is a better place for it.
Our dear taxonomy has a couple of branches, one describing drugs, one describing names of drugs. Over time and for understandable reasons the two branches got somewhat intertwined and tangled. The wrong plants in the wrong place as Beth Chatto would never say. Because quality librarianship shares a lot in common with good gardening, our crack team broke out their hand shears and their spades and took to weeding. Both drugs and names of drugs now being considered tidied. Lovely.
Not having quite enough work on their hands, a wee while back our crack team of librarians took on yet another task. This one to bring their spreadsheet expertise to the House of Commons Library’s Parliament Facts and Figures series. In recent weeks two more have found their way to the polished and published pile: Emergency debates in the House of Commons since 1979 and Maiden Speeches in the House of Commons since 1918. Excellent work, Librarian Claire.
It’s been a little while since we reported on the efforts of Librarian Anna and Shedcode James to tidy and normalise the Rush database. Somewhat understandably, both our heroes being otherwise occupied with more urgent general election matters, that work has been on the back burner for a wee while. That said, they have found time to tidy and normalise peerage types, constituency formation years and constituency disbandment years. And even reunited Mr Speaker with his dad. Lovely.
If you come here for our procedural mapping making efforts - and, let’s face it, who wouldn’t - we’re afraid you’re in for some slight disappointment. As proficient a map maker as Librarian Jayne might be, the general election has taken its toll on us all. Still, we can’t leave our dear reader empty handed.
You’ll be pleased to learn that, despite the pressures, Jayne managed to fit in two bits of map making. Firstly, everybody’s second favourite procedure map - the draft affirmative statutory instrument - has gained new steps describing the laying and possible withdrawal of Government Actuary reports. These, we are told, are laid before Parliament pursuant to section 142(1) of the Social Security Administration Act 1992, as amended by the Social Security Contributions (Transfer of Functions, etc.) Act 1999 and sections 150(8), 150A(5) and 151A(6) of the Social Security Administration Act 1992. So now you know.
Jayne has also managed to churn out a whole new super affirmative map, this one covering instruments laid under section 268 of the Investigatory Powers Act 2016, such as the the Investigatory Powers (Communications Data) (Relevant Public Authorities and Designated Senior Officers) Regulations 2020. Now there’s a mouthful. This one was more of a team effort, Librarian Ayesha lending a hand and Michael his usual hindrance. Our regular reader will no doubt be delighted to learn that all three - count em - clocks have also had their calculations added to our beloved egg timer. If a job is worth doing, it is worth doing properly being our usual mantra.
With all this going on, you’d think we’d have lacked time for our usual outreach and engagement efforts. Not a bit of it. Thursday of week 26 saw Librarians Anya and Silver head up East to give a talk to the Semantic Data conference open brackets Taxonomy, Ontology, and Knowledge Graphs close brackets. Some of the usual ground was covered, particularly around the difficulty of solving hard problems when your approach to computers tends toward digitising individual processes in individual offices. Which definitely saw some audience heads nodding. It will surprise no one - least of all you, dear reader - that Domain Driven Design and Cynefin also saw an outing. After all, it’s treating complex problems as merely complicated that gets us into a lot of these scrapes. Well done Anya. Well done Silver. We’re all proud of you.
