As our regular reader will well know, the triplestore that sits behind Parliamentary Search has not been touched by developer hand for the best part of a decade. Indeed, the vendor informs us that it’s the only triplestore of that vintage outside of a museum. In recent weeks, our Jianhan has taken the bit between his teeth and embarked on an effort to finally upgrade the dear old thing, in some attempt to make it fit for the early 2020s. Given the number of pipes into the triplestore and the number of pipes out, this has proved no simple matter.
Weeknotes passim have covered the many and varied pipes to be fixed, fettled and fine-tuned. This week saw a couple more blocks put in place, the upgraded triplestore now coming complete with both a backup and the appropriate software to deploy it as a service.
It is now Saturday. Our Jianhan has been peering at his computer since 6:20AM. Our crack team of librarians also gave up a sunny, Spring afternoon and clocked on at 2PM to apply their keen testing eyes to the problem. Unfortunately, things did not go quite to plan. Something in the pipes between the new triplestore and our Solr instance went awry, Member contributions to oral questions taking a bit of a data integrity hit. Nobody quite fancying spending another weekend peeping at pixels, the release has now been put on hold til Easter recess. Between now and then, we’ll need to make a new plan.
Not everything sticks first time and not all news is good news. Nevertheless, we appreciate your efforts Jianhan. And those of Anya, Claire, Emily and Ned. Undaunted, we go again. Blasted computers.
Over in the frontend, Developer Jon has finally freed himself from a whole slew of tiny tickets consolidating our work on search result and object pages. Which must come as some relief to the lad. This week we’re pleased to report that the Claw-based copy button now works and our page widths have found a nice middle-ground between portrait and widescreen. We told you our ticket granularity was approaching sub-atomic.
Happily, moving those cards to the happy pile has freed Jon up to do some actually interesting work. He must have been wondering if that would ever be true again. To that end, queries typed by users no longer head straight to Solr, but are instead first passed to our taxonomy service API where an attempt is made to match words and phrases to both identifiers and synonyms. A combination of what the user typed and any synonyms and identifiers are then sent to Solr, meaning searches for ‘bird flu’ now return results for ‘avian influenza’. And vice versa. As we said last time out, close to magical.
For those of a ‘view source’ mentality, Jon has also added what actually gets sent to Solr to our revised and upgraded Claw implementation. Just click on the link above; press Control, Alt and D; scroll to the bottom of the page; and you’ll get a peep inside his magicians hat. Lovely stuff.
In equally good news, Young Robert’s continuing adventures deep in the bowels of Azure are also starting to pay off. We now have a domain set up and an IP address to point said domain at. All that remains is a little Cloudflare configuration and the two ends of the tunnel should meet in the middle. Well, that’s not quite all, but most of the other tickets have acronyms your regular correspondents don’t pretend to understand. So we’ll leave that there.
On the subject of our taxonomy - more accurately our thesaurus, but that’s much harder to say - efforts to liberate it continue to scamper along at a fair old pace. This week, Silver and his Data Language colleagues have been plugging our Solr API into Data Graphs for the purposes of ingesting all subject indexed Research Briefings. Or at least the ones from the House of Commons Library. It is possible that one or two House of Lords Library briefings also escaped their bonds, but that was merely an accident. And should hopefully be fixed now.
There was the briefest of brief hoo-hahs, when we found out we were accidentally slurping up topic indexings rather than subject indexings, but that’s the perils of maintaining two similar but quite different taxonomies in one thesaurus for you.
A second hoo-hah broke out when Librarian Ned’s changes to the taxonomic hierarchy failed to propagate to Data Graphs, dangerous dogs showing an unwelcome parental attachment to animals in general, when Ned had quite firmly and quite rightly insisted they were a type of mammal. That is also now fixed.
Taxonomic integration aside, we’ve also made a small but nevertheless substantial change to the data model. What had been a class of specialists linked to subjects by means of a specialism property has been subjected to an additional nudge of normalisation. Subjects remain subjects, specialists are now just people, and a new class of specialism has been added to join them. All of which called for a quick revisit to our wireframes. Yes, actual wireframes. Like it’s 1996 or something.
As of last week, the resulting Subject Specialist Finder™ application was finally unveiled to our research colleagues. We’re told feedback has been forthcoming. What changes we’re being asked to make will have to wait until Monday.
If you tuned in last time out, you’ll know that Shedcode James has been hard at work upgrading assorted applications. Or our ‘productivity suite’, as Young Robert might say. This week he’s added our written answer bots, psephological efforts, procedure browsable space™ and the beloved egg timer™ to the safe and secure pile.
He’s also removed a good deal of unused cruft and rolled out rollbar across everything, before concentrating his efforts on our committee papers application. Which was blowing up for two reasons. Firstly, Michael hadn’t put in place a check for the existence of content before attempting to add a last updated date. Top professionalism there, Michael. Secondly, we were seeing all kinds of format exceptions triggered by some bot or other that was hitting the site with requests for RSS that didn’t exist and will never exist. Good luck with the old AI, Californian friends.
Way back in week 5, we went off on one of our overly-detailed explanations of what happens to proposed negative statutory instruments now that the European Statutory Instruments Committee is no longer with us.
This time out, our pair of PNSI procedure maps have undergone further contortions as they were forced to react to the laying of a PNSI by the Department for Transport and the subsequent laying of a PNSI by the Department for Business and Trade. Librarian Jayne took out her trusty cartographic pencil, adding new steps for consideration by both the Transport Committee and the Business and Trade Committee. And that, we thought, was that. At least for a wee while.
Imagine then our surprise when we got wind of yet another PNSI in the pipelines. This one to be laid by the Cabinet Office. In Jayne’s temporary absence, Librarian Ayesha found herself culpable for cartographic duties. The pencil was duly deployed, which is why our PNSI Commons committee component map now finds itself decorated with yet more consideration - this time by the Public Administration and Constitutional Affairs Committee. A step we’re expecting to see on the receiving end of its first actualisation early next week. Excellent penmanship, Librarian Ayesha.
Still with the map making, we’d been long remiss in our description of motions to consider. Such things are usually seen on occasions on which a statutory instrument is debated in a Delegated Legislation Committee or, indeed, in Grand Committee. Since our map making exploits started, back in 2017, we’ve acknowledged their existence but never applied our full map making skills. Perhaps unsurprisingly - having considered a matter, it seems odd to be asked whether you have indeed considered it and then answer in the negative. That said, Parliament is never less than surprising, and we’re aware of a couple of occasions on which the answer to the question has indeed been no. For that reason, Librarian Jayne has now added explicit motions to consider to all pertinent maps. A decision on whether we might travel back in time and actualise such steps, remains up for discussion.
A few weeks back, the Electoral Commission got in touch to point out that their figures for the 2024 general election did not match ours. Causing some degree of confusion if not consternation. Data Scientist Louie set out to get to the bottom of matters. And the bottom of matters he got to.
It turns out that, whilst we calculate valid votes in an election by summing all the valid votes for all the candidates in that election, the Electoral Commission take a different approach. Their vote figures are arrived at by subtracting the number of invalid - or spoiled - ballots from the total number of ballots issued. A calculation that, by choice or by accident, results in abandoned ballots being counted as cast. So that postal vote you taped to the front of your fridge but didn’t get round to sending, the Electoral Commission would have included that in their votes calculation. The House of Commons Library would not. With this new news in mind, we’ve updated our election model to include a new property for ballots issued and a new comment for valid vote count. So next time our figures disagree, we at least have a note to self.
Now, you might think that democracy has been around for long enough that the associated calculations might be standardised. Apparently not. The Commons Library calculates turnout according to the number of valid votes cast divided by the electorate. When most other people appear to include spoiled ballots in that sum. A matter that continues to confuse at least one of your regular correspondents.
Mistmatched sums aside, Louie did find some discrepancies in elections recorded and reported by Lambeth Council. Which means we now have revised figures for elections in Clapham and Brixton Hill, Vauxhall and Camberwell Green, Streatham and Croydon North, and Dulwich and West Norwood. Hopefully, that’s the last of the 2024 corrections now issued. Though we never count our chickens.
In other news, Statistician Carl came through with a new spreadsheet of 1997 constituencies. All of which have now been uploaded and assigned to their appropriate boundary sets, being England, Northern Ireland, Scotland and Wales. Unfortunately, back in those days, the ONS only issued geographic codes for England and Wales, which presents a whole new set of problems for data integration going forward. Well, going backward going forward anyway.
Finally, we’ve started work on importing Librarian Claire’s spreadsheet of maiden speeches, a requirement first raised by resident constituency expert Neil. So far, we’ve got as far as creating yet another Trello board and absolutely no further. Though hopefully that particular Trello board will be a lot less long-lived than some of our others.
It has been noted by our crack team of librarians that government position titles on Member pages link not to the position but to the containing department, which feels like a semantically incorrect use of hypertext. Gareth Thomas’ position as Parliamentary Under Secretary of State, for example, linking to the Department for Business and Trade despite the position having its own page on gov.uk. This noting has been passed on to colleagues in the Parliamentary Computational Section in the hope of rapid rectification.
In the interests of more accurate information management, a policy change has also been initiated. Where a minister holds positions in more than one department and takes a period of leave from all government positions, we had been creating a ‘Minister on Leave’ position for only one of those departments, because the alternative ‘looked strange’. We now prize accuracy over aesthetics and create a ‘Minister on Leave’ position for every position absent of an incumbent. Quite obviously, it would be far better if we could model periods of interruption to an incumbency, but that avenue is not yet open to us.
It would be a miserable life if our noses were constantly pressed to the grindstone. All work, no play etc. With that in mind, in week 11 we undertook a whirlwind tour of Whitehall and beyond.
Tuesday night saw Anya and Silver take to the stage at the Governmental Computational Section HQ over in Whitechapel, for yet another outing of World IA Day - the London edition. Sadly, a recording is not available but slides on the subject of a knowledge graph for the House of Commons Library have been placed online for your delectation. A late breaking editorial decision was taken to remove many of Michael’s Wardley map slides. A decision I think we can all be grateful for. A drink was taken.
On Thursday Cabinet Office Kelcey braved the West Coast ‘Mainline’, leaving the capitol for a trek down to London. Person and baggage remarkably intact, we bumped into him just outside the Institute for Government’s rather plush HQ. Venturing inside, we met Philip for yet another chat about public bodies, government departments, positions therein and incumbencies thereof. Philip talked us through the IfG Ministers Database and a number of lessons learned. All agreed, none of this is particularly easy. A drink was taken.
The very next day, Kelcey and crew were up bright and early for a trip out to Kew. A couple of fairly disastrous train journeys behind them - well done, Silver, well done Michael - once safely ensconced in the concrete carapace, they were joined by Catherine, Jenny and John for yet more - yep, you’ve guessed it - chat about government departments and etc. Our understanding of what is and what is not a public body took a bit of a tumble. Hopefully for the better. John seemed fairly insistent on setting homework. You should read the Framework Agreements, he said. You should read the framework document guidance template, he added. We promise we will John. Thanks, as ever, for having us. Again perhaps, when the weather picks up? A drink was taken.
