As weeknotes open, we find ourselves, once again, in the unfortunate position of stumbling into the classroom, staring at our shoes, and mumbling yet another excuse for the tardiness of our homework.
For the 2024 general election, we rolled out our Candidates Database Tool and hoped for the best. That bit of software dated back to the halcyon days of Oli, and had not been touched by the kindly hands of a developer for a good many years. Colleagues in the cyber security contingent were also unimpressed, taking one sniff and declaring it smelled funny. The time had obviously come to replace.
Since then, different colleagues over in the Parliamentary Computational Section have been beavering away, expertly guided by the Library’s chief - and indeed only - Data Scientist, Louie. This week, the fruits of their efforts finally saw the light of day, and we now have a new, useable, and indeed secure replacement in the shape of Election Manager. A bit of software that we hope will not only replace the Candidates Database Tool for general elections, but also help us to simplify our by-election workflow. Which could, in all honesty, do with some simplification.
If you are our regular reader - hi, Alan - you’ll know that we divide our dwindling years into quarters. Part of our previous ‘quarter’ has been spent specifying the feeds from Election Manager required to populate our Election Results website. This quarter saw the output of data in exactly the format specified. Magical. Computational ‘expert’ Michael stepped in to write a new set of import scripts and everything went smoother than we could have dreamed possible. Like a knife, said Michael. Through butter, he added.
This is because, happily, Election Manager shares the same core data model as Election Results. Meaning that all the work that had to happen between Louie and our crack team of librarians to reshape and augment data from the Candidates Database Tool is no longer necessary. Thereby saving time, reducing the potential for errors that inevitably creep in when you’re decanting data through Excel, and we’ll be able to publish verified information in a much more timely fashion. It’s remarkable how simple life gets when two teams can agree on a common model of all the things and shared identifiers for those things. Go us.
Feeling quite pleased with progress, Delivery Manager Lydia, Data Scientist Louie, and rake task runner Michael, popped open our election-themed Wardley map and rejigged a pixel or two to reflect our new and improved reality. Legacy components were removed, dependency graphs much simplified, and things that had been marked red for ‘on fire or end of life or unsupported’, disappeared from view. If only the same could be said for our other value chain maps.
Never one to take a chance, Michael realised this would all be for nothing if the next general election came around and he’d won the lottery or, perhaps more likely, stroked out. For that reason, he organised what Young Robert might call “familiarisation sessions” with both Developer Jon and Young Robert himself, who are both now fully conversant with the general election process, what happens in what order, and the assorted buttons they need to press as a result. They even made a start on a new Trello board. Because of course they did. Belt and indeed braces.
In actual election news, team:Phil had to cope with the Clacton by-election, which - given the number of candidates - was a little like consuming four by-elections in one sitting. In all honesty, that did not go so well. Though this time through no fault of our own. With both Librarian Phil and Librarian Deanne taking well deserved vacations, Young Librarian Harry was left to mind the election shop. Things were going swimmingly, data entered, checkboxes ticked, cards moving right, until Librarian Anna spotted a discrepancy in the numbers. It turned out that the local authority somehow managed to publish two lots of voting figures that failed to agree. We’re quite used to official returns featuring numbers that fail to align with those published in the popular press, but we think this is the first time we’ve seen a local authority disagree with itself. At the time of typing, it is one week later, and your regular correspondent has just typed heroku pg:push psephology DATABASE_URL --remote psephology-production for what he hopes will be the first and last time. Which means verified numbers for all 34 candidates are now available for your delectation. Phew.
The only other change over in psephology land is a new page on the Election Results website outlining the House of Commons Library’s approach to verifying election data. Our ‘unique selling point’, as Young Robert might say, and well worth inclusion.
Sticking with Michael - and why not, he’s typing these notes, he may as well take some credit - he and his work wife Robert have also been busy rebuilding our Library Knowledge Base™ application. The old prototype served its purpose but was also a slightly shonky JavaScript-powered affair. And you all know how we feel about JavaScript. You only had to glance at it and the blasted thing would try and fail to lazy load some random pixels for no apparent reason.
Building atop the firm foundations established by Shedcode James, we now have a like-for-like replacement but one capable of rendering a list of more than ten items. And a plan to finally take down the prototype website and put the new one live.
In the short term, the Knowledge Base aims to provides a more computational replacement for our Subject Specialist Finder™. A ‘product’ that currently only exists as a PDF on the corporate intranet. Said PDF files our top-tier Commons Library specialists by the subjects they know most about. The better for front of house librarians, Members’ staff, and Members themselves to route their enquiries to someone who knows enough about a particular subject to be able to help. In the longer term, it may become a Knowledge Base from the Library, aimed mostly at an internal audience of Members and their staff, or it may become a Knowledge Base for the Library, aimed at an even more internal audience, putting everything our researchers need to know within touching distance. Or it may split and become both. Opinions differ.
Either way, it will remain internal, containing, as it does, contact details for our crack team of researchers. So we cannot provide a link, and, when we say its use of taxonomic transitivity is really building to something quite exemplary, you’ll just have to take our word for it.
Progress has also been made on three other projects for internal eyes only. The first is a replacement for our Procedure Editor application which sits behind most of our procedural map making exploits. Well, it doesn’t really sit behind them as such. What sits behind them is a team of crack librarians who know a lot about parliamentary procedure and care even more. Leaving the fragile humans to one side, it enables our statutory instrument service, our treaty tracking service, and, better yet, our Procedure Browsable Space™.
The truth is, our dear old Procedure Editor is a little long in the tooth. And makes considerable use of the aforementioned JavaScript; not a technology that ages well as data volumes grow. So we’re aiming to replace it with something that’s a little more node and edge compliant. Progress this time around has been good, Librarian Jayne and computational day care worker Michael having loaded most of our procedural data to its putative new home in Data Graphs. If you’d like to check in on our progress, there is a handy picture here, accompanied by some of the worst SQL you’ve ever seen.
Sticking with Data Graphs, we have a couple more projects on the go. The first is a possible replacement for our Odds and Sods Information Service, being ably led by Librarian Emily. OaSIS is a tool to fill in gaps in our information management where source systems have failed to make feeds available, and yet another example of a tool that’s a little long in the tooth. As of this week, we have a documented data model in the form of an RDF ontology. This one being a little unlike our other ontological efforts, being less a description of a domain and more a specification for a software system. Still, it has been liberally sprinkled with comments, which will shortly make their way into Data Graphs.
Finally, we have our publications explorer, a project that came about through chance, happenstance, and volunteer spirit. In the period between Developer Jon leaving us and Developer Jon returning to our welcoming arms, Librarian Anya and her computational sidekick Michael found themselves with time on their hands. Time which Anya - perhaps foolishly - volunteered to spend on helping research colleagues to explore - and hopefully improve - the data used to describe their published output. Having got their hands on a dump of the data that sits behind the current research briefing publication pipeline, they once more dived into Data Graphs with a new and improved model based on a series of domain modeling workshops held with research colleagues. Librarian Susannah has been assisting Michael with yet another model mapping exercise and even more of his shonky SQL - we miss you Rachel - to reshape the somewhat patchy data we have into a new, more descriptive model. A picture of progress to date, and the SQL it’s taken to get there, can both be found here.
If you’ve made it this far and tuned in last time out you’ll probably be wondering, “why no mention of your thesaurus service upgrade? You seemed to suggest that was going quite well.” And indeed it was. Until suddenly it wasn’t, our careering juggernaut of planned progress crashing face first into the wall of computational reality.
The problem stems not, for once, from a change in the model - which our Jianhan has covered - but from a change in the identifier pattern. The old application uses auto-incrementing integers as concept identifiers, identifiers of that pattern being scattered far and wide across a slew of systems. The new application takes a more modern approach of keying all concepts off GUIDs. When the project kicked off, the vendor assured us that old style integers could still be stored and queried by. Which happily turned out to be true. They also assured us that it would be possible to add a new concept and the computers would add one to a number thus assigning the new concept an old-style identifier. What they didn’t explain - or we failed to hear - was that adding one to a number was not functionality that arrived out of the box, and instead we’d need to pay for consultancy to make that happen. Happily, it turns out that Delivery Manager Lydia employs a similar style of haggling to Indiana Jones in a Tunisian marketplace. So that slight snaggle should be sorted shortly. Onwards!
The computational conundrum of adding one to a number aside, Librarian Anna has confirmed that the Indexing Service component is happily conversing with the new thesaurus API. She’s also been working with Developer Jon and Librarian Jayne to ensure that changes to the content type part of the thesaurus propagate to new, old Parliamentary Search as expected. Which they do. Which is nice.
One unexpected and somewhat unwelcome change did emerge from testing, whereby concepts intended for the parliamentary intranet only, turned up in Jon’s query expansion code - weeknotes passim. The pipes causing that pollution have been snapped shut, Librarian Phil has tested, and everything is back to working as expected.
Still with Jon, still with query expansion and still with bug fixing, Librarian Jayne clocked a problem with searches confined to a date range. “That will be query expansion choking on the range query syntax I expect,” said Jon, sucking on his plumber’s pencil and whistling through his teeth. Before promptly fixing it. Which is better than you get from most plumbers.
In less visible news, unless you’re a search engine - and we doubt even search engines bother to read these notes - Jon has also added meta description text to all our page types. Why not go view source on this beauty.
With most of the search aliasing work now under Jon’s belt, he also found a little time to incorporate Librarian Anya’s tweaks to our roadmap page which now better reflect current reality.
With librarian attention focused on testing search, testing the thesaurus management upgrade and testing the thesaurus management upgrade in the context of search, eyeballs have been a little distracted to focus too heavily on matters procedural. That said, we have made a change or two to our Procedural Browsable Space™. Changes we hope will be appreciated by our dear user. First up, by the judicious application of a new step collection or seven, our work package document list is now broken down by document type, rather than just by date. Which hopefully improves scan-ability.
Secondly, there were a series of views which we’d wanted to make but the SPARQL queries proved rather expensive, our aged triplestore getting tired and timing out. Not that triplestore tiredness would deter Librarian Jayne. Removing the pen from behind her ear, she sketched a note or two and announced she’d come across a new way to implement negation in SPARQL. And still she baulks when called a backend engineer. The upshot of that is our step list for a procedure now comes complete with actualisation counts. Or the number of times a thing has happened to an instrument, in much plainer English. At least since we started collecting this data back in 2017. It is a useful way to look at a procedure and check which procedural steps happen often and which happen hardly at all.
Finally, we’re delighted to announce that colleagues over in the Parliamentary Computational Section have rolled out the fix they first implemented to search on the statutory instrument website, to fix the same problem on the treaty tracking website. Which means both searches now cope with what Microsoft still insist on calling “Smart Quotes”. Incorrectly in our opinion.
For those of us in the, erm, Whitehall and Westminster ‘bubble’, a machinery of government change follows the arrival of a new Prime Minister as surely as night follows day. It was little surprise then that July saw the cogs of government disengage, the gears shift round, and the cogs clunk back into place. Most of the coverage in the popular press concentrates on departmental and personnel changes at the Whitehall end of SW1. Meanwhile, at the Westminster end, we not only have changes to government positions to take account of, but also changes to both answering bodies and laying bodies. All of which team:Phil dealt with, with aplomb.
Tidying up the loose ends, this week saw the decommissioning of two now defunct answering body bots, and the arrival of two new ones. If you’re interested in matters pertaining to computers, culture, media or sports, you might want to give @ddcms-answers a follow. It posts whenever the Department for Digital, Culture, Media and Sport answers or corrects an answer to a written parliamentary question. It’s also available on Mastodon, for the free and libre crowd. Stage right, the Department for Business, Innovation, Science and Trade written answer brief is covered by @dbist-answers. Again with a Mastodon equivalent. If you’re not someone who partakes in what Young Robert would inevitably call “the socials”, RSS per answering body remains available. Of course it does. A full list of our many and varied automated accounts and feeds can be found on GitHub. Why not treat yourself to some excellent content?
Much as John Key didn’t have much of a Christmas, our librarians have had a busy summer. Librarian Ayesha’s efforts have been focussed on 1,400 items of select committee material lacking a corporate author. We’re happy to report that that gap has been filled for all material published between 1986 and 2013. Meanwhile, Librarian Emma, ably assisted by Librarian Josh, has been taking the feather duster to our records for acts from 1990 to 1992, and from 1999 to 2008. The bit in the middle is underway. These records were created in various predecessor systems and were missing a link to the version of the bill gaining Royal Assent. Not only this, they also went back through the bill records, linking all versions to their preceding version. Still in the world of linking things, Martin made progress linking debates to SIs, completing records for 1993-1998, and Josh has been checking links between questions and questions asked “in pursuant to” questions. And finally, Librarians Jason, Jayshree, Emma, Kirsty and Steve have been casting their expert eyes over the indexing of research briefings, POSTnotes and questions written, oral and business.
Top work, as ever, Librarians.