Dear reader, welcome back. If you tuned in last time - and of course you did, of course you did - you’ll know we were fresh off the back of another round of ‘Agile Quarterly Planning’. Perhaps the most important line item - if not the most visible - was a renewed commitment to upgrading our taxonomy management software, which is running two major versions behind what is considered current. That work had been put on hold whilst our Jianhan took the single triplestore bull by its proverbial horns - weeknotes passim.
This time out we had hoped to get a test environment in place so Jianhan could continue the long slog of updates to downstream systems, mandated by a switch from a taxonomy based on Zthes to something more modern and SKOS-shaped. So then what happened?
Events, dear boy. Always with the events. Firstly, Parliament prorogued. Never an unexpected event, but one that always seems to catch us by surprise. The problem is less to do with prorogation and more to do with the change of session. Which too often leads to issues with identifiers as identifiers are often unique in a session, and it can mean things occurring at the borders becoming confused as to whether they belong to the future or the past. And always means planning and tweaking the machines and crossing of fingers and monitoring until the danger zone is cleared. Secondly, something went awry in the upstream Hansard pipes meaning we not only failed to receive the Commons debate on the King’s Speech, but Lords’ Hansard went missing in totality. All less than ideal. Our Jianhan and colleagues found themselves paddling like heck to get our basic content ingest working again. Which has been quite a palaver. Understandably impacting upon his ability to set up a test environment for taxonomy upgrades. Even our Jianhan only has 24 hours in his working day.
In better news, he does tell us he’s yoked himself back to the taxonomy machine and progress is being made. Godspeed, Jianhan.
Back at the top end of the old value chain, Developer Jon has also been busy, though with considerably more to show for his efforts. A small tweak to the search results page means the session facet is now ordered reverse chronologically, rather than by the number of results found in a given session. Which makes much more sense.
Over on the object pages, we had hit a slight snaggle with a private notice question. It turned out that one of the 22 contributions had two contribution types when any fule kno they should only ever have one. This caused confusion in the pipes between triplestore and Solr, the wonky contribution never making it as far as our search index. Quite understandably, Jon had been working on the assumption that, if a private notice question declared itself to have 22 contributions, all 22 would be available to link to. He has now been disabused of that notion and adapted his code to be a tad more defensive. Meaning that private notice question is now fit and working again. The wonky data has also now been fixed. Because of course it has.
Pluralisation and singularisation is not a trivial matter in Parliament. We appreciate the occasional clerkly feedback we receive about these weeknotes, the kind gestures towards our not infrequent typos, and consequently would not dream of exposing abominations such as ‘Order in Councils’ or ‘point of orders’ to the open web. The longer Jon spends on the company books, the more singularisation issues he seems to hit. Our dear reader will be delighted to learn that we think we’ve tracked them all down, and Jon has fixed them all. The briefing series ‘Lords in Focus’ is now back to being ‘Lords in Focus’, and not, as last week, ‘Lords in Focu’. So that’s good.
Singularisation also reared its ugly head when our crack team of librarians came to test the ‘embedded query’ links from the summary sentences on object pages back to the search results. In this case, some of the links were populating the search box with a taxonomic identifier rather than its more user friendly label; not a path to confidence that your search was what was intended. In other cases, the label was being used but in its newly singularised form. Which meant the query in the search box was legible, but the search didn’t work. Both of those issues are now considered fixed.
Way back in the mists, there was, at some point, a split in the taxonomy whereby a small number of concepts were duplicated as ‘topics’ and managed in a whole new hierarchy. This was for the purposes of providing high-level browse of research publications. This fork was not something our crack team of librarians have ever been particularly happy about; work on the Library Knowledge Base™ and Publications Explorer™ hopefully point to a brighter future where taxonomic transitivity takes over the heavy-lifting. For some reason, when Jon started work on the object pages we asked him to display topics. Quite why we asked for this, when we intend to have rid of them, we have no idea. On the positive side, Jon showed not a hint of churlishness when we asked him to remove them again. Which he has now done. Thanks Jon. And sorry.
Once Jon gets himself clear of tidying up our mistakes, it’s back to the real work of fully implementing query expansion - more weeknotes passim. Happily, Librarian Phil has been kind enough to pop together a document setting out a whole host of query expansion permutations that our search application needs to handle. Which Jon has now turned into a set of automated tests. It only remains to write the code to pass the tests and that bit of our journey should be firmly behind us.
It’s a rare week that sees no actual procedural map making. Sadly, this is one of those weeks. Glancing at Trello, it would appear that Librarian Ayesha has picked up the task of removal of motion not called steps. So far, the poor lass has removed nine of the buggers. Only another 32 to go. Grim determination and endless tea making on the immediate horizon. Slow and steady, Librarian Ayesha.
In better news, our Jianhan managed to fix the integration between our Procedure Editing application and our beloved Egg Timer™. This particular little lookup grabs all the instruments currently before Parliament, checks whether their procedure involves a clock, gets the date on which the clock started ticking and asks the Egg Timer to provide a new estimate of when the clock should end. Which saves an inordinate amount of time for our crack team of librarians whenever expected future sitting dates change. Which they do. Quite a lot. Without that integration, they’d be stuck updating a hundred and odd instruments every few weeks. A mind-numbing task they’re far better off without. Much appreciated Jianhan.
Zooming up to the top of the procedural value chain, we’ve spent the last few weeks polishing our nifty little Procedure Browsable Space™ to add new CSV views, actualisation counts to business step lists and flags for committee concerns and motions tabled to work package lists. At least where possible.
On the former, we’re chuffed to announce that we now offer CSV views for primary legislation work packages (sample data only), secondary legislation work packages, and treaty work packages. So that’s nice.
On the latter, we’ve also added flags - annotations highlighting work packages where a committee has raised a concern or a member has tabled a motion - to listings of all work packages enabled by a given piece of legislation, work packages that happen to be before Parliament enabled by a given piece of legislation, all work packages engaging a given calculation style, and all work packages that happen to be before Parliament engaging a given calculation style. Neat and indeed tidy.
In less good news, there are a handful of queries we’d like to run, but our aged triplestore runs out of puff before returning results. Which means we cannot yet apply flags to either the full listing of all work packages or the listing of secondary legislation work packages. And we can’t apply actualisation counts to lists of business steps in a given procedure. For those reasons, bits of our code now have two lots of queries in place: those that work and are being run, and those labelled ‘with flags’ that don’t work and are not currently called. We live in hope we can swap these two things around if and when our triplestore ever gets the upgrade it so richly deserves.
In purely cosmetic news, Young Robert’s attention to detail means the navigation strip for an organisation accountable to Parliament is much less busy. And links to both our beloved Egg Timer™ and legislation.gov.uk a little more prominent. Which should please John if no one else. It’s the pixels that count.
Librarian Jayne and her computational helpmates Young Robert and Michael have also been taking another look at the work package parsing code, last touched back in lockdown. They’re now fairly certain they have a rough idea of the SPARQL queries needed to return the data to allow the parsing. Once the data’s assembled, it only remains to rewrite the parsing code. Which may well occupy the next several months. If you remember with fondness the days when these notes were all about ternary logic gates, you’re in for a treat. It’s going to be incredibly boring, we promise.
Work continues apace on replacing our aging Odds and Sods Information System - yet more weeknotes passim. As of this week, Librarian Emily has double checked the existing Data Graphs implementation and made notes of where she’s found it wanting. She’s also done a full tour of the old application, mapping data attributes to content types. And, because the underlying database cannot be trusted to align with the user interface - please don’t ask, we can hardly believe it ourselves and we get cross - Emily, Jayne and Michael have done the same work at the database level.
With those three pieces of information at our fingertips, we’ve taken a second pass at a domain model, nuked most of what was in Data Graphs and started to replace it with something hopefully better. It is now possible to create Acts of Parliament, Church of England measures, Transport and Works Act order applications, and impact assessments. Other content types to follow in short order. Emily continues to test.
Aside from a couple of new links to Commons Library dashboards from our English constituency pages - Free school meals and School workforce, should you be interested - all other psephological work has been very much under the waterline.
Our second - possibly third - attempt at staged publishing for the next general election has been signed off by Statistician and Product Owner Carl, so that code is now live. And should make no visible difference. If you do spot a visible difference, something has gone very wrong. You know where we are. Do please get in touch.
Sticking with the next general election, there is also a new, quite small - but not insignificant - Rake task to be run at dissolution. This will, in theory, close the current Parliament, open the next Parliament, create the general election, and create all the elections in the general election. It has, of course, been tested, but is obviously difficult to test in anger. If you’re a Rubyist - or even if you’re not, it’s hopefully quite well commented - why not put Michael’s mind at rest and take a read?
Would it really be weeknotes if Librarian Emma had not taken a feather duster to a couple of the corners of our ‘information space’? No, it would not. This time out, Emma has:
cleaned up all statutory instruments with more than one identifier
cleaned out the back of the bill cupboard, removing any incorrect corporate authors
If you’re a clerk, and absolutely no offence, we really do need to have word about session citations at some point. Even Erskine May would be appalled.
On a sunny Wednesday evening, Librarian Anya and her computational whiz kid colleague Michael, ventured south of the river to bid what they hope is not a final farewell to House of Commons clerk Martyn. A splendid man who, when faced with our endless requests for information, has been nothing but patient and kind. We’ve talked to all kinds of people on our travels and Martyn is one of a select group of old-school proper proceduralists whom we value so dearly. With a firm emphasis on the is, because you never lose that level of knowledge.
The next time someone suggests parliamentary procedure is somehow ‘arcane’, there is an answer. Simply contact Martyn, maybe take him down the pub, maybe buy him a Guiness, and maybe ask him some questions. We guarantee he’ll be delighted to answer. People that know stuff often are. And a thing cannot remain ‘arcane’ if you take the time to sit down with a nice chap over a pint of Dublin’s finest and have it all explained. Parliamentary procedure is complicated admittedly and complex on occasion, but never arcane.
Best of luck, Martyn. We hope to see you at the next Study of Parliament conference in January. Between now and then, you will be much missed.
