Last time out, we wittered on for so long that our dear reader got in touch to ask, ‘are they called weeknotes because they take a week to read?’ Not being churlish types, we muttered that they take more than a week to write, pouted a little and moved on. Nevertheless, we have resolved to ‘cut the crap’ - as Young Robert might say - and get straight to the point. Let’s talk about the work.
For only the second time in its illustrious history, our Librarian of the Week Trophy goes to a non-librarian. For only the second time in its illustrious history, that non-librarian is our Jianhan. It gets tiring typing about the number of plates our Jianhan has to spin just to keep the show on the road. We can only assume it’s tiring to read about it too. God only knows how tired Jianhan must be. This week, our Jianhan has been mostly spinning two plates: patching up ongoing efforts to upgrade our Solr instance and getting everything shipshape for the imminent arrival of non-Ministerial corrections. Let’s start with Solr.
As our dear reader will know, we’ve been attempting to upgrade the instance of Solr that sits behind Parliamentary Search, skipping six major version en route. Mostly, this work has gone rather well. Performance improvements, in particular, have proved most encouraging, with searches that took 10 seconds now taking less than one. Given the number of searches we handle, a back of a fag packet calculation suggests we’ll save the country a whole working week when we finally do go live. But before that can happen, there are more hurdles to jump, mostly concerned with Solr 9’s aversion to anything resembling punctuation. We think we’ve spotted three problems:
given a quoted query containing a colon, Solr 9 flat out refuses to return results.
when shown an apostrophe in the form of a smart quote, Solr 9 does return results, but not the same results as with a non-smart quote apostrophe and not in the same order.
when presented with a hyphen, Solr 9 looks confused and refuses to return results unless you encase the whole query in quotes. But not smart quotes.
Never one to stare defeat in the face, our Jianhan took out his trusty computational spanner and wielded it in the general direction of the many and varied Solr configuration files. “Why is Solr 9 so hard to configure for fairly obvious use cases?” he wondered. We don’t know Jianhan. We can’t say. Anyway, thanks to his hard graft and diligence, we believe the first two of those issues are now fixed. Librarian Ned continues to test, leaving hyphens as the only mole left to whack.
Jianhan’s Solr configuration efforts are even more remarkable given they took place in his limited down time, in and around preparing the ground for non-Ministerial written corrections. Weeknotes passim. The actual changes were fairly simple. What wasn’t simple was the requirement for changes to code that hadn’t been looked at by human eye for the best part of a decade. Code that was sat on boxes that had also seen better days. Jianhan was forced to deploy two new virtual machines, one for the search application, one for the indexing application used by our crack team of librarians. That done, the new code could be deployed and deployed it was.
We’re happy to report that the modified indexing and search applications are working as expected, and ready to cope with corrections flowing through pipes from both Hansard and the written question and answer service. All such content is now appearing in the correct template in the indexing application, meaning our crack team of librarians are able to take the wheel and annotate as necessary.
We’re pretty confident that all the work that needed to happen to allow for non-Ministerial corrections is now complete, but we’ll only know for sure when the first non-Ministerial correction enters the pipes. And we have no idea when that might be. To that end, Librarian Anya has contacted Hansard colleagues and asked them to wave when they spot the first one. Or “give a heads up” as Young Robert might say.
If you’ve ever had the good fortune to meet us down the pub, you’ll know we’re consummate model makers, at our happiest with a whiteboard, a text editor and some Turtle. That said, we’re also keen to avoid the all too obvious pitfalls of the Enterprise Data Model, being well aware that no data survives contact with reality and no model survives contact with data. For that reason, once we’ve applied the final bow to a model, we like to prod it and poke it with actual data and actual code. Showing the results of that prodding and poking to domain experts remains the best way to test the models. At least that we know of.
Work on our election model started way back in 2016, but early efforts stopped short of describing constituency areas, boundary sets and the legislation creating them. That changed late last year, when we spent some time with geographic expert Neil and emerged with an almost fully-formed geographic area model. No mean feat, given the somewhat baffling nature of UK geography. By combining the two models, taking some election result spreadsheets published by the House of Commons Library, a copy of Postgres and some Rails code, we were finally in a position to prod. And indeed poke.
Both prodding and poking were forced to take a backseat when we learned that the data model underpinning the existing election results website had failed to take boundary changes into account and was due to be decommissioned. At which point, our noodlings were forced to take on a more professional vibe, as assorted librarians and computational ‘experts’ scrabbled to turn our prototype into an adequate replacement.
On Monday, following some inevitable last minute data wrangling, those efforts finally came to fruition and our election results website went live. Not a thing we’d originally envisaged. Top work Librarians Anna, Emily, Ned and Susannah. Top work Statistician Carl. Top work Data Scientist Louie.
If you’ve been following along from home and are more interested in features than ‘product’ releases, you’ll be pleased to learn that we’ve also added a whole slew of new data. The new constituencies that will come into effect when Parliament next dissolves, have been added and linked to their new boundary sets, new boundary sets in turn linking to the legislation implementing them. New constituencies in place, we’ve also added notional results courtesy of Rallings, Thrasher et al, together with Statistician Carl’s constituency area overlap data. Code wise, not much has changed, but - following a request from Carl - our party listing tables are now sortable by column headers. Lovely stuff.
Should you wish to build your own election result website, the spreadsheets for the 2015, 2017 and 2019 general elections - all much improved by librarians Anna and Emily - have been uploaded to the Commons Library website by Carl. Fill your boots.
Obviously, now the website’s live, we may well get a bit of time back to take everything we’ve learned and add it back into the election model. Which, let’s not forget, was the point of all this before distractions set in.
Over in search frontend world, developer Jon has polished off the item views for both papers being submitted and papers being ordered to be printed. Which means we are - at least as far as we know - now item view complete. A proper milestone, quite the achievement.
Elsewhere and slightly offstage, Michael put down his computational spanner and donned his facilitating hat, to run a series of workshops with assorted librarians. Minds combined in an attempt to work out, for each content type, what should appear on a simple result item, what should appear on a detailed result item and what should appear on neither. Thanks are due to Librarians Anna, Anya, Ayesha, Claire, Jayne, Phil and Steve. Neither Anya nor Michael were quite convinced that the simple view was quite simple enough, so they grabbed Librarian Jayne for a third pass. We’re told all three emerged happy.
Whilst preparation for search result work continues, Jon’s been keeping himself busy with tidying the page furniture - headers, footers and whatnot - and churning out error pages covering 401s, 404s and, everybody’s favourite 500s. Or get off my land, you’ve made a mistake, we’ve made a worse mistake. As they’re known in the trade.
Jon has also started work on the skeletal outline of result pages, including the simple and detailed view flipping widget, the result number control widget and the search result order widget. All of which may or may not be live by the time these notes meet your eyes. Perhaps most excitingly, he’s also starting to explore result faceting. Which, for now, is exposed as a JSON object inside a details element. So not quite ready for stakeholder eyes yet. But, we all know stakeholders. Only ever there for the last mile 1.
It’s been a long, long time since we had the opportunity to mention our beloved egg timer. Too long perhaps. But this week it makes a welcome return. Please wipe away that tear of joy, dear reader.
The egg timer makes its encore following a whole bunch of Parliament / Government confusion and kerfuffle around the laying of National Policy Statements and the calculation of scrutiny times arising from such layings. Back in January, Librarian Jayne was approached by the Liaison Committee, asking if it might be possible to include such statements on our statutory instrument service. And by the House of Commons Journal Office to ask if we’d mind including a calculation for them in the egg timer.
Unfortunately, we were unable to comply with the first request, because, whilst National Policy Statements are a form of delegated legislation, they are not in fact a statutory instrument. Unfortunately, whoever set up statutoryinstruments.parliament.uk made the mistake of forgetting to talk to domain experts. If they had, they may have created delegated-legislation.parliament.uk instead. Or procedures.parliament.uk perhaps. Or, preferably, parliament.uk/procedures. There are days when we still miss our old beta website. Dries eyes. Moves on.
Happily we were able to comply with the second request, only the minor matter of adding a new calculation style to our arsenal standing in the way. That new calculation style is now added, National Policy Statements are in the database, calculations have been tested and one more small bit of confusion has been moved to the fixed pile.
Under the watchful eye of Librarian Claire, Shedcode James has finished up the latest bit of work on our standing orders website. The admin side of the application has gained more in the way of crumb trails and a bug with the listing of published and unpublished revision sets has been cleared up. More visible progress can be seen on our standing order fragment pages, where text changes between major revisions are now highlighted in a fashion that even works for Michael’s colour divergent eyes. Lovely stuff.
Placing the cherry atop the cake, James has done a fresh import of the ParlRules data which means we’re now running with no mentions of mouses. If you do spot any mention of mouses, please do get in touch. It’s a sure sign that something has gone wrong.
Back when we first mapped out statutory instrument procedure - and, by golly, that feels a lifetime ago - there was a moment at a whiteboard when JO Jane first explained that scrutiny reserve for the Joint Committee on Statutory Instruments - as set out in House of Lords public Standing Order 73, no anchor link available guys - could be dispensed with if the question to do so was put to the House and the House agreed. Michael remembers this as being the first time he heard the words standing orders and dispensed with in the context of whiteboard maps, and it was at that moment that a lightbulb lit and he finally clocked we were mapping procedure, not just any old process. Anyway, on with the story.
Until recently, we’d treated disposal as a black box. It either happened or it didn’t, but there was no explicit motion and no explicit putting of the question on that motion. That a motion to dispense could be tabled and disagreed to was not captured by our maps or by our data. Because Librarian Jayne cannot see an I without dotting it and cannot see a T without crossing it, that limitation has now been … disposed of. Scrutiny reserve and motions to dispense now have their own component map, which Jayne has plugged into both the draft affirmative and made affirmative maps. The end result can be seen on the timeline page for the European Union (Withdrawal) Act 2018 (Exit Day) (Amendment) Regulations 2019. Unfortunately, you’ll need to scroll, because that page sadly lacks anchor links. Head down to the 26th March 2019 and you’ll see the motion to dispense being tabled, debated and agreed to. Lovely.
Jayne would also like it to be known that she has amended her SPARQL query library to capture both scrutiny reserve motions being debated and scrutiny reserve motions being agreed to. Of course she has.
Back in 2023, our crack team of librarians took over data wrangling duties for the Commons Library’s Facts and Figures publications. All 58 of them. Assorted spreadsheets in assorted shapes were handed over and the slow but steady process of redesigning and refactoring them began. This week Librarian Anya moved the Attendance of the Prime Minister at Prime Minister’s Questions since 1979 publication across to the happily refactored pile, meaning we’ve now ticked off a grand total of 11. Only another 47 to go. Godspeed Librarians.
A wee while back, friend of the family Alex introduced us to his colleague Owen. Owen has been tasked with writing a ‘data strategy’ for the University of Cambridge - not an easy job - and Alex is always one to encourage a chat. Back in week 13, Anya and Michael joined Owen in pixels for a good long rant about strategy and idealised future states and knowing where you’re starting being more important than dreaming of where you’ll end up. Cynefin may have been mentioned. It usually is. A pleasure to meet you, Owen. Do stay in touch.
Edward Wood. It’s been an absolute pleasure and we hope to see you down the Bookbinders in January.
1 Not you JO Jane. You’ve been pacemaking every step of the way.