These notes tend to be written top-to-toe in a single sitting. Usually over the weekend. Often with a mild to middling hangover. As neither of us are quite as young as we used to be, we generally find ourselves beginnning with little to no memory of what’s actually happened. Which leads to opening with promises of brevity. Several hours later, we’re up to our oxters - as Anya’s dad would say - in open tabs of Trello boards and the inevitable wall of text follows. For that reason, this week’s outpourings come with no such promise. Let’s just get stuck in and see where we end up.
New, old parliamentary search continues its consolidation journey toward coherence. As the days tick by, the work gets ever more granular, the accompanying cards more granular still. We won’t assail our dear reader with the intimate details of what’s been fixed over the last few weeks. Please rest assured that Librarian Jayne and Developer Jon’s breakneck game of Trello-based ping-pong resulted in 14 more cards making their way to the happy pile. By way of a quick summary of highlights: tweaking an existing search to kick off a new search now preserves the user’s selection of results per page, detail level and result ordering, which is about as RESTful as we’re ever likely to get; title attributes on filter links display the scope note where present; the application no longer panics when faced with zero results; and testing with assistive software has commenced. Much better than the service we’re attempting to replace we’re told. Well, you’d certainly hope so.
Developer Jon has also been tidying the markup to ensure that all our pages pass the HTML validation test. Which they now do. Ish. There remains one problem where the validator complains about something that’s perfectly valid according to the HTML5 specification. And another problem whereby Rails helpers add autocomplete=’off’ to hidden fields as a workaround for a bug in Firefox. Given we’re not paid to fix Firefox, that’s definitely someone else’s problem. At this point, our dear reader may well be thinking, why bother? 90% of browser code is forgiveness. Does any browser really care if your markup is valid? Again, you’ll have to trust us. This kind of attention to detail greatly pleases our two resident old men. Let’s face it, they get few other pleasures in life.
These tidies leave us with four consolidation cards on the checking pile, and five still to go, before we enter the next phase of work. And what might that be, you may well be asking. Well … for the moment, new, old parliamentary search takes the user’s search string and passes it straight to Solr. This is all rather different to production search, where the search string is first sent to SES - our taxonomy API. This call enriches the words typed with both identifiers and synonyms. Which is why a search for ‘bird flu’ also returns results for ‘avian influenza’. And vice versa. The taxonomy lookup is the bit of search that makes it not rubbish and, occasionally, close to magical.
This then is Developer Jon’s next piece of work. Or would be if his contract wasn’t about to run out. Marvellous. Delivery Manager Lydia continues to look for workarounds, in a way that only Delivery Manager Lydia can. Absolute whirlwind that lass. We’d be lost without her.
The production search implementation is further complicated by someone back in 2012 suggesting it should ‘work like Google’. Which is the kind of thing people say in workshops. In practice, ‘work like Google’ was taken to mean that a search for Diane Abbott should return the same results as a search for “Diane Abbott” and not a search for Diane AND Abbott. Somehow removing the necessity for users to (learn to) quote phrases. This requirement led to a whole bunch of string tokenisation, splitting the query string into candidate phrases, all of which were sent to SES in the hope at least some of them would match a term in the taxonomy. An implementation that’s been going wrong in unexpected ways for the past 13 years. Will we attempt such shenanigans again? We’re not really sure. It’s been observed that “we’ve failed to make it work for more than a decade. It seems unlikely we’ll make it work now.” A sentiment we’re tempted to concur with.
Librarian colleagues are often asked, how do you cope having two young, thrusting whippersnappers like Robert and Michael around? They have so much energy. And stamina. Do you not worry they’ll move too fast and break things? Librarian Anya takes the mic to intone - reader, we do indeed worry. That’s why we brought in Shedcode James to add a veneer of professionalism to their scattergun efforts. And professionalism he has indeed added.
The past couple of weeks have also found James in something of a consolidation phase, casting an eye over applications both large and small and upgrading them to meet more modern standards. Or ‘hardening’ them, as Young Robert would probably say. Heroku instances have been updated, Ruby instances have been updated, Rails instances have been updated, Gem dependencies have been updated, database backups have been added, procfiles have been introduced, logs have been raged at, rollbar rolled out and SQL queries updated. And that’s not all - additional computational magic includes automated dependency checks, security checks and a little test that makes sure applications actually start and show something. All in the cause of making everything more safe, more secure and easier to maintain. Going forward, as Young Robert might find himself forced to add.
So far, James has ticked off eight applications: bill papers, committee papers, Library feeds, the MNIS Prodder, the regnal year generator, the Resul Umit general election dataset, our attempts at improving upon the FCDO treaty website and last - but by no means least - our beloved egg timer.
The latter now also benefits from what Robert would undoubtedly call ‘observability’, the notoriously fickle Google calendar integration has been set up to alert all concerned the next time it inevitably fails. There is still work to do to act upon these notifications. We remain hopeful that one day the machines can be taught to recover from such failures without puny humans needing to get their hands dirty. But such work is on hold until James escapes from his current ‘dev-ops’ hell. Sorry James.
In exciting, late breaking news, as we type these notes, our added observability has just observed its first Google calendar sync failure. In previous circumstances, we’d have known nothing until Librarian Jayne turned up for work on Monday morning, checked the sync checker, clocked the time discrepancy and prodded Michael for a database reset. But forearmed is forewarned, as they say. Nice work, James.
Way back in the mists of time - way back, way way back - we did some work with Messrs Hennessy and Korris, mapping out cheatsheets of cartographic cascades across all our procedure maps. Cascades in this case being things that should - at least in theory - automatically happen when a different thing happens elsewhere. One example being where a made affirmative instrument is revoked or lapses or is rejected by the House of Lords, any extant approval motion in the House of Commons should be caused to be withdrawn. This week our crack team of librarians finally found the time to act on this information, both maps and data being updated accordingly. Cracking stuff, that should prove most helpful when we finally get our procedure parsing code back up and running.
In other map-making news, Librarian Jayne has also added steps and routes describing scrutiny reserve for the Joint Committee on Human Rights, to both the made affirmative remedial order procedure map and its data. Another standing order, digitally satisfied. And a standing order we would definitely link you to, if only we could. Sighs.
Our reader will no doubt recall that - again, some time back - we expanded our JCSI reporting step to take account of the criteria under which the committee might report on an instrument. Back then, JO Jane ran the show, and declared that, whilst she was perfectly happy for us to capture the data, she did not want it to appear on the website. Perhaps fearing more noise than signal. With JO Jane now departed and happily scampering around retirement fields, Librarian Jayne thought it best to check in with the current clerk of the JCSI and the Counsel for Delegated Legislation in order to establish whether opinions had changed. And changed they had.
Both parties had been unaware that we’d been collecting the data and were more than happy to have it appear on the website. Not only this, the Counsel for Delegated Legislation has also asked for a regular report on which reporting criteria are seen most often. Luckily, since the introduction of step collections, adding reporting criteria steps to the website is merely a matter of adding them to the website visible step collection. And our rinky-dinky SPARQL endpoint and query library make reporting on numbers a trivial change. More satisfied customers. Lovely.
In our final piece of procedural news, we’re delighted to announce that moves to retire our procedure editor application and replace it with a Data Graphs implementation. also appear to be going swimmingly. We now have all of the model in place, and about half of the data migrated. Though, given the data awaiting migration requires more in the way of munging, maybe we should claim we’re about a third of the way there.
Efforts to liberate our taxonomy from its software-imposed straightjacket continue to progress rather nicely, potential derailments avoided by the ever watchful eye of Librarian Susannah. As of this outing of ‘week’notes, we’ve set up a new Data Graphs project, pointed that project at our new taxonomy message queue, added piping to keep our list of subject indexed research briefings up to date, revised the application code to apply taxonomic transitivity on read rather than write, imported team:Thesaurus’s mappings of Library specialists to terms in the taxonomy, imported contact details, and adapted the web interface code to take its data from the new project. We’ve even figured out how to take an email address and pop the user into a Teams chat or call from an HTML link. A feat that inspired childlike joy in our Michael. Highly unusual, given his customary attitude to Teams.
Unlike most of our usual efforts, there is personal data involved here, in the form of both specialists and their contact details. Getting a Teams call is bad enough. Getting a Teams call from a complete stranger does not bear thinking about. So, for once, we cannot provide links. You’ll just have to take our word for it.
Still on the subject of contact point modelling, we’ve had a long-standing problem displaying contact information for Members on the website. Whilst the underlying database has a single table that appears to do what was needed, the MNIS administration application split the population of that table into two types: postal addresses or web addresses. Every time a Member got in touch to say they’d quite like to list a postal address alongside its online equivalent, we were forced to politely decline, shuffling in our seats and looking at our shoes as we explained they could have one, or the other, or both, but not both together.
Happily, colleagues in the Parliamentary Computational Section came to the rescue and the application constraint has at last been removed. Meaning the physical and the digital can happily co-exist as a single contact point in the database and a single card on the website. We hope our customers - and indeed their customers - will be pleased.
The change also introduced a brand new field, in the shape of notes for public display. Librarian Emily has been combing through our old, fake, “additional details” addresses - a longstanding hack to enable notes such as a constituency office’s opening hours appear on a web page, close by the constituency office’s contact details - and transferring those notes to the, erm, notes field. Very satisfying.
Those of an ontological bent will be delighted to hear that Librarian Ned and his trusty computational helpmates have given our house membership model a much needed lick of paint. Clerkly eyes, should they be reading, are encouraged to take a look.
Shedcode James having quite a lot on his plate at the moment hasn’t stopped him from lending a helping hand with our Rush database. That application has also been subjected to an upgrade, though - being better maintained in general - that was confined to bumping it to Rails 8.
Librarian Anna had also spotted a couple of places where our date of birth redaction had gone awry. Not because of any fault in the redaction code, but rather that assorted strings describing name changes also came complete with dates, some of those dates being birth event related. That has also now been fixed. Thanks James.
And finally, in a somewhat timely coincidence, the latest in our series of Parliamentary Facts and Figures publications has just emerged from the Library paint shop and hit the newstands. This one describing Members of the House of Commons imprisoned since 1945. Dive in. Enjoy.