Newest thoughts →

January 2023

After one day of using Airflow, here are my poorly informed thoughts on it.

First, what is it? It's a task and workflow authoring and scheduling tool that uses python. I think of it as Cron for the cloud. You can use it to create and run data pipelines.

However, because this API is newer, the documentation is somewhat substandard and peer to peer support is very immature. So there's quite a bit of trial and error.

It's nice that it uses python. It's really easy to get your head around the decorator syntax. The newer TaskFlow API in particular is cool.

Also, the Xcom system that allows inter-task communication is challenging in my view. If the goal is to create many micro tasks for easy debugging etc but then you make it really hard to pass data between tasks... That's a problem.

Furious tonight. UK government line that they will "protect the most vulnerable" is an outright lie. I've just seen, via FOI, that the vaccine taskforce will "not proactively seek out new Neutralising Monoclonal Antibody therapies" - the only hope for immunocompromised people.

I also have no idea what "proactively" means in that sentence. It's not as though they find these therapies by trawling through bins or something and they just stop this activity. These things are trialled and then end up in the decision process.

The other thing people don't realise about rheumatoid arthritis - besides that it's an autoimmune disease - is that each treatment you try takes up to 3 months to work. If it doesn't work, you have to wait another month to wash it out. So you can only try 3 new drugs per year.

It can be a LONG time suffering before you find a treatment that works. And there's basically very little that can be done while waiting - not even decent pain relief.

Homph. Just 1.5 more days and we'll restart the rheumatoid arthritis treatment. I've never wanted something so desperately.

Right. Today I am getting my head around Airflow to generate annotated JSON of our artifacts that includes preservation data.

First workflow, is go!

This week's tech focus is converting my digital preservation source imports into Apache Airflow tasks so that we can annotate the live JSON and present a standardised Labs API that merges in new metadata ideas as we build them.

Just saw an advert for a language learning tool that suggested that reading/listening to a novel in a different language was the best way to learn. The novel suggested was the famously light read... Moby Dick.

The 10th anniversary of tici taci records this year. I was obsessed with their output when I first found them in 2014 or so. Can't believe I then got to the stage of being signed to them!

Does anyone know whether the @internetarchive has a full list of indexed URLs available anywhere? Yes, I am aware that this will be sizeable 😂

FFS. As though those too ill to function can just "return to work". Also, perhaps we could have considered this before abandoning all public health measures around a condition with known longterm health consequences?

A relatively good day! Pain about manageable. Woke up and researched performance of different python API frameworks; chatted with my boss re API structure; made some electronic funk music; did some work on my book, Paper Thin; read half a novel; and played a card game on Skype.

I also saw The Great Pottery Throw Down for the first time, which Mrs Eve was watching. I've got slightly involved with it now!

I've spent many months just trying to stay afloat with health stuff. Took some downtime today to record some synths, guitars, and singing for fun.

Liz Truss, not content with costing the UK £40bn with her dodgy political ideology, wants to return to public life and her "pro-growth agenda". The shortest and most disastrous term of a Prime Minister ever, why is she not ashamed?

The week since my hospital release has gone much better than I'd hoped and doing my new role is satisfying and distracting from the pain. This evening is NOT good on the suffering front, though. 6 days until we can treat the rheumatoid and I am counting down.

OK, @flydotio is extremely impressive. Got from zero to deployment of my first Django app in 45 minutes or so using their automated Docker build system.

Today's fun: 1. Add a web fronted and REST API to the preservation checker tool. 2. Learn how to use @flydotio to deploy it 😁. 3. If time, start writing some test cases to make sure this thing is actually working on a comprehensive level 😂.

I'll also be packing the @CrossrefOrg annual data file of 134m records (currently compressed gzip JSON) into a SQLite database.

So, my basic SQLite index of the @CrossrefOrg annual data dump (134m records) allowing direct metadata retrieval from the compressed store by work ID (DOI) comes to 16.5GB.

I've just realised that I can probably substantially downsize this as it's storing a full file path, rather than just the filename with a base directory provided at lookup-time.

Second project for today was creating an index of the Crossref annual data dump, mapping DOI to a JSON location inside a specific gzip file. This will allow local DOI resolution/metadata retrieval from the data file. 7.5 hours left to go on the build for ~134 million records.

The overhead of Python's print method is quite extensive. A 3-hour iteration task (134m records) bumps up to 17 hours with each record being printed.

Today I will integrate PKP's private LOCKSS network into my digital preservation checker and begin work on a wrapper library for the Crossref annual data dump.

Another bad outcome of the untreated rheumatoid arthritis with which I am struggling: the pain and vascular inflammation wreck your blood pressure.

... Which, in turn, is terrible for the kidneys.

The doublethink that the Mail loves to encourage among their pensioner readership is that "they worked their whole lives and paid tax in order to get a state pension" but that everybody else who benefits from the welfare state somehow didn't contribute enough.

I have escaped from the hospital! And I am armed with pain relief.

But the kidney virus situation is worse than ever. Up from 16m to 93m copies. They've warned me of future kidney failure and lifelong dialysis as the most likely outcome.

Imagine being someone who is chair of the political party that supposedly preaches financial competence and then being pleased to announce that your £3m fine for paying tax incorrectly was for being merely "careless not deliberate".

Hospital sleep routines are brutal.

5am: blood pressure check
6am: morning drugs
6.45am: anti-bedsore injection
7.20am: IV pump alarm goes off
7:50am: blood draw/tests
8:30am: breakfast

Considering previous blood pressure check was at 11pm, it's not a great healing schedule.

And that's before all the other nighttime interruptions.

The less good news this week is that it's totally unknown as to whether they can save my kidneys. But there's no choice - the rheumatoid arthritis is now also attacking my vascular system and damaging that. And I can't live with the RA pain, which is so severe that I am sick.

The BK Virus that is destroying my kidneys thrives on the immunosuppression used to control the rheumatoid arthritis.

Kidneys are stable today. Bad but not getting worse. Pain team have given me morphine for the worst moments of rheumatoid arthritis control. Renal are reviewing my case tomorrow and, if they are happy that it's safe, I'll be discharged.

One day, I hope very soon, I'll return to tweeting about scholcomm, metadata, literary studies, technology, and all things nice. For now, though, my life remains dominated by life-altering medical problems. It is hard to find space for much else at the moment.

Today was particularly horrific. I didn't sleep all night due to awful arthritis pain. This morning was so bad - the pain in my neck was like a drilling into the spine - that I burst into tears when rheumatology came to see me as I just couldn't handle the pain anymore.

Then, this afternoon, my kidney results came in and they are very bad. Createnine levels still spiking upwards. Scheduled for an ultrasound.

So no discharge, high levels of pain, and kidney failure. Not a great day.

Owing to the rarity of this condition, I'm being written up as a case study report for a kidney conference. Should have asked to be a coauthor.

"Yeah, your EGfR is appalling, but your H-index is looking great!"

That careers advice - "don't do something unless it's a line on your CV" - well, I'm just applying that to kidney failure.

So, I was about to be discharged from hospital... But today's kidney function tests are a disaster and they want to keep me in.

It turns out that apparently only 65 people without kidney transplants, worldwide, ever, have had this BK virus nephpropathy problem. Great.

"Make that 66"

A small gripe: every day in hospital they bring a menu and ask me to select what I would like to eat. There are usually 4 options. After almost 2 weeks, I think I've got what I asked for twice. Perhaps it's a voting system and everyone just gets the most popular option 😀

Today's "fish pie" was actually lasagna

Survived the second day of exhausting immunoglobulin and... dare I say it... everything feels a bit calmer today. Kidneys aren't collapsing further (though they might after the IVIG). Rheumatoid arthritis is behaving better than it has in months. Perhaps things are improving...

Today has been gruelling. First my rheumatoid arthritis flared horribly & we struggled with pain control all night. Then I had the high dose IV immunoglobulin, which is exhausting and leaves you washed out. Finally, I need a full 10-day course of Remdesevir. So no discharge yet.

Day 29: after a course of IV remdesevir, I am finally testing negative for Covid! I will now be given high-dose IV immunoglobulin over the next two days to help clear the BK virus. This is quite an exhausting therapy.

Perhaps the least amusing statement a doctor made to me today was "we need to get you out of hospital as soon as we can so that you don't get any more infections here". Good to know how safe hospitals are!

It's as though my recent brush with Aristophanes is reaching out to me through the crossword in hospital. The answer is a phrase I've only ever seen in cryptic crosswords.

Play ducks and drakes

That you can confuse breaking your ribs for the pain of rheumatoid arthritis is quite a good benchmark for non-sufferers, it seems.

Mixed news.

* Covid is clearing. 👍
* BK Virus not. 8.5m up to 16m copies. But kidney function is stable at 28%. May need more IVIG. 👎
* I've recently broken my ribs. I'd thought they hurt but assumed it was rheumatoid. 👎
* We will start tocilizumab soon for the arthritis. 👍

Feeling rubbish and reading slowly, but finished @sadiseshiah's Utopian Drama this evening. From Aristophanes's The Birds (which she once made me teach!) up to the contemporary, this was a great read of how it can be generative to stage the impossible.

"Is it surprising that prisons resemble factories, schools, barracks, hospitals, which all resemble prisons?"

(I'm actually very grateful for my isolation room given immunosuppression - but it's also quite bleak and lonely.)

I was perfectly prepared to spend years in a lockdown state - had even grown a little used to our solitary confinement - but then my kidneys started to fail and I had to go to hospital weekly. Doing so is how I caught Covid and ended up hospitalised.

Lots of friends are quite shocked by my Covid hospitalisation, but it's why we've shielded for just under 3 years - and it's the reality that vulnerable people now face. Covid is still life threatening for many people with immune deficiencies. There's no societal protection.

It's not possible for me to "live with Covid". Even if I make it through this episode, another reinfection could be just as problematic, if not worse. The government and social acceptance of the vulnerable as collateral for zero public health measures continues to horrify me.

Day 24 of Covid and I am in an infectious diseases isolation room in hospital. I'm on 3x different IV antibiotics to combat bacterial infections, IV steroids so I still get those even if I'm sick, and have started Remdesevir to treat the Covid. CT scan showed Covid on the lungs.

I think that only one person - my wife - is allowed to visit me on my infectious diseases isolation ward/room here. Sorry Toby, but you will have to make do with Skype.

Day 23 of Covid and I have been hospitalised. Covid, BK Virus, and a suspected bacterial infection are overwhelming my damaged immune system. I'm on IV antibiotics, IV steroids, and immunology are discussing antivirals. Have to have a CT scan, a PET scan, and an echocardiogram.

So I went to hospital for regular blood tests today. The bad news: they want to admit me for IV antibiotics fearing I have a secondary infection. The even worse news: there are no empty beds in the entire hospital. I've been here 9 hours so far and am on a drip in A&E at present.

Well, I've been up all night being violently sick. Not what I wanted. Is that from: Covid? BK Virus? Kidney failure? Rheumatoid arthritis?

Problem is that if I can't digest my steroids, I have to go into hospital. Again. 😭🏥

Still fighting covid, BK virus, kidney failure, and badly controlled rheumatoid arthritis -- feel shattered! -- but did good work today on my preservation-status checker. It now imports CLOCKSS, LOCKSS, Portico, Cariniana, and HathiTrust data and can match these against a DOI.

After quite a bit of pain, my HathiTrust metadata importer is running. Time for a cup of tea.

It crashed while I was having a cuppa.

This morning's progress. Digital preservation status checker imports CLOCKSS, LOCKSS, and Portico data. Next up: PKP's LOCKSS network.

Disgusting.

The NHS is on its knees with 500 per week dying and these bastards think the "most dystopian" thing is having to wear a small piece of cloth over your face. Grow up.

Day 1 at Crossref and I began work on a question that I've wondered about for ages: what proportion of journal articles assigned a Crossref DOI are digitally preserved in a mainstream archive, as required by the PILA agreement?

Eventually, we can run this against the sampling framework to get a picture across the entire DOI base or on specific members/publishers.

I've then written a matching function that fetches DOI metadata using the REST API and looks up whether there's a corresponding entry in the archive that matches the volume in question.

To answer this requires us to rebuild the Keepers Registry from the base data sources, because now KR is owned by ISSN, there's no API. So going back to the raw sources is the only way. We then have to parse the idiosyncratic formatting of each archive's holdings.

So far, I've got an import for CLOCKSS and begun work on Portico.

Today's my first day at Crossref! There's a lot to get my head around, with many complex interacting technologies and systems. I hope, as with everything, to work as much "in the open" as possible. I'm also definitely not firing on all cylinders yet as covid drags on...

I've got codeine, tramadol, paracetamol, steroids, antibiotics. Let's do this.

https://www.theguardian.com/society/2023/jan/02/nhs-under-intolerable-and-unsustainable-pressure-say-medics

We can't have proper taxation and a functioning healthcare system because a government of millionaires would rather not be slightly less wealthy. And so 500 people will die preventably every week so that hedge fund managers can have more money than they'll ever be able to spend.

If that's what motivates you in life -- to sit on piles of unusable wealth while other people die -- then there really isn't much to be said. But it seems that's how it is.

Shifting gears in my brain, clearly, as I had a dream last night about how to redesign the Crossref python API client. But first, today, it's more book editing.

Daily editing work on Paper Thin has now begun. It's reassuring to have gone back to the start and found that the introduction is not in a terrible state!

I really dislike writing acknowledgments. Not because I dislike thanking people but because I'm terrified of doing a near-comprehensive list and leaving out someone crucial. I also once had a peer reviewer tell me off because the list wasn't extensive enough.