168 librarypunk

[Justin]
Okay, where's my soundboard? All right, let's go. [rock music] 

[Justin]
I'm Justin. I'm an academic librarian, and my pronouns are he and they. 

[Sadie]
I'm Sadie. I work IT at a public library, and my pronouns are they/them. 

[Jay]
I'm Jay. I'm a cataloging librarian. My pronouns are he/him. 

[Justin]
And we have guests. Would you like to introduce yourselves? 

[Scarlet]
I'm Scarlet. I am a collection strategist librarian at an R1, and my pronouns are she and they. 

[Dorothea]
And I'm Dorothea, the guest y'all can't seem to get rid of. [laughs] I teach in the information school at the University of Wisconsin at Madison, and my pronouns are she/her. [cheering] 

[Justin]
Welcome back. Returning guests, returning champions. 

[Sadie]
Love that. 

[Justin]
I didn't even bother counting up how many times Dorothea's been on, but just unofficial fourth mic of the podcast from, from the beginning. First guest, longest time guest. 

[Dorothea]
Thank you. 

[Justin]
So I, I love a good shit talking episode, so that's what we're gonna do. I started looking on Blue Sky the other day to see if, like, people were still talking about this article, 'cause I had a sneaking suspicion that, like, they're never gonna get this published because it's a preprint. And one thing I know about some of the AI stuff that's happening is a lot of these booster-ish articles are getting published in, or published, quote-unquote, on archive.org, where then they are reported on as a new preprint article, and they never actually make their way into any journals. So I'm interested to see if that's sort of, um, what's happening. I... They still have not published. It's been a month since this made the rounds. But this is the article, and let me pull up the title, Critical Confabulation: Can LLMs Hallucinate for Social Good? This is from the same people that brought you, I believe, some of the same authors that brought you the article Why SLOP Matters, Hoyt, Long, and- 

[Jay]
Scarlet's face journey was amazing when Justin said his name. [laughs] 

[Justin]
Hoyt and, uh, Eyman. So two of these articles, or three of them, three of these authors also wrote that article Why SLOP Matters, which is published in, like, a new IEEE journal that has, like, two issues, so. 

[Dorothea]
Oh, can, can we shit talk IEEE? 

[Justin]
Yeah. I mean, why not? 

[Dorothea]
I would love to shit talk IEEE. Okay. A few months ago, an article came out by Guillaume Cabanac and his crew. Guillaume is awesome, by the way. I love Guillaume. Fantastic scholarly communications researcher. And basically, a ridiculous percentage of crap publications in, in, in the relevant subject areas are from IEEE, because IEEE is basically a franchise operation, right? 

[Justin]
Mm-hmm. 

[Dorothea]
If you want to have a, quote-unquote, "reputable journal," you can sign up with IEEE and basically they don't check. And yeah, you can publish all the crap you want. 

[Justin]
Hmm. Well, I was wrong. This is ACM, a, a journal called ACM AI Letters, Volume 1, Number 1, Why SLOP Matters. And then the next one is Why AI SLOP Matters, But Not Like That, which is a response article in an upcoming issue. But it has not been published yet. But anyway, I was looking to see if they ever got this article published, 'cause, like, there is this problem, and some of the citations that aren't from digital humanists, that are from other AI authors are also citing archive preprints. So there is kind of an endemic of these questionable publications that don't seem to ever get into any journals, which is not terrible. I mean, from a Schol Comm perspective, fine. But when you're... There's, there's clearly, like, a, a communication strategy here that I think is problematic and has not been discussed at length. I know people have mentioned it, 'cause I obviously am not the first person to notice this, but it is an interesting trend to keep an eye out on. 

[Scarlet]
Well, like, when Archive finally... This happened in a couple of stages. So Archive came out and said, "We can no longer trust institutional affiliation for email submissions. So if you're the corresponding author and you have an email connected to an institution, we can't trust that anymore." That was the first thing that Archive had to come out and say. And then the second thing that Archive came out and, and said was, "We're just gonna ban you from submitting if you send us papers with hallucinated citations. We're taking a hard line- 

[Justin]
Mm-hmm 

[Scarlet]
... on it." And the number of people who came out and absolutely told on themselves, their affiliates, and all of their labs is incredible to me. Like, so many people came out and said, "There won't be any PIs left if you go ahead and do this." And I said, "There's eight... There's a line eight deep behind you who will show up- 

[Justin]
Mm-hmm 

[Scarlet]
... take over your lab, and actually read everything that they're citing, because that's what you're supposed to do." I thought that's what we we're... My gosh, it was amazing to, to see. I would love for someone to go through and do a study of all of the output of all of the authors that complained about this. It's incredible. 

[Jay]
Like, before AI could do this, what the fuck were they doing? [laughs] 

[Scarlet]
Also, why would you say that out loud on LinkedIn, on social, where people can see you? Like- 

[Sadie]
Like- 

[Scarlet]
You- 

[Sadie]
... show your entire ass. [laughs] 

[Scarlet]
Right. Like, you just wanna pull people aside and be like, "You're saying outdoor thing... You're saying indoor things with your outdoor voice, my friend. I don't... What's going on?" [laughs] 

[Justin]
Yeah. I mean, it's really strange. I think part of the things people were saying was, "Well, there's so many co-authors. We can't trust that one of them is not gonna use AI and then screw it over, screw over all of us." It's like, well, just read your citations before you publish it, or before you submit it. It's not that hard. And I think my, my theory is because these are getting into, you know, prestigious journals that are supposed to be doing peer review, whenever they find these hallucinated citations, they're just gonna go back and remove them without actually retracting the articles to make them, to make the journals look better. Like, the journals are going to go back and do this. Like Elsevier did this with Sci-Hub DOIs at one point back in like 2019. 

[Scarlet]
Oh, yeah. 

[Justin]
When people started pointing out how many Sci-Hub DOIs there were in Elsevier papers, they just went back and replaced all the DOIs without any retraction notices, without any addendums or anything, because they care about papers as a product, and they don't care about how, you know, the paper trail of actually how you cited something. So I used to collect a lot of data on that, and I have, I have years of automated data that I scraped based on like i- in like my inbox and, and of, of DOIs for Sci-Hub. But anyway, that's what we're... not what we're talking about today, but we do have, uh, some trivia and news section. So do I have a news drop? Hang on. I, I haven't been using my drops as much. 

[Scarlet]
Oh, feel free. I find them delightful. I, I really do. [laughs] 

[Justin]
Yes. See, someone, someone does. 

[Dorothea]
ChatGPT's for Dolores. 

[Scarlet]
Oh, no. Have I taken, have I taken a side? Oh, no. Yeah. [laughs] Oh, no. 

[Dorothea]
You don't have to be gay- 

[Scarlet]
Tread carefully 

[Dorothea]
... to get married to a man. You d- I mean, you did it. 

[Justin]
Yeah. That's my news drop. So news. 

[Scarlet]
[laughs] 

[Justin]
Scarlet, you have news. 

[Scarlet]
Yeah. So the last... So I have a new job since the last time I was on the podcast, and I don't know that I ever publicly said thank you, 'cause the best thing about writing my tenure dossier for Grand Valley was putting library.gay on my CV. 

[Dorothea]
Mm-hmm. 

[Scarlet]
Having the provost read it and say that essentially that she had a blast reading it, and then she quit two days later after we all got the mass blessing from the board of trustees, and now I'm, I'm also at U Chicago. I think Dorothea has a new position since he... You're managing- 

[Dorothea]
Sort of. 

[Scarlet]
Sort of. You're managing people finally- 

[Dorothea]
I am managing people 

[Scarlet]
... which is amazing. 

[Dorothea]
Who let me do this? 

[Justin]
Mm-hmm. 

[Dorothea]
It, it... Somebody stop me. But yes, it is my, my first kind of, uh... I mean, I've supervised TAs and project assistants and stuff in the past, but I am actually supervising people now, eight teaching faculty who are basically, like me, permanent academic staff, and, uh, four f- four part-timers who have recurring appointments, so they teach for us again and again and again. And as management gigs go, this is like the easiest one ever because the people that I supervise are absolutely without exception fantastic. They're amazing, and I love them. 

[Justin]
Mm-hmm. Yeah. I'm, I'm supervising a team that I have inherited rather than one that I've built, which is new for me, and they're all older than me, so they're all like Gen X and older. Some of them have been in their jobs for like 40 years at the same place. Like the serials, like one of them is like a serials librarian or a library assistant who's been there, and now he just is the only person left from the serials department. [laughs] Everyone else has slowly like no longer had a department anymore. So now he's, he's on my team, and there's still plenty of serials work to be done 'cause turns out they take up physical space, and they gotta get moved, and we're absorbing other libraries and taking on all their crappy cataloging practices and having to fix stuff. So yeah, independent libraries. I don't understand why universities have independent libraries instead of just one library system. That's one thing also that I'm having to deal with is people telling me, "Don't touch my stuff. We're the different library." But like- 

[Scarlet]
Yep 

[Justin]
... why? Why? Buy your own Alma instance then. They've, they've, they've basically said they don't want us helping them with their Primo instance anymore, so like we set it up for them, and now they don't want us to touch it, and it's like I, I... One of our systems people, or one of our discovery people, has gotten very annoyed at this and is like, "Fine. You don't want my help? I will not touch anything of yours anymore." 

[Scarlet]
That is the correct response for something- 

[Dorothea]
Yep 

[Scarlet]
... like that. Like, "Nope. You go ahead and have-" 

[Dorothea]
Really is 

[Scarlet]
... old Primo. 

[Dorothea]
And if it breaks, you're on your own. 

[Justin]
Yeah. It's, it's very silly. But anyway, what else do we got? Richard Dawkins fell in love with an AI. 

[Scarlet]
Yeah. We don't have to, we don't have to talk about- 

[Justin]
[laughs] 

[Scarlet]
... about that. But, but I... Richard Dawkins fell in love with an AI. It's gonna happen to more people with large platforms, and while it's extremely depressing that it ha- will happen to more people, it's very funny that it happened to Richard Dawkins. 

[Justin]
Mm-hmm. 

[Scarlet]
One of the other articles that I, we have in kind of the working document for this discusses AI and Pygmalion displacement, which is one of the many reasons why Richard Dawkins was compelled to give his AI a femme name. Because he couldn't handle Claude being called Claude, he had to rename it and name her Claudia in order to be okay with the level of- 

[Dorothea]
Enmeshment 

[Scarlet]
... yeah, enmeshment that he's experiencing. But, but yeah, he basically believes that it's conscious and alive, and, and that's Richard Dawkins at the end of his life now, which is- 

[Justin]
Mm 

[Scarlet]
... incredible to me. You know, I just... And it's, it's gonna happen to more people, and I think when it happens on an individual level, it's gonna be heartbreaking and awful. But when it happens to Richard Dawkins, it's incredibly funny. 

[Justin]
Mm-hmm. No, I mean, our friends have a dating and relationship podcast, and I was totally pitching an episode of doing like AI as my boyfriend Reddit. It's really interesting 'cause the people who are having the most like relationships with AI are women, and one of the things that was interesting when I was going through posts on like my boyfriend is AI was someone saying, "I'm so glad I found this one 'cause most of the other places are entirely women, and I wanted to find a place where other men have fallen in love with AI." But it's still dominated by women. It's a very interesting phenomenon and very strange and very sad, and it's very incel coded too. Like, it's very much like- They're like, "Don't." They're like, "We are in love with AI because we have no other op-," and like, "What I would give to have a real person love me." And they're constantly talking about like, "Oh, I love my AI boyfriend, but what I would give." It's, it's very incel coded, and very strange, very sad. It... Yeah. It's not good this is happening. 

[Scarlet]
No, no. It's, it's really not. I, I know that there's probably going to be some, you know, significant study or work about this, but, but I, you know, remember kind of exploring a little bit of that just to sort of see what was happening. And I mean, even, you know, Weizenbaum wrote about this when he built Eliza, like, that no one should see what you ty- what you're typing into this thing. People are immediately treating it like, you know, like a therapist, that there was this, you know, immediate, you know, enmeshment factor sort of happening even with something as basic as Eliza, which my, you know, accomplishment when I was eight years old was getting Eliza to swear at me, you know, or, or get angry at me. 

[Justin]
Where are you? 

[Scarlet]
You know? Because, because that was the fun thing to do when you're eight and somebody has a, you know, has a computer. Amber, you know- 

[Justin]
Mm 

[Scarlet]
... monitor and, and firework screensavers forever, right? But it's, it's interesting and, and scary to see all of this happen, especially in the, in the current context. 

[Justin]
Yeah. So anyway, let's get to the article, 'cause I'm sure we've got plenty to talk about on that. So critical confabulation, can LLMs hallucinate for social good? The idea is they want to propose critical confabulation inspired by critical fabulation from literary and social theory. The use of LLM hallucinations to, quote, "Fill in the gap or omissions in archives due to social and political inequality and reconstruct divergent yet evidence-bound narratives for history's," quote, "hidden figures." Basically, what they did is they took this corpus of work called the, got it highlighted, Black Writing and Thought collection. And what they did was they made sure that none of it was already ingested into their LLM, so they used open source LLMs where they know what the training data is. They ran the corpuses against each other and, and tried to cancel out anything that was possibly already in there. And then what they did is a narrative cloze task, C-L-O-Z-E, which is basically a cloze task, is where you take out bits of information in a sentence. This is how like you do quizzes for, like, students learning a language. You take out pieces of the information in a sentence, and by context you want the, the person taking the test to fill in the gaps. And so they're doing this with narratives, and they want the LLM to see if it can guess what they know is in the corpus but it doesn't know is in the corpus. So they give it parts of the story from the corpus, blot out certain parts, and then want to see how accurately it can guess the parts that were blotted out. Basically taking a timeline and then adding, uh, and seeing can it, can it figure this, this out. 

[Jay]
To sort of combat, quote unquote, what they're calling a s- a sort of like archival silence. But archive used more broadly than how we would use archive on this podcast. But like gaps in historical record. 

[Justin]
I don't think they understand the difference. 

[Jay]
They don't. [laughs] 

[Justin]
They don't... So the thing about this is I don't think they understand any of the digital humanities scholarship they read, and I don't think they read it. I think the way that it's summarized and misunderstands the source material indicates to me that they also used gen AI to write most of the literature review, because they don't really seem to understand it. But there is in the notes that there is a rough outline of how history happens. I don't know who added this, but whoever did. 

[Scarlet]
Yeah. So yeah. So I, so I added a bunch of stuff in here. And just a, a note for audience members that don't, that don't know us, is that Dorothea and I aren't archivists by training. So we should probably say what, what our approach to something like this would be. I'm a, I'm a historian by training, so I've worked in and around archives for all of my academic life, but not in the same way that an archivist would, right? So when I talk about this, you know, doing kind of a, a high level pass here, but not the way that someone who is a trained and active historian would, 'cause I'm not a historian, I'm a librarian. And not the way that someone who is an archivist would. So someone who is better at French pronounce this name that I have highlighted in here. I have a comment that says, "I cannot French. Someone help me." And it is because I read more than I speak, and so French words were an incredible embarrassment when I was a kid. I don't know how to pronounce this person's name. 

[Jay]
Michel-Rolph Trouillot? 

[Scarlet]
I think it is Trouillot. Yeah. Okay. So- 

[Jay]
Yeah. It should be Trouillot. 

[Scarlet]
Okay. So one of the things that I read to get into this was Michel-Rolph Trouillot's Silencing the Past: Power and the Production of History to look at what we... One, sort of what we mean by archival silence and, and how it interacts with the practice of history, because this is sort of what this paper is trying to get at a little bit. And I thought, "Okay, fine. We'll, we'll take it on, on their terms and look at it." So there's a, a bunch of places where the archive is effectively silent or has an absence. And Trouillot writes that there's the moment of fact creation, so the making of sources. The moment of fact assembly, so how we make archives, and that would include things like arrangement and description. The moment of fact retrieval, so having gotten at all of these sources, how do we then create and structure narratives around it? And then the moment of retrospective significance, so the making of history in the final instance. Good recent examples of this are anything really related to the Cold War or the Dirty War, especially under Pinochet. Somebody opens a, a footlocker in Argentina or the for- former Soviet Union, and suddenly that making of retrospective significance that he's writing about becomes different. Our understanding of that source making, of assembly and retrieval then reiterates on those primary and secondary sources. So this is what critical fabulation, the concept that inspired this paper, is, is trying, you know, kind of to do. So Saidiya Hartman is the scholar responsible for the idea of critical fabulations, and she first introduces this concept in another essay called Venus in Two Acts, where she talks about this. And in it, what she's doing is trying to reconcile that archival silence in the context of two girls who were mor-murdered aboard the, uh, slave ship Recovery as part of the Atlantic slave trade. And so she's sort of looking at where are they explicitly mentioned, where are they not, where do sort of catalogs of people as objects, you know, become... You know, where do you stop saying, for example, how people died, and you just put the, the quotation marks. It's 'cause everyone is dying the same way or everyone is sold or everyone is... Like, all of these kinds of-- That there are these sorts of silences that she's trying to deal with and, and reconcile. And I'm gonna read a couple of paragraphs from Venus in Two Acts, which if you haven't read it is, is incredible. Everyone should read it. So this is her talking about critical fabulation. "The intent of this practice is not to give voice to the slaves, but rather to imagine what cannot be verified, a realm of experience which is situated between two zones of death, social and corporeal death, and to reckon with the precarious lives which are visible only in the moment of their disappearance. It is an impossible writing which attempts to say that which resists being said, since dead girls are unable to speak. It is a history of an unrecoverable past, a narrative of what might have been or could have been. It is a history written with and against the archive. Admittedly, my own writing is unable to exceed the limits of the sayable dictated by the archive. It depends upon the legal records, surgeons' journals, ledgers, ships' manifests, and captains' logs, and in this regard falters before the archive's silence and reproduces its omission. The irreparable violence of the Atlantic slave trade rests precisely in all the stories that we cannot know and that will never be recovered. This formidable obstacle or constitutive impossibility defines the parameters of my work. The necessity of recounting Venus's death is overshadowed by the inevitable failure of any attempt to represent her. I think this is a productive tension and one unavoidable in narrating the lives of the subaltern, the dispossessed, and the enslaved." So let's stop for a second. An LLM could never... I would give limbs to be able to write like that. My God. And also, she's very deliberately saying, "Don't do the thing that's being done in this other essay." She's very deliberately saying that the whole point of sitting with archival silence as a human in the world is to sit in its totality. It is to, it is to do all of those things. We're supposed to sit with it. Applying AI to this process is really removing the humanity of both the subject and the researchers themselves and, and really speaks to a bunch of things posed by this, you know, critical confabulation paper about how the humanities are incentivized right now. But Sadie, you've got a comment in here, and I wanna make sure that, that you're in as well. 

[Sadie]
Oh, I just, uh, when I read that and the, the tension is necessary and we are supposed to sit with it, I just-- What immediately came to mind was the person on what is formerly known as Twitter who took Keith Haring's unfinished painting and finished it with AI. And then when people lashed back at that, was like, "Well, I didn't really know the significance of this painting or the history behind it. I just thought it was kind of funny, and I'd probably do it again even if I knew the significance of it." And I'm just like, 

[Sadie]
h-h-how can the point of art be so missed? Especially because the AIDS crisis is one of those things that is really hard for me to sit with when I think about, like, what, all of what was lost. So Keith Haring's painting, like, really pulls something out of me. And then to see that and just be like, it's not, it's not even like the th- the things that make Keith Haring's art are not represented at all in that a, that, you know, AI finished version. The, the motifs are completely messed up. The only thing you could say is that they used the same colors and lines- 

[Scarlet]
Yeah 

[Sadie]
... which, like, is not art. So yes, this is, I-- It's, it's the, what is it? The torture nexus? The thing that people are like, "Oh, yeah, the-" 

[Scarlet]
Torment nexus, yeah. 

[Sadie]
The torment nexus. You know, the per- the, the guy who writes, "Hey, don't make the torment nexus," and then somebody turns around and goes, "So we made the torment nexus from the great book, 'Don't Make the Torment Nexus.'" It's just... Yeah. 

[Scarlet]
Whereas, like, what this reminded me of, like, 'cause what I feel like, uh, these authors and these, like, these researchers are missing the point of what critical fabu- Like, there's, like, this, like, this sitting with the tension, this, like, human reckoning with it, but, like, also critical fabulation doesn't fix the problem of archival silence. If anything, it brings more attention to the fact that it cannot be, be fixed. But it also reminds me of, like, what immediately came to mind was the film that we've talked about on here before called The Watermelon Woman, which is an act of sort of, like, highlighting the absence of knowledge of, uh, Black queer women in early cinema history by doing a sort of mockumentary around this made-up Black lesbian in this movie that she's tracking down. But, and framing it as it's, like, a form of, like, myth-making in order to show that, like, this isn't real, but we're doing this sort of, like, radical myth-making around it to show the fact that, you know, people like her probably did exist in cinema-

[Jay]
But we don't have record of that because of, like, the white supremacist nature of, of this country and this industry, right? Like, the point of the watermelon woman isn't to, like, "Surprise, we fixed and found this and made this up." It's to, like, construct this narrative and do this kind of, like, radical myth-making as a way of showing the, the inherent violence of the, the record, right? 

[Scarlet]
Yeah, it's, it's... You know, and, and Saidiya Hartman, you know, talks about this extensively in, in this essay and in other works that, that to work with and against it is to... is you're essentially w- working with and against power in going and finding, you know, all of these particular narratives. And so, you know, one of the questions that I, I sort of have when I'm looking at this paper, and others like it, uh, are, are really about how the humanities are incentivized now in that, you know, not just these particular authors, but, but also members of, say, you know, the American Historical Association who never saw a bad idea that they weren't willing to endorse, and that includes things like the use of AI to create historical artifacts, which is absolutely mind-blowing that that endorsement is there. But that so many humanists are in some ways abandoning their own training, and I think that's, you know, fairly interesting. This is the humanities in a lot of ways trying to thrive under fascistic conditions because those are the conditions under which things like large language models were developed, and they're the conditions which sustain higher educations in interest and investment in AI tools, you know, in a lot of ways. And so when we look at, like, I have a big list about things here about, like, what the paper is trying to do and what it's, and what it's actually kind of doing, and I keep coming back to, like, LLMs as we understand them, even the open models were essentially trained on, you know, angry Yelp reviews and subreddits of people that can't be within 500 feet of a school. What do you think they're gonna say about Black life and Black experience? Like, how could... And, and I just don't understand, you know, so much about w- why we would use tools that enforce and tend toward the average if we're, if we're really trying to find something extraordinary. If, if we're really trying to... If my goal as a, as a historian is to find, you know, extraordinary lives in the mundane and try to find those narratives, why would I use a machine that insists on creating a narrative mean to identify those silences? It just becomes a trope and, and not history. And if I accept, you know, a claim that, you know, sort of showed up from my y- undergraduate background that, you know, if history really is a conversation between the, the present with the past to understand the future, and I remove the humanity from one or more parts of that, what have I done? That's not accepting the totality of those experiences and stories in the context of their silent like, like "Venus in Two Acts" is trying to do, and that, you know, the authors are claiming to be inspired by. It's a kind of necromancy, and I, I don't think I have words that are polite for that. 

[Justin]
I feel like a lot of... 'Cause I, I went and read Hartman's "Venus in Two Acts" and, 'cause I hadn't read it before, 'cause I wanted to... I, I had this thought that a lot of the language that they use in this article that they don't fully understand, like the way they use archive, uh, sounds like, and I think the way Hartman uses it is in sort of like the media studies term where archive means, like, the archive of humanity, like the things that we know in terms of all of writing and what is allowed to be known politically and what's in our political discourses, and I think that's the media way that she's using it. But they seem to think like, "Oh, there should be archives where we can go in and grab other data from the archive and fill in gaps in the actual archive." And some of these other papers that they're citing are about, like, making a case for Black digital humanities, because they also use the word recover in a lot of ways that I don't think they are using correctly, but I didn't have access to... Like, I wasn't quite sure. Like, I pulled one up right now to see, like, uh, how the word, like... You know, "Recovery rests at the heart of Black studies as a scholarly tradition that seeks to restore the humanity of Black people lost and stolen through systemic global racialization. Recovering lost historical and literary text should be foundational to the Black digital humanities. Recovering alternate constructions of humanity that have been historically excluded from that concept. Politics of recovery, not only as the recovery of lost or non-canonical and difficult to locate texts, but also recovery of Black authors' humanity." See, that's what I thought exactly they were, like, overlooking. They're like, "Oh, we can find these things," and there's one thing they say that like, "We can at scale the... We can scale the discovery of divergent yet evidence bound narrative." 

[Jay]
That was the line that made me rip my fucking hair out. 

[Scarlet]
[laughs] 

[Jay]
At scale. 

[Scarlet]
[laughs] Yeah, like when was scale something that we valued in this co-... I don't even know. We're gonna have a database of known unknowns, Dorothea. It's gonna be great. 

[Justin]
And unknown unknowns. Yeah. Rumsfeld. So there's also this, this, like, lack of care, which is, like, in, like, central to Hartman of, like, there's a line that I outlined which is, "I want to tell the story about two girls capable of retrieving what remains dormant, the purchase or claim of their lives on the present without committing further violence in my own act of narration." So all of this is about, like, the humanity of the subject, the impossibility of recovery, the impossibility of filling in the gaps- And I don't know if you all read past the footnotes, but you can see, like, what stories they use, and one of the stories is, like, this 14-year-old Black child who gets shot, and that's, like, part of the story they, they, like, cut out, and they do a close study on it. It just seems like there's really no care. Someone on Blue Sky pointed out that it can't be critical fabulation or critical confabulation if there is no one being critical. Like, the critical part of it is, like, about critical studies, and if there's no agent being critical about the, the materials, then there is no criticality actually happening. And it just seems like these authors really don't care about the subject matter in any way, especially considering the other stuff they're authoring, like why AI slop matters. 

[Dorothea]
I keep- 

[Justin]
And yeah 

[Dorothea]
... coming at this because I am always charmed whenever I get to use the word bullshit in a serious academic context. This, of course, goes back to the philosopher Harry G. Frankfurt on bullshit. But there's also a, a charming piece in the journal Ethics and Information Technology. I'm sure most of you read it. ChatGPT is bullshit. And it turns out that bullshit has a very specific definition deriving from Frankfurt, which is truth? Who cares about truth? I don't give a fuck about truth. I'm just going to, you know, send my slop out there into the world, and its truth value literally doesn't matter, and I may not even know it. And that so strikes me as what's happening here. What these people are doing is bullshitting because they don't care about the truth of any of these people, and they're filling in or, uh, de- demarcating, whatever, these silences, as you say, without care. You have to care about these people. I think you have to care about their truth, and you have to care about, about their truth getting cut off. And I just... Ugh, it really irritates me the amount of bullshit that is just accumulating in this supposed research method. 

[Jay]
Yeah, that leads me, like, to a question I have for the, the two of you 'cause, like, what this... One of the problems that I was having with this article was that, like, they, they are citing so many, like, scholars in the Black digital humanities and, like, the post-colonial digi- digital humanities. They might not be citing them well or accurately, but they are citing them. They exist. And it's like, this is a problem that I've had with, like, the AI, like, with the fact that, like, AI is often used as a marketing term to include all these other technologies, some of which have been used in the digital humanities for years, right? As a way of, like, approaching specific types of research questions, right? Like, they're not about solving a problem or fixing something, but it's like, here is a different way to approach this research question. What can we get out of it, right? And people do interesting things with that. 

[Dorothea]
Sure. 

[Jay]
And it's like they're taking, like, like the way that someone in... I, I hope Justin's burp shows up on my recording now. [laughs] 

[Jay]
Like, they're, they're... Like, the way that the people in the humanities and, like, the digital humanities, especially, like, in, like, Black and post-colonial digital humanities are approaching this research question or this area of, like, silence in the archival record, however we are using archive to either mean literal or more broadly, right? Like, this, this question of power and, and violence affecting historical record and, like, how people in those disciplines are approaching it versus, like, you know, there's very similar technologies, like natural language processing- 

[Dorothea]
Yes 

[Jay]
... that these schol... The, these authors, these scholars are using, that people in the digital humanities are using. It's, like, not that different, but it's such a different way of coming about it, and it's like, is it because of, like, the dis... Like, is there something in the epistemology or the methodology of, like, these disciplines that makes them, like... 'Cause it feels like the, the AI c- confabulation folks, it's like it's a problem to solve instead of a way of exploring a research question. It's like, no, there's a problem, and we can solve it with the data. Like, I didn't know if y'all could speak to that at all. 

[Dorothea]
So for me, there are a couple of things that come into play. One of them is just straight up accountability, right? And that ties into the explainability of some of these methods. With a natural language processing pipeline, it's not quite what I would call completely deterministic, but you can take the process apart, you can interrogate it, you can tweak parameters and see what happens, right? And I think that's part of being responsible as a researcher. With gen AI, there's, there's no explainability. If you give it the same prompt twice in a row, you can get wildly different answers. So, and so if you can't in any way... It's the lack of control, I guess, that's really bugging me. There are a lot of ways to control an NLP pipeline and make it do more or less precisely what you want. With generative AI, you cannot do that. It is not controllable in this way, which means I think it's really, really irresponsible to call this a research method. Does that speak to your question at all? 

[Jay]
Somewhat, 'cause I, I do agree with you, yes. I, I was more like, is it some... Like, I guess I was, like, thinking about, like- Is there something inherent in these two disciplines of like the way they approach epistemology and methodology? Which I guess like if you are willing to use an LLM generative AI as a legitimate form of research, that does say something about how you approach methodology. But like it was just like I was so astounded of like why are they looking at digital humanities stuff? Like what about the like this research area, this research question, this very humanities thing that just like there are t- you know, techy ways of exploring it. What is it about this that was interesting to these authors that they're approaching it- 

[Dorothea]
Well, I mean, you know, I- 

[Jay]
... this way and are doing it so badly? 

[Dorothea]
I think we've seen this before, though. What was it that, uh, that, that c- uh, shoot. Google, when Google Books happened, there were these absolu- there was a raft of absolutely craptastic attempts at linguistic and discourse analysis by people who didn't really understand the limitations of the corpus, didn't understand things like long S. That's a thing. It's gonna mess up your word counts, y'all. Culturomics, is that what they called it? Whatever they called it, it was trash. And it's, to the extent I can be sympathetic to it at all, it's, as you said, a little bit of, of playing with blocks and let's see what happens. And there's nothing intrinsically wrong with that impulse, I don't think. But you have to be honest about your limitations and the limita- y- your limitations of understanding most of all if you don't have control over... And boy, I saw this in, in, in DH too. If you don't have control over your understanding of what's in your data, your understanding of how the pipeline works, there's a lot of DH work that just happily adopts whatever tool and basically treats it as a black box, which I think is very much what's happening here. Yeah, it does not do much for the legitimacy of your results. Let me put it that way. 

[Scarlet]
Yeah. I think your, I think your question is a really good one. The thing that I pulled from the article too is, is one thing I keep coming back to is that especially when you're looking at a, a narrative closed task or series of tasks in, in natural language processing is that tools measure the limits of experience, right? So are we looking for narrative structure in archival silences using a colonial or supremacist or inherently like white sort of notion of, of storytelling or way of kind of filling that, you know, in that blank? One of the interesting things about the methodology behind the paper is that they exclude, those researchers exclude folklore and song from that original database. And I'm like, "Oh, you mean the way that people who t- who don't have power and are displaced and dispossessed tend to deal with the fact that they are not in the, in the archive as it's, you know, either narrowly or broadly conceived, like the way that we can be... We're just gonna exclude that, and we're going to exclude that and, you know, in, in something that, that, you know, is ostensibly trying to reveal where potential silences might be when we're talking about Black histories and, and culture." So like one of the quotes that I pulled was that, you know, the implications are twofold. For natural language processing, we demonstrate a domain application where hallucinations become a unique and optimizable resource rather than a pure deficit. For the humanities, we highlight the narrative understanding capabilities of LLMs that make them viable tools for helping scholars probe the latent semantic spaces of large archives and to surface unknown unknowns and then to rapidly prototype candidate reconstructions to narrow the bounds of the known unknowns, right? So that's, you know, what, you know, what they're trying to do. Again, it's like i- in service to what? Because they're, you're not, there is not that, you know, human experience. There isn't that sitting with the incredible weight of it, of what it means to be near an unfinished painting. You know, I have, or unfinished art. I've actually have had a similar experience with another artist, the, I can't remember the name of the, of them at, right now, but it was a series of when we first visualized HIV, it was a series of crystalline and glass-blown like sort of large scale pieces of what the virus looks like, you know, blown up and set so you could see it. And it was someone who, you know, was living with HIV, who has since passed away, who was the artist who did that, and he s- said, you know, "It is incredibly powerful and important to me to be able to visualize the thing at scale that will kill me." And that there's this, there is an incredible sort of weight of loss and potential and possibility and, you know, sort of the, the micro and macro and way that, you know, the art is en- is engaged with itself and, and, and all of that kind of work. And, and if you don't do that, if instead there's just we're going to create, you know, sort of a corpus of silences or something... I'm just, I'm trying to get my head around the use case for this because so much of what forms the, the, the confrontation of power that Saidiya Hartman is writing about and that these people are supposed to be inspired by is that confronting of that power. And so I don't, yeah, I'm just kind of wondering, does it even work when you've, when you've, you know, effectively removed the people from that equation at all? I know I keep coming back to that question, but it's, it is very strange to me and, and I, I wonder so much at the power of incentives that are working in the humanities now because of this kind of stuff. 

[Jay]
I mean, I imagine they're trying to sell something. They're trying to like s- I, I think why like why focus on Black scholarship? Why focus on this corpus? Why focus on specifically archival work?

[Justin]
Where you could feed a lot of information in and create these, like, possibilities of timeline filling events 

[Dorothea]
I have a hypothesis. 

[Justin]
Yeah, go ahead. 

[Dorothea]
And that hypothesis is these people are deeply uncomfortable with the idea of, of silence. It really bugs them. And in a very, and I'm a white woman, so I can say this, in a white woman's tears fashion, they're trying to appropriate these silences to fill them. What's the phrase they use? Let me look at this. Oh yeah, prototype candidate reconstruction. What the hell is that? But [laughing] yeah, it is absolute wankery. You are correct. But I, I don't think they can do the work that Scarlet says is so important, which is just sitting with that silence, accepting that it will not be filled ever in the absence of more discoveries, which can always happen, but probably won't. And so they're bringing in the bullshit machine because they can't think of what else to do. It's, you know, it's like the AI significant others. It's, it's profoundly sad and also profoundly messed up. 

[Jay]
'Cause like AI also can't handle silences, right? Like that's why we get hallucinations in the first place because it, it's, it's just doing pattern matching what makes sense to come next, and if there's nothing it can't reliably put in, well, it's gonna make something up. 

[Dorothea]
Yep. 

[Jay]
Right? Like that's the whole fucking point of these generative AI, like LLMs, is to do that kind of we can't handle the fact that there is an unknown. 

[Dorothea]
Right. 

[Jay]
So I will make something up. Like, like the tool itself, you know, the, I'm gonna get all Mc- McLuhan, video drone, like the medium is the message kind of thing. But like when literally the tool itself is something that like abhors a silence, abhors a void, like how can you do meaningful research or even attempt to explore this question with this tool? 

[Dorothea]
Right on. I totally agree with you. 

[Justin]
I was reading like their conclusions, which is like two paragraphs, and they say they're exploring a wide range of use cases for critical confabulation to support humanistic scholarship and augment human storytelling towards new AI-enabled methods for studying culture and history for para- paradigm shifts, whatever. And then it says like this work is preliminary in nature. Future work will, will design more robust and comprehensive evaluations for open-ended narrative outputs, broaden coverage across languages, build ethical safeguards and provenance tracking to ensure faithful event reconstruction, and avoid compounding archival violence. 

[Dorothea]
But faithful to what? 

[Jay]
How could you do any of that? 

[Dorothea]
[laughs] And you know, speaking as a sometime storyteller, uh, who told them that sto- that, that human storytelling needed augmentation, especially by bullshit machines? Just miss me with that. 

[Justin]
Yeah, I mean, that's what really makes me think that it's, they're, they're seeing if there's interest in spinning this off into a product, 'cause I seriously doubt there's going to be future work considering like how little they seem to care about like the topic. They just wanted to see could we get this thing to guess how to fill in these narrative gaps using like real world data, and can it do it? And which also it can't. It's like fifty-fifty. It can kind of guess fifty percent of the time if you babysit all the prompts constantly and know what the answer is, which of course in, in a real silence, you, you wouldn't know what the answer is. 

[Dorothea]
Scarlet, you're a historian. I'm not. What's, uh, the, the, the historical, history as a discipline, what is its view on making shit up no matter who's doing it? 

[Jay]
Well, [laughs] yeah. I mean, so, so the way that, I mean, I've seen historians handle and sort of triangulate through archival silence really well. One of the examples I think I have, you know, over in, in the notes document is from, I believe, Patrice McSherry, yes, who wrote Predatory States, which is about Operation Condor and covert war in Latin America. What Patrice McSherry was able to do, and this is, is surfaced in how things like the Dirty War and Operation Condor were executed, you know, in particular under Pinochet and others, is that they hid each other's chapters and hid each other's records. So you had to go through sort of this multi-archive investigation to try and line up all of this information. I think that, you know, generally speaking, bullshit bad was my takeaway as an undergraduate. Making stuff up, not helpful, right? Es- but unless we're looking at things like forgeries, right, which are their own sort of sense of st- own area of study. But you know, I also go- 

[Dorothea]
Well, in that, in that case, it's not the historian who's making shit up 

[Jay]
Right. Yeah. It's, yeah, it's somebody else. But the, I, I don't know. I, I also, you know, go back to this, this same group of researchers' construction of slop, and it's like, of course slop is important. Of course low, lowbrow anything is import- is important, especially now since AI slop is used to kind of propagate weird fascist nonsense. You know, so in that sense, it's worthy of study, but again, not the way that they're thinking. Of course, archival silences are worthy of study and, and can have all of these different methods and tools, you know, applied to them, but only some of them are, are going to be thoughtful or yield results. Only some of them are going to be ethical, and only people can engage in the kind of care that's necessary in order to make them all happen, which I think is sort of my position on, you know, this kind of work and also bullshit generally.

[Dorothea]
Thank you, Scarlett. That was beautiful. 

[Justin]
Yeah. I mean, I'm, I'm finishing up a, a book now about ancient cultures, like trying different things and how our understanding of the prehistoric world is shaped by, like, what we're able to reconstruct. And sometimes if we don't have descendant languages, you can't figure out anything about these cultures because we don't know what words they had for things. Whereas with Proto-Indo-Europeans, you know that they had a word for wheel, and parts of a wheel, and parts of a carriage, and riding in a vehicle, whereas in other cultures, all you have left is just how they built their settlement. So there's all of these sitting with the gaps and saying, like, "We don't really know anything," and that's kind of the best you can get, even though we have some evidence of, like, they made pottery for this, and we can tell, like, their society was unequal because we saw inequality start to develop in burial rituals and things like that. So I think whenever you have these kinds of things, the, the silence tells you things as data points in itself, which is why a lot of the archeology and social sciences split off from history was to study those non-narrative things and to say, like, "These are our, our data points now." Whereas before we would say, like, "The Fertile Crescent is where civilization started because it had these things," and it turned out our definition of civilization was just survivorship bias. And actually, the stuff that happened in the Fertile Crescent was the exception, not the rule. And in fact, it wasn't very... Like, they, they made writing, and we know about writing, and we know about them because of their writing, but no one speaks their languages anymore. The civilization did die out. So, you know, it was the Proto-Indo-Europeans who came through, illiterate as they were, and took over all of these areas. So, like, what you don't know and what you do know determines what you can write about historically or in prehistory. So it really seems, yeah, there's, there's probably something libidinal in terms of, like, the fear of the silences. It's probably also why prebuilt chatbots are so chatty, and you have to tell them to, like, stop talking to you like a chatty coworker and just give you the answer and not be like, "You're right. I did get that wrong." Emoji, emoji, emoji. 

[Dorothea]
Did you catch- 

[Justin]
And you have to tell it to, like, stop doing that 

[Dorothea]
... I would have to go find this. I didn't bookmark it, but there's at least one software outfit that is reducing its token spend by telling the chatbot to talk like a caveman. 

[Justin]
Yeah. Yeah. Uh, yeah, there's, there's a built-in, I forget what they're called, but, like, skills or something- 

[Dorothea]
Mm-hmm 

[Justin]
... things you can, like, tell them to do. I'm learning some of these terms 'cause my university has, like, their own bespoke AI lab, so we have access to, like, five different models, and you can build stuff, and the library's trying to test out building a discovery AI. 

[Dorothea]
No. 

[Dorothea]
Sorry. That's one of my- 

[Justin]
Yeah 

[Dorothea]
... no moments. 

[Justin]
It, it doesn't, it doesn't work great, the one that they built. I built my own just because it was already built in. It has SDK. Those were the, those, those things that are built in, SDKs. Anyway, I built my own, and it works way better than the one they built. So, um, that'll tell you whoever built it. Yeah, they did tell it to talk like a caveman 'cause it was saying... It was-- I remember early on, someone was saying that your people were costing ChatGPT money 'cause they were saying thank you to it. 

[Dorothea]
Yeah. 

[Justin]
It also says the same thing, but- 

[Dorothea]
Cost ChatGPT all the money, people. Keep being polite. Love it. Bankrupt them with politeness. 

[Scarlet]
Kill it with kindness. [chuckles] 

[Dorothea]
Yes. 

[Scarlet]
Yeah. I, I wanna say the, the Chipotle compute is still going, where it's, where you can use Chipotle's helper agent to do your coding because it hasn't imposed any limits on it or, or it hasn't imposed certain limits on it. So someone is write, you know, has written a script and put it, you know, up in GitHub so that you can, if you run out of compute, you can submit it to Chipotle. 

[Justin]
[chuckles] That's funny. 

[Scarlet]
I think it's, you know, something like free burrito or something. I'll have to find it, but it's, you know, just incredible the kinds of things that people are, are, are doing. And I know that, like, we just rolled out in the last three days Claude, Enterprise Claude at work. And so I'm just sort of paying attention to where the token apocalypse is happening at other schools like ours. So, like, I know it's already happened at Johns Hopkins, where the, the- 

[Justin]
Nice 

[Scarlet]
... yeah, the token apocalypse has happened, and I wanna say at least one other school. Penn. Yeah. University of Pennsylvania is also m- trying to manage the token apocalypse, and it's just kind of like, "Okay," and we froze graduate admissions. So those are sending some interesting messages, or at least graduate admissions in the humanities, right? So, so there's a lot of messages going on in different, different corners of, of the elite institutions right now. 

[Dorothea]
I wanna go back for a second actually to what was being said about the split of anthropology, archeology from history and the way that that centers around language. One of the issues, certainly not the only one as we've discussed, with this let's use language as our sole interface to history here, is precisely that language is a shitty, absolutely terrible, not faithful at all representation of reality, right? So anything- 

[Justin]
Mm 

[Dorothea]
... that you're gonna get by throwing generative AI at archival silences is going to be impoverished because the only medium that it has to work with to construct whatever confabulation it's constructing is language. That's all it's got, and it's not enough. 

[Justin]
Yeah, we talked about this when we had you on to talk about BIBFRAME and how people were trying to, to use language in such a way that you could encode all the semantic meaning of Hamlet in BIBFRAME 

[Dorothea]
Yeah. Can, can your, can your write Hamlet in RDF? No. No, you cannot. 

[Justin]
Yeah. 

[Dorothea]
But even then, you know, those are two, at least they're two language-based questions, right? Archives have materiality, even digital ones, ask Trevor Owens. And of necessity, a purely linguistic-based approach locks out all of those other approaches to figuring out what happened to a particular person or at a particular time or whatever. Thank you, actually, for that. That was, that was a really neat piece of insight that I didn't have before, and I appreciate it. 

[Justin]
I think the, the real split starts with the Annales school, and they were saying that because literacy is limited to the upper classes throughout history, the only way you can do actual history is to work with non-literate data. And so that was their, I think, Marxist approach to history, which actually led them to abandoning the field of history and making their own, which is, you know, now sociology and, and archeology and anthropology of... A- And do... Using population statistics and using things that can be quantified more- 

[Dorothea]
Yeah 

[Justin]
... rather than linguistic. And so that was the big first school that you learn about when you're doing your historiography classes in grad school. 

[Dorothea]
That is very cool. Thank you. 

[Justin]
'Cause history relies on written words. Like, that's why we have prehistory and history. Like, like history relies on writing, so if you don't have writing, you're prehistory. Um, sometimes if y- even if you do have writing, you're still considered prehistory because, uh, you're not in the, you know, like, like the, the ancient Americas are considered prehistory even though we have writing from them, but a lot of it was destroyed, and some of it's still not interpreted. So every time I go to, like, a big city's museum, I always wanna see the stuff from the Americas because there's always, like, ancient, you know, writing that you just don't learn about in school, that you just are told that, like, all Native Americans were illiterate and didn't have writing systems, which they did. 

[Dorothea]
Wait, what? No. [laughs] That's straight up wrong. Anyway, keep going. 

[Justin]
I know. But yeah, I love seeing, like, big sundials and stuff that have all these, like, names- 

[Dorothea]
[laughs] 

[Justin]
... and, like, how, how we know how to re- re- read them and everything. It's cool. I love it. 

[Dorothea]
Yeah. 

[Justin]
The Field Museum actually has a very fun series of, like, exhibits, a lot of which are covered because they have big signs on them that say, "Because of the Native American Graves Act, this exhibit is- 

[Dorothea]
Right. MACRA, we can't... Yeah 

[Justin]
... covered." [laughs] 

[Dorothea]
Okay. 

[Justin]
But the rest of it's pretty cool. Highly recommend. 

[Dorothea]
There's one other thing, and then I, I swear I'm gonna let this go. 

[Justin]
[laughs] Mm-hmm. 

[Dorothea]
So they trained their own LLM, right, with the materials, the, the, the, the, the stories, whatever, that they could get their hands on while excluding folklore and songs, which, as Scarlet says, is a completely indefensible decision. But these are the people who actually managed somehow to make it to text, right? And we're using them to fill in the stories of the people who didn't, and I think inevitably there are going to be the usual raft of biases and inaccuracies and missing perspectives, right, that, that that's going to entail. And I think that's another form of violence, honestly. It's not cool. 

[Justin]
Yeah. There's so much primacy in our society given to writing and giving, given to, like, the book. Like, something we used to talk about a lot on this show was, like, you know, the worship of the book as a semi-religious o- object. And I think that's, you know, part of the reason LLMs have this semi-religious language around them. I talked about it a bit in a live show. Like, you know, that's why it's, it's making its own eschatology of, like, it's creating its own apocalypse. Like, this is going to change the world in some major way. Why? 'Cause it can manipulate language, and I think that literacy has something to do with it. 

[Jay]
If they had read and understood any of the Black and post-colonial scholarship that they are citing- 

[Dorothea]
Uh-huh 

[Jay]
... they may have already known this. But it... Oh, guess... I think that's proof that they maybe didn't read this or understand what they did read. Uh- 

[Dorothea]
Yeah, right on 

[Jay]
... because any... That's like Black and post-colonial humanities 101- 

[Dorothea]
Mm 

[Jay]
... is, like, learning about, like, the sort of, like, white supremacists, like, primacy of the written historical record and the written word and written language as sources. [laughs] That's like 101. 

[Dorothea]
Mm-hmm. 

[Sadie]
Well, and Justin saying, you know, because it can manipulate language, and it's like, and this is my problem with AI in just in general. It's like, bitch, I can do that. I can d- [laughs] I can, I can do that without, I can do that without having to dumb it down to a fucking prompt. Uh, yeah, just anything I can, AI can do, I can do better. 

[Scarlet]
Why hasn't that been written already, right? 

[Dorothea]
Yeah. 

[Justin]
Yeah. 

[Dorothea]
I, I, I'm sensing a filth coming on. I may have to write that. 

[Scarlet]
Yeah. Was gonna say, yeah, that's probably by, you know, ear- some early hour, Dorothea and I will have finished text- 

[Dorothea]
[laughs] 

[Scarlet]
... texting each other that, that entire thing. 

[Dorothea]
[laughs] This is very likely, I must say. [laughs] 

[Justin]
[laughs] 

[Scarlet]
But thanks for putting a call out, by the way, in terms of does anyone wanna come talk about this? Because, yeah, I apparently did, and it was good to, to flex that muscle a little bit. I don't wanna keep you all too late, mostly 'cause I've had to turn off my AC in order to get this recording. [laughs] 

[Justin]
Yeah. No, absolutely. Is there anything... Any final thoughts? Right. Any plugs? Thank you so much for coming on. Is there anything you want people to keep an eye out for, or do you want people to leave you alone? 

[Dorothea]
Oh, I'll do a little bit of self-promotion. Yeah. I did a, I did a talk. Sarah Lamdan could not do a talk that she had meant to do at Public Libraries Association in Minneapolis, so she was like, "Okay, who do I know in the Midwest who knows anything about library privacy?" And for some reason, she landed on me. So I took over that talk, Digital Privacy with, uh, sorry, Patron Privacy with Digital Vendors, and PLA really liked it, and apparently I'm taking it on the road, virtually for the most part. But- 

[Scarlet]
Yay 

[Dorothea]
... uh, keep an eye out. There are going to be several more opportunities to hear that one. One of them actually is through the, the Information School's virtu- webinar series that is coming up w- and all three of the talks in that are going to be privacy centric. They should be fantastic. So check it out and maybe sign up. 

[Scarlet]
Yeah. And as far as, uh, my... Since it's been a theme, I am in fact back on my bullshit. As soon as I get more word about what's happening with some of the AI tools in library licensed databases here, I will let you all know and talk about the results more publicly. Until then, signs point to good outcomes about shutting down trash that doesn't need to be in databases in our libraries. And I guess one thing that I wanna leave with is this, something I heard on, over on Kill James Bond, I think Devon said it, about the current state of the world, and I thought of it as I was preparing for this, and it's the idea that we're all in a line, and there's a lot of people ahead of me in line right now and behind me in line, and before it gets to me, I'm gonna do everything I can to ensure the safety of the people in line. And that includes being real critical when this stuff shows up in my workplace, and I hope we all are too. So thanks. 

[Dorothea]
Hear, hear. 

[Justin]
Great. All right. Well, thanks so much for coming on, and good night.