image: thumbnail.jpg
Hacking Productivity with LLMs (Nic Crane, NC Data Labs) | posit::conf(2025)
💬 Speaker(s): Nic Crane
📝 Abstract:
Over the past 18 months, I've used large language models (LLMs) in a series of experiments to improve my productivity as a developer on open source projects in R. I wanted to see if these tools could help with the repetitive, frustrating, or time-consuming parts of development, freeing me up to focus on the interesting problems. The results were mixed: some of these experiments were wildly successful, but others were clear failures.
In this talk, I'll share what worked, what didn't, and what I learned. I'll explore important questions to ask around the kinds of tasks LLMs are good for and where they fall short, how to use LLMs to build lasting skills, and why in some cases, investing in human connection pays off more than AI assistance. posit::conf(2025) Subscribe to posit::conf updates: https://posit.co/about/subscription-management/
Transcript#
This transcript was generated automatically and may contain errors.
Helo, nic Crane ydw i. Rydw i'n ymgynghoriadwr arbenigol ac adnoddwr ffyrdd ag awgrymu. Heddiw rydw i'n siarad â chi am Hacking Productivity gyda LLMs ac mae rhai brofiadau wedi'u gwneud dros y 18 mlynedd diwethaf.
Fodd bynnag, roeddwn i'n meddwl yn ddiweddar am swydd rydw i'n ei wneud yn ystod fy nghyfnod. Roeddwn i'n gweithio ar y prosiect, ac fel rhan o'r prosiect, roedd gen i'r swydd diwethaf o adnoddau database yn fform CSV, ac roeddwn i'n ei ddatblygu i Excel, ac roeddwn i'n ymwneud â ffyrdd a ffiltrau gwahanol, ac roeddwn i'n eu cael yn unig yn iawn, ac roeddwn i'n ei rhoi i'r cymdeithaswyr.
Roedd y gwaith ei hun yn anodd, ac roedd hi'n ddiddorol iawn. Roedd hi'n gweithgaredd gwirioneddol iawn, ac os oedd pethau'n mynd yn iawn, roeddwn i'n rhaid i mi ddechrau o'r dechrau eto. Roedd hi'n mynd i'n rhaid i mi fod yn risg iawn o sefydliad yw ILD, Imminent Laptop Defenestration.
Ond, yn hytrach na rhannu fy mhysyn a mynd i'r ffyrdd, rydw i'n dechrau cyrraedd ymwneud â pheth mwy o ddefnyddiad gwirioneddol. Roeddwn i'n creu gweithgaredd gwirioneddol, ac roedd y gweithgaredd gwirioneddol yn cysylltu â'r ddatabase. Gall y defnyddwyr ddefnyddio'r ffiltrau a'r pethau sydd eu hangen arnyn nhw eu hunain, ac roeddwn i ddim yn rhaid i mi wneud y gwaith anodd hwn bob dydd.
Felly roeddwn i'n hapus, roeddwn i'n gallu sefydlu ar bethau mwy ddiddorol, ac roedden nhw'n hapus. Nid oedden nhw'n cael ymddangos arnaf i mi ar y pwynt honno. Ac, mewn gwirionedd, y peth gyda chreu gweithgareddau a phethau i gynyddu'r pethau, mae bob amser yn ymwneud â'r risg, mae'r cyflwyniad rydych chi'n ei ddod o hyd yn oed, a'r amser rydych chi'n ei ddod o hyd i'w rhoi, yn gallu cymryd llawer o amser y byddai'r tasg sylfaenol yn y lle cyntaf ei hun.
Ac rwy'n credu, mewn gwirionedd, mae hynny'n rhywbeth y bydd llawer ohonoch ni'n meddwl amdano ar hyn o bryd o ran AI.
Ac felly, fel y sgrifennodd Cat Hicks yn ei gynhyrchiad y diwrnod diwethaf, nid yw'r unig beth y gallai'r tŷau hyn ei wneud, ond mewn gwirionedd, ein profiad o gysylltu â nhw. Ac mae hynny'n rhywbeth sydd wedi ymddangos iawn gyda fi. Dw i wedi treulio llawer o amser, dros y 18 mlynedd diwethaf, yn ymwneud â llyfrgellau LLM, ac yn ceisio cynyddu gweithredu mewn fforddau gwahanol. Mewn gwirionedd, mae hynny'n helpu iawn, ac mewn gwirionedd, rydw i wedi gysylltu â llyfrgellau cyfan i rhywbeth sydd ddim wedi gweithio.
Felly, dydw i'n mynd i siarad â chi am fy mhrofiad o hynny, ac mae tri cwestiynau rydw i bob amser yn ymddangos ar hyn o bryd pan rydw i'n penderfynu a ydw i'n defnyddio llyfrgell LLM mewn gwasanaeth neu ddim.
Felly, y cwestiwn cyntaf yw, pa mor anodd y bydd y gwasanaeth? Rydw i'n mynd i ddangos hyn trwy siarad am rhaglen R rydw i wedi bod yn gweithio arnyn nhw'n ddiweddar. Felly, mae'r rhaglen ei hun yn ei enw Daisy UI Widget. Mae'n rhaglen HTML widgets, ac mae hynny'n golygu ei fod yn defnyddio rhaglen HTML, JavaScript, CSS a R, ac mae'r defnyddwyr yn gweithio gyda'r rhaglen, mewn gwirionedd yn R.
I weithio gyda'r rhaglen, bydd y defnyddwr yn creu ffram ddata gyda dyfodol a sylwadau. Byddent yn ei enw Daisy Function Timeline, ac byddent yn cael ein timeline, fel yr un ar y dref yma.
Felly, rydw i'n penderfynu, pan oeddwn yn dechrau'r prosiect hon, i weld a allwn ei wneud o'r blaen gyda llyfrgell LLM. Felly, rydw i'n defnyddio copilot GitHub, ond rydw i'n dod ac wnaethu pompio neu gweithio i rhoi codi.
Felly, sut a oedd y prosium yn mynd? Mae'n dweud, ymterwyl! Na, nid gwyddon ni'r rhan honno, ond dywodd rhywbeth y byddai arall yn rhaglen R. Ond, pan fe wnaethon ni wneud y ffinctrion, mae'n debyg o'i radd, roedd yn teimlo iddo y data Jeavoscript a Cyflog CSS oedd ar gyd gydol y chwaraeth mae o nich du. Roedd propfon hyn yn braiu cadarn.
Concludais, Nid oeddwn i'n gwybod lle i ddechrau gwneud hynny. Roeddwn i'n dweud, os yw'r gwaith yn eithaf anodd neu anhygoel, gallwch chi ddechrau a chynhyrchu ychydig o codi, ond pan fyddwn ni'n dod i'r pwynt hwn o ddod i'r cwestiwn cyflym, bydd pethau'n gweithio'n iawn yno.
Felly, ar hyn o bryd, roedd gen i ychydig o'r rhaglen, ond, yn hytrach, roedd fy ffrind i'w helpu i'w datblygu, ac roeddwn i eisiau gweld os oedd LLMs'n gallu helpu. Felly, roeddwn i'n ddechrau gwneud ymlaen ar y tasg ddau.
Felly, roedd y slot AI rydw i wedi'i ddod allan o'r copilot GitHub wedi dod â'r ffordd hwn o weithio gyda'r rhaglen, lle roedd y rhaglen FunctionDaisy yn cymryd llys o llys yn ei chynhyrchu. Ond mewn gwirionedd, ar gyfer pethau fel hyn, mae'n fwy cyffredinol i gael rhaglen data fel cynhyrchu. Felly, yr hyn rydw i'n ei wneud yma oedd newid y rhaglen o llys o llys i'r rhaglen data.
Felly, roedd e'n teimlo fel tasg. Roeddwn i'n gallu deall sut i'w wneud fy hun. Nid oedd e'n cymryd llawer o amser, ond roeddwn i eisiau gweld beth a oedd yn digwydd os oedd gen i LLM i'w wneud.
Iawn, felly, sut mae'r rhaglen hon yn mynd? Mae'n bwysig iawn. Felly, roeddwn i'n digwydd gyda'r codd anhygoel hwn sydd wedi bod yn llwyr o llwybrau rydw i erioed ddim wedi gweld. Roedd y codd yn ymwneud â llawer o testau. Ar gyfer y rhan fach o codd, rydw i erioed ddim wedi gweld llawer o testau yn ymwneud â hyn. Roedd y codd yn teimlo'n fwysig iawn i gael mynd i gysylltu â'i gilydd. Roedd y codd yn ymwneud â'r codd yn ddiddorol hefyd.
Felly, roedd y codd yn gweithio ac roedd yn ymwneud â'r hyn sy'n ei ddweud. Ond, pan roeddwn i'n mynd trwy'r codd, roedd llawer i fynd ag e. Rwy'n teimlo bod cadarnhau codd yn mwy o wrsg cymdeithasol. Gall fod yn mwynhau i gysylltu â phobl. Ond rydw i'n dod yn ôl i gofyn, a oedd y codd i'w defnyddio'n ymwneud â'r cyfrif? a oedd y model arall a'i ddefnyddio? a oedd y data a ddifrifoeddodd arno? A oedd y codd hwnna'n dda iawn? Ac, yn siŵr, ro'n i'n gwerthfawr i fynd â'r cwestiynau hyn, ond yn ddifrifol
i ddefnyddio'r cyfrif. Felly, os yw'n ei ddefnyddio'n ymwneud â'r cyfrif, yn y cyfrif hon, gallwn gael rhywun yn clicio ar gyfer unrhyw beth yn y cyfrifiad, ac yna gwerthfawr yn cyfrifio'r cyfrif fel y gallwn, efallai, ddangos rhai teistiau sy'n ymwneud â'r cyfrif. Ac roeddwn i eisiau gweld a allwn i'w ddefnyddio hyn.
Felly, eto, roeddwn i'n dod â'r codd. Roeddwn i'n dod â'r codd. Roeddwn i'n dod â'r codd. Roeddwn i'n dod â'r codd. Felly, gallwn ei ddefnyddio fel hyn.
Felly, ie, os yw'r tasg eithaf hawdd, mae'n teimlo fel, ddangos mwy o amser, os yw'r tasg eithaf anodd, dwi ddim yn gallu ymwneud â'r codd, ond mewn gwirionedd, os yw'n fwy anodd, mae hynny'n lle ddiddorol.
Ac mewn gwirionedd, pan ddewisais hyn fel hyn, roeddwn i'n sylweddoli model i ddatblygu a datblygu. Felly'r syniad hwn, roeddwn i'n mynd i weld a allwn i ddatblygu LLM i ddysgu i mi sut i'w wneud yn fy hun, sy'n mynd i'r cwestiwn cyntaf. Pa rôl y byddai'r LLM yn ei chwarae?
Felly, ar hyn o bryd, dewisais ymgyrchu rhai o'r llyfrgell yno am ddefnyddio LLM mewn addysg. Ac roeddwn i'n darganfod papur gan Dr. C. Lynn am sut y gall ChatGPT helpu fel adolygiad dysgu bywyd.
Felly, yn y papur hon, mae Dr. Lynn yn mynd trwy'r fframwaith hwn i ddysgu a phethau sy'n benodol y gall ChatGPT helpu gyda hyn. Felly, mae'r ddau gyntaf yn y fframwaith yn sefydlu'r golau dysgu, sefydlu a chysylltu'r arian cyhoeddiol ac yna ddefnyddio a chynhyrchu'r gweithgareddau dysgu.
Felly, dewisais i roi hyn. Felly, i fynd yn ôl i'r tasg yna, y tasg ymgyrchu bywyd. Roedd hi'n teimlo'n tasg strech sy'n perffaith i'w defnyddio yma, oherwydd rwy'n cael profiad o'r tasg hwn. Rwy'n gweithio gyda'r tasg DT. Felly, mae'r tasg DT yn anodd tasg html bywyd. Mae'n ymgyrchu tablau ac mae ganddo ffeithiol yno y byddwch chi'n gallu defnyddio'r tasg yn y fframwaith a chynhyrchu pethau o hynny.
Felly, rydw i wedi gweithio ar hyn o'r fframwaith allan. Ac rwy'n gwybod ychydig am Javascript, ond ar hyn o bryd, roeddwn i'n gwybod llawer iawn am tasg html bywyd. Ac roeddwn i ddim yn cael profiad o ddefnyddio reaktiwnau cyffredin yn Javascript. Felly, mae'n ymddangos fel tasg html bywyd.
Setting learning objectives
Felly, dasg 1. Sefydliadau dysgu. Rwy'n hoffi feddwl am hyn o ran sut y byddwn i wedi'i wneud os nad oeddwn i'n cael help LLM gyda hyn. Felly, rwy'n meddwl y byddwn i wedi bod yn dweud, iawn, mae'n amlwg i ddefnyddio tasg allan, iawn, ac yn mynd allan gyda hynny.
Ond mewn gwirionedd, gann about my GPT, it asked me a number of questions. It asked me exactly what I was trying to achieve, how long I wanted to spend in it, and what past experience I had. And from my answers to the questions, it ended up coming with this really clear and concise goal. And it broke it down into three practical steps. So, capturing the event in Javascript, then passing that to Shiny, and then testing it in an app.
Now, could I have done this myself? Sure. But was I likely to? Yeah, probably not. So it just left me feeling like I really knew exactly what I was trying to achieve here.
So the second step, then, locating and accessing resources. So, again, like, doing this on my own, I'd probably end up Googling things and have, like, 50 browser tabs open and end up down a rabbit hole of kind of reading about a slightly different component. But with ChatGPT help, I had a couple of questions. So how do I prefer learning? I said I prefer learning by doing. I like having kind of snippets of code that I can run and see what happens and play with. And it found me really quickly just two resources that were perfect for the task at hand. So, firstly, the book Javascript for R, which has a free copy available online. And secondly, a blog post from 2016 by Dina Tali that's all about more or less exactly what I was trying to do.
So, great. And I guess this had stopped me kind of having to switch between kind of planning and organizing and coding, and I could just focus on reading up and writing some code.
So then, finally, the third step for this was this idea of adopting and executing learning activities. So there wasn't much back and forth between me and ChatGPT at this point, because we kind of talked a lot about kind of my past experience and things, but it came up with this three-step process I was going to follow. Firstly, I was going to have a look in the DT package and kind of try to reverse-engineer the code in there. So seeing if I could find the code that actually implemented that row selection to learn more about exactly how that worked. Secondly, build a minimal working demo, so in a tiny little shiny app, just showing capturing a user event. And then thirdly, add it to the actual package.
But it wasn't just like this three-step process that was useful. Around this, it did things like it gave me time boxes, so I didn't end up spending too long on each step. It also had useful little bits and pieces, so with the bit around reverse-engineering DT, it actually gave me a link to the exact place in the package that I needed to find the code to read, which was really helpful, not having to track that down myself. And it just feels like a lot of these things are all about reducing the level of frustration that we can end up having when we're trying to figure out a new thing.
But even better than that, it also suggested ways that I could use ChatGPT to help me with these. So for example, when I was reverse-engineering DT, it mentioned, okay, any code you don't understand, paste it in, and we'll go through step by step. And again, that was super useful. When I was building my minimal demo, the first time I ran it, it wasn't quite right, and just being able to paste it straight in and figure out what wasn't working, again, just reduced that frustration. And then when I added the event tracking to the app, we ended up in a situation where it wasn't working, and I was asking why, and the LLM actually asked me, well, what do you think you should need to click on in this situation? And it really felt like a really great just learning activity and being tutored on something.
And so I deem this definitely a success. There's a write-up on my blog as well with more detail if you're interested. And I feel like, for me, it was just a really enjoyable learning experience because I feel like I'd done the work myself. It wasn't the LLM doing the work for me. It was scaffolding my learning and pointing me in the right direction and figuring out what questions I needed to be asking and what the right answers were, but just not taking over. And I genuinely felt quite empowered as a learner.
It was scaffolding my learning and pointing me in the right direction and figuring out what questions I needed to be asking and what the right answers were, but just not taking over. And I genuinely felt quite empowered as a learner.
When is a human the better choice?
So equipped with these ideas of the kind of stretch zone projects and using the LLM as a tutor, I decided to take these ideas and apply them to a completely different project. And this brings me to my third and final question. When is a human the better choice?
So I'm one of the maintainers of the arrr package, and I decided I wanted to have a go at contributing to Pyarrow, the Python arrow implementation. I decided to work on a task related to magic methods. Now, if you don't know what they are, it doesn't matter too much for this, but it felt like the perfect stretch zone task in that I've got experience of the arrr package, I can write some Python, but I didn't know much about Pyarrow and very little about the internals of Python libraries in general. So it kind of felt in the right place.
So next, I needed to figure out what role I wanted the LLM to play. So I decided that I was going to write most of the code. In this case, you know, this task for me was about learning. So I needed to do that myself, really. I also wanted to write the test, just so I could be really clear about what the intentions of the code that I was writing. And I was going to get the LLM to help me with any code tweaks. Maybe there's bits of how things are done in Python that I don't really kind of get with package internals. So I was going to use it to help me understand errors and to explain any new concepts, like magic methods, which I had no idea what they were at the time.
But also, I had some other help from my friend, Elenka, who is a Pyarrow maintainer, and she was just going to jump in anywhere I got stuck.
Okay, so how did it go? Well, actually, I feel like this is almost like a kind of fight between LLMs and human knowledge, almost like a kind of retro video game with different levels.
So the first level, then, rubber duck. So I think probably a lot of people in this room have experience of that kind of rubber duck debugging thing, where, you know, you've got a problem with some code, and in explaining it to an inanimate object, you actually find yourself getting closer to the solution than you were in the first place, without even getting an answer back. And I found here, actually, when I was trying to create the prompt to kind of put into the LLM, half the time, trying to get something specific enough that it wasn't going to come back with nonsense actually helped me figure out a lot of the problem myself, which was great.
The second level, the LLM as an assistant. So there were points at which I was trying to build PyArrow, and it's got C++ under it, and I'm not super experienced in C++, and I ended up with various errors in the build logs, and instead of having to troll through and figure out what was the problem and what was happening, having an LLM just help me pinpoint it really quickly was fantastic.
So the third level, the LLM as a tutor. So as I mentioned, I had no idea what magic methods were before this, so I got it to come up with a very quick sort of mini-tutorial-type thing, give me a really quick, simple example of exactly what it was, and it just made me feel way more prepared for the task I was trying to complete.
But actually, there was a point at which LLM help didn't really take me any further. So there was a bit where there were two different conventions of how to approach magic methods in Python, and there was just conflicting information there. I could have asked an LLM, but it felt like actually, with that context and the context of PyArrow, this needed human intervention. So I spoke to Alenka, who helped me by just explaining things from her past experience and her knowledge of this, and yeah, I feel like having somebody acting as a mentor on a project is just so important in so many cases.
But actually, kind of like beyond this, I got the pull request done, I got it finished, but I wanted to figure out where would be another place I could contribute, what were other tickets I could take on that would be kind of fun, interesting, and a good learning experience. And I mentioned it to a few other people collaborating on the project, and they started actually tagging me on issues. And again, I think this thing of having somebody acting as a sponsor, helping kind of figure out opportunities to kind of push you further, again, so, so important to have that human kind of side of things there.
So actually, it's kind of not about sort of LLM versus human knowledge, it's more about just where to kind of rely on each.
Key takeaways
And so to kind of go back to the overall kind of question of how can I be more productive using LLMs, I think it's important to keep in mind that I feel like LLMs shine in the stretch zone, so like projects that can just take you a bit further than where you are at the moment. Not all the time, but sometimes, actually using LLMs to scaffold the work and not doing it for you can be really valuable. And finally, like not forgetting the importance of human connection, like really thinking about when to rely on people instead.
And I think by doing all of these things, we can kind of shift from passive use of tools to like really active learning, and that can help to build long-term skills and productivity. Thank you.
I feel like LLMs shine in the stretch zone, so like projects that can just take you a bit further than where you are at the moment. actually using LLMs to scaffold the work and not doing it for you can be really valuable.
Thank you so much, Nick. We did have one question. Did you make the art for your slides because they're so awesome? Yeah, I got obsessed with Canva the other day. Oh, no, the other ones. Do you know what? I saw Hadley's talk at USAR where he'd done a lot of that and was like, well, that's a good idea to pay homage to. So, yeah. Cool. Yeah, thank you so much. Thank you.