Why AI Adoption Fails Even When You Have the Right Tools with Rob Collie

Speaker 20:51

You you can't rely on the LLM to keep secrets. Yeah. Yeah. I mean, I helped build the first version of Power BI at Microsoft. Backgrounder also needs to be certified and also secured. Some backgrounders in particular could be very, very, very sensitive. Yeah. Like our strategic plan for competing with our chief competitor for the next six months can be a backgrounder, could be one of the most sensitive pieces of information at your company. Because once you give it to the LLM, you can't trust that it won't spill the beans.

Speaker 11:19

Yeah, sadly. That's like concept, awesome. For one task, great. For a thousand tasks, what's the hierarchy? What's the canonical structure?

Speaker 21:28

Like, like that's where it gets tricky. You know, there needs to be certification, but it's mostly a human thing. How do I do it? So, okay, there's two things. First of all.

Speaker 11:39

Hey everybody, and welcome to another episode of Using AI at Work. My name is Chris Daigle, and we're gonna have another exciting conversation about AI. You know, um, one of the things that uh I encounter a lot when talking to executives with our services side is that a lot of leaders believe that buying AI subscriptions gets their company into AI, right? That they're ready to go. We've got ChatGPT licenses, we've got cloud licenses, we've got uh uh co-pilot licenses, right? But actually, what we hear a lot, and I heard this on a conversation uh telephone call with a friend who's a capable marketer and that sort of thing just 20 minutes ago. You might hear your employees saying this. I can't trust AI. I have to rewrite everything that it gives me, or it'd be good, but it just doesn't know enough about our company or about our business. And today we're gonna talk about how we make sure that the tools that you're using have the necessary information about the company and the context and what everybody does and those sorts of things, so that people stop saying, eh, it output's okay. I'd just rather do it myself to, oh my gosh, I can't believe it just did that. And today our guest who's gonna be sharing uh insights on that based on his new book is Rob Collie. Rob's background is with Microsoft, and then he built uh about 13 years ago his own agency called P3, kind of primarily focused on Power BI and Microsoft analytics and the Microsoft ecosystem. But like any savvy uh entrepreneur and technician, he could see on the horizon that there's going to be some big changes coming with AI. And when it began to threaten his business, he set aside time for kind of his own AI vision quest to understand what AI meant for the future of his company, his clients, the services that they were offering. And he shared with me on our pre-interview that what he learned during that process moved him from scared to excited. It's led to a book that we're going to talk about, new AI services for his firm, and uh a pretty uh robust internal adoption at P3. So, Rob, before we get started, anything you think the audience needs to know about you and uh what you've been up to?

Speaker 23:42

I think you did a great job. I mean, uh I helped build the first version of Power BI at Microsoft. Um, and so yeah, that's where I grew up essentially, was it Microsoft? And um, yeah, I saw what Power BI was going to do to the BI industry, and I wanted to be part of that. And then so um, you know, P3 Adaptive, um, we originally were called Power Pivot Pro at the time because Power BI was called Power Pivot at the time. Um, we were the first uh consultancy dedicated to Power BI. Um, the first one that was and dedicated to delivering the results at the speed that Power BI could, which I knew that, again, from my interactions with the industry, I knew that most consultancies weren't going to be wired to actually deliver the promise of it. They were just gonna kind of withhold the the gift and milk it, you know. And I wanted to see how fast we could get. You know, could we could we get as fast and delightful as possible and and keep utilization high by being so delightful, by being so fast that we had more projects to work on. Heck yeah. Rather than turning every project into this, you know, gargantuan monolithic, you know, no one ever wins type of thing. And uh that's proven to be a heck of a journey. Uh we've you know proven to be, you know, very effective at that, very proud of what we've built. Um and like you said, like, well, here comes AI. Yeah. And um you know, both threat and opportunity. Yes. But threat tends to get you off the starting line a little a little better sometimes than opportunity. Fear is a powerful motivator. Um, and like you said, like I I had the the luxury of being able to set aside like a couple of months, several months really, to just go and tune out what was happening on LinkedIn, tune out all of the influencers, you know, and go and get really hands-on with it in a way and go on this journey and came out the other side going, oh wow, that's that's it. Like it's uh like like this sort of like shocking clarity that it wasn't as yeah, it wasn't as bad as as you expected, but it is still very different.

Speaker 16:01

So let me ask you, what was that? Uh, because there's a lot of people listening to this and they're kind of in the process of that that journey. What was your ideal outcome from this? Okay, hey, I'm gonna just do an immersion in this. Where did you start? What tools were you using at the time? I know things change every week here.

Speaker 26:16

Well, you know, like you were saying at the beginning in the intro, like we had already done that thing, which was uh the obvious thing, which is go buy subscriptions for everyone at the company. Like everyone at the company had an open AI chat GPT subscription. We had a chat GPT team subscription, you know. And that's that's everyone's obvious first move, you know. Like obviously I can't be critical of that because we did exactly that. Yeah.

Speaker 16:40

Yep, everybody does.

Speaker 26:41

Very sympath, very sympathetic to it. Uh, it's kind of like a phase you have to go through, you know. And um, you know, and you go and you look at the you know, we'd look at the adoption of it, and very few people were using it. Um, you know, remember, we're a technical firm. Like we're building yeah, data pipelines and and business intelligence solutions for our clients every day. And, you know, AI is really better at writing code than it is at even conversing it with you in English. Yeah. Like um uh code is easy mode.

Speaker 17:16

So let me ask me, I'm gonna dig in on this a little bit because I I think the process is always uh helpful for other companies who are winging it, kind of right. So you bought the licenses. Was the assumption that, hey, we're gonna give this to everybody and they'll know how to use it? Was there a training effort that went into this?

Speaker 27:32

I think it's not terribly different than what anyone else does, right? Like you you um you take that obvious first step, which is what we did, and then we were kind of like, I I think to be honest, we're just kind of hoping that the problem just kind of figured itself out. Yeah. You know, that people would just start using it. And like we have very smart people who work for us. Yeah. Um we have a very high hiring bar, and we don't have a lot of bureaucracy. So, you know, if it was going to sort of self-figure out, I think it would be more likely to happen at our company than it than most places. Yeah. But it didn't. You know, you know, and I there's this weird um and very hard to even recognize gap. This is one of the things that I think is makes AI so difficult, is that when you use it off the shelf in its chat GPT.com form or whatever, play that AI off-the-shelf chat form, and you use it for something general purpose where it doesn't need to know much about your company, it's amazing. It's magic. Uh it's it's just so devastatingly successful that uh you kind of start to believe, okay, AI is magic. We oversimplify and we we come to that conclusion. And then when you turn it towards a business use, you don't really even notice when you cross the point on your journey of where it started to suck. You know? Yeah. Like, and even if you're noticing that it's not performing the on these business tasks the way that it was performing on the general tasks, you you tend to blame yourself for it. I as someone who used to build software for a living, one of the things that I point out to people is like when when software throws an error, like an error message pops up, most human beings' reaction is, what did I do? What did I what did I do wrong?

Speaker 19:20

Yeah.

Speaker 29:21

Right. And I always say, at that moment, I say to them, no, a software engineer somewhere made a mistake. You didn't do it, right? Like we put you in a position to make that mistake, right? Like, um, it's not your fault, it's it's the software's fault. And so we have that, I think that same self-critical nature when it comes to something brand new like AI. I'm not good at it. And so yeah. And so you whether you're a leader or whether you're an individual contributor using these things, it's it it all feels the same. You don't really, not only do you not understand why it's it's it's failing you, you're not even necessarily super conscious of the fact that it is.

Speaker 19:59

You just notice something something's off. It's not as good as it you thought.

Speaker 210:03

It it yeah, like it feels like a slog now. Yeah. And it didn't, it doesn't feel like a slog on other things. Like when you use it like in your personal life.

Speaker 110:11

How did you turn the corner?

Speaker 210:13

So um I just wrote a is I was just writing last night a Substack post about this. Like the thing that really turned the corner for me was installing Claude Desktop. Mm-hmm. Okay. Um, about about a year ago, maybe probably a little bit more than a year ago now. And I didn't even understand why Claude Desktop was a thing. Like, why would you want a desktop version of the chat that's available on the web, right? Um and it's really telling that OpenAI had a desktop version of Chat GPT as well. I just never heard of it at the time. They had it even before Claude had theirs. But Claude's strategy was to bring with Claude Desktop was to, through you know, the use of MCP servers, through the use of extensions, which you know, just think of them as plugins, yeah, to bring AI into your work environment. Okay. And so the the actual experiment, the the project that I ran for myself, the thing I kind of and it wasn't even that hard, right? It was I didn't write any code for this, was I connected Claude Desktop to a Notion database in the cloud and started teaching Claude how to write marketing copy that was on brand for us. And and it used Notion to store rules and guidelines that it was coming up with so that when every new chat when if I came back and started a new chat, it could hydrate itself from those rules, become informed. And now suddenly I had something that you know is like actually passes the test of the word agent. Yeah. It's something that is customized for a purpose. And when I saw that it was access to information about your business, and not the ability to read it and also the ability to update it in some cases, that put me in a whole different mind space. And I that took me from zero to ninety in terms of understanding. And I um in my book, which we can talk about the book, I mean, I think it's just as interesting to talk about things in general, but like my my book is titled Fair Game. Uh Customizing AI to your business is easier than you think. It's a good title. And uh, I mean, it is if you realize that customizing is what you need to do. Yeah. Um, and I don't think people realize that. So in hindsight, I think I might even, I mean, I think it's accurate. But in terms of meeting people where they are, I think I would reconsider the that subtitle, actually. But um in the book, I call this the knowledge cliff. Okay. So you've got the stuff that's built into the LLM's brain. Like it wakes up knowing more about everything basically has a PhD in every topic that human beings have ever issued a PhD in, and it doesn't even need to search the internet to have that knowledge. Like I am not anywhere near worthy of having a Wikipedia article about me. Right? I mean, I am I am many tiers below being Wikipedia worthy. Yeah. And yet in the book, I test it, Opus, Claude Opus, even at that point in time, could tell me about my own writing style without searching the internet. That is bananas. Because it's because it was trained. I I was part of its pre-training. Got it. Like articles that I had written and people talking about my previous books about Power BI and things like that are baked into its brain. It knows that much. It it's someone relatively anonymous like me, it knows about. Yeah, yeah. Now, and then and then if it wants to know more, it has access to the web search and we don't even notice anymore when these LLMs search the web. Yeah. Right? Like it just takes a little longer for it to come back with its answer. We don't really distinguish between what's on board in its brain when it wakes up versus this next level down, which is go search the internet. Yeah. But then when it comes to your business, splat. Yeah. It's just, it knows nothing.

Speaker 114:28

It shows up size, location, anything it can pull from your website, but that's it.

Speaker 214:32

Yeah. That's it. It it shows up every day like a new hire.

Speaker 114:38

Mm-hmm. Mm-hmm. PhD and everything.

Speaker 214:42

So what you were describing in the intro, I think is have you heard the phrase bot setting? Yes. Okay. Explain. You're the first person. Okay. You're the first person I've talked to.

Speaker 114:53

I might have heard it from you actually in our pre-interview. Really?

Speaker 214:56

Yeah. Okay, maybe, maybe. But anyway, let's just let's give you credit one way or another. You're the first person I've talked to in the AI space. That's like in the practitioner space. I've heard it said from somebody else. Who's heard this phrase? Yeah. Okay. So even though I I've been interviewed by multiple media outlets about bot sitting, none of us had ever heard of it. I had to go look it up. Um, and bot sitting is what happens when you buy the subscription for everybody. And then you go and you look and you see that no one's using it. So then you encourage usage. Yeah. You you give them some some sort of rudimentary training, because you have to do that in order to then apply pressure. And then you sort of incentivize in some way or another or enforce that they use it. And this knowledge cliff what it of it just not being just doesn't know anything about your business means that you're turning your employees into new higher trainers.

Speaker 115:56

Uh-huh.

Speaker 215:57

Yeah. All the time.

Speaker 115:59

So we call that seasoning. And that it doesn't happen overnight. That takes six months of somebody actively using the tool before they would it would eventually be like, okay, I'm recognizing patterns. So big lift. Yeah.

Speaker 216:12

So can you can you explain more about that for me? I so I I've I haven't heard that phrase used that way way before.

Speaker 116:18

So in and it was just something that I came up with. I would notice that in my own personal use, the more I used a certain model, whether I spent a lot of time on ChatGPT or Claude or Jim and I or whatever, it would start to recognize, oh, it would put things in memory or it would, you know, self-store information about me to where finally the basics I didn't have to introduce. No, it didn't understand the nuance of the business. It didn't, and when I go on site with clients and we're talking about this and they say these things, I'm I tell them you got two options. The options we're going to discuss today on this podcast, so keep listening. Or you just hammer it out for you know the next three to six months, and eventually the model's like, I kind of understand who you are, right? That your specific account kind of understands who you are because it's it's stored memories and it's seen patterns and it can access other projects and chats for reference. But that's a pain in the ass. It's just it's it's the long way. There's a shorter way.

Speaker 217:59

Well, it's a long way, and I think it also is even when you it's that time in it's it's woefully incomplete. Agreed. Because because the very next day you're gonna be talking about something brand new that you haven't talked about 15 times before, and it doesn't know some nuance of your business process that's just never come up before, you know. And so, like this isn't a controversial take. It just it I I think it's just the truth. It's that it's ultimately the company's job if you want AI to work really well, if you want AI to deliver on its hype. Yes. The company's job is to build systems, and they're not that hard to build. They're not trivial, but they're not that hard either. You don't have to be the AI researchers to do it. You build systems that make sure that when the AI wakes up to be interacted with, that it it's either it's got a mixture of things handed to it immediately before the user does a thing. Yeah. It gets it gets like neo in the matrix, I know kung fu download of the particular business context that it needs to solve that particular problem to help you with that particular thing.

Speaker 119:18

I I want to just address this to the listeners. Like that's a very important point. If you are getting the licenses for your folks and you're kind of like leading the charge internally, it's your responsibility to give them this level of context so that you've got consistent output, right? Like everybody's drawing from the same um beliefs about the business. And I think that I've never heard it put that way. We we do it by default, but I've never really considered it's not the employee, it's not the user's uh responsibility because whatever they come up with, that's probably several degrees removed from what the the board or the the C-suites came up with at the company retreat, right? So good point.

Speaker 220:04

And there's there's all kinds of difficulty here. Um like I'm very sensitive or sympathetic to like leadership's um the pressure that's on leadership in this situation. Like um, you know, the the board is leaning on the CEO, they're leaning on the C-suite. What are we getting what are we doing about AI? We you need to be doing something about AI. Uh the board doesn't understand what to do, they just know that we need to do something.

Speaker 120:32

Yeah.

Speaker 220:32

You know? Uh and there's two problems for leadership. Number one is that like I describe it in the book, AI, as being a second kind of computing. Okay. We've all grown up with a with one flavor of computing, which is software, CPUs. And we all have an tremendous knowledge about what software can and can't do, an intrinsic, instinctive confidence in it. You talk to any CEO, and if you truly got everything that they understood about software onto the page, it would fill a 200-page book. There's a lot of trust there. You know, they they they might not be able to build software, but they know what software can and can't do. Yeah. You know, they've had plenty of exposure to it. They don't have that same experience with AI. And it's just a second kind of computing. It's it's a very different kind of computing. And so they need to have that level, that same level of of sort of fundamental understanding of it, and like sort of like ramping people up on that. It takes some time, and that's why I wrote the book. The book, you know, the book is meant to give you to give you that. But the other problem that they have is that um, you know, org charts exist for a reason. Like there's an org chart because the CEO can't see everything. No human being can see everything. You need to divide and conquer. And so much of what is relevant to AI is happening out on the edges of the organization. Right. And so if you uh AI, successful custom agentic AI has to be implemented close to the business. Yes. Down, kind of down in the details. But just like you were saying before, it's still the company's responsibility to do it rather than the individual's. Yeah. And this is just reality. It's not, it's not necessarily the way we'd want it to work, right? But this is the nature of the challenge. And so anyone right now who's under a lot of pressure to, you know, implement some sort of grand AI strategy at their company has has been handed a problem that is far more difficult and it it's it's it's far more difficult and it's formulated wrong even from the beginning. But the people who are applying the pressure don't know that. Yeah. And the people who are in the gun site sort of absorbing that pressure also can't don't have the language yet, the vocabulary to express that it's harder than that. Does that make sense?

Speaker 123:01

It does. So the the impact of having access to a chat GPT session or Claude session that has all of this context is a completely different experience than me saying, you know, tell me about you know uh account-based marketing strategies that are working in 2026, right? So now the question that begs to be asked, and and this is where we get into the sauce here how do I do it?

Speaker 223:28

So, okay, there's two things. First of all, the the AI companies and the big the big uh software companies are no dummies, they are building, increasingly building systems. That allow sort of like civilian level customization. Yeah. So like like Claude Cowork, for example, is another, I think, a huge uh huge stepping stone for people who are trying to, if you're trying to get your hands around this stuff. Yeah. Like I um you don't have to go to Cloud Code. Uh like the idea of having a folder on your computer, and it can even be like a OneDrive folder, so it syncs across, you know, and you have a folder per per task, a folder per work stream. Um just for a a silly personal example last night, I sat down with Cloud Cowork and started telling and and and gave it a folder. I created a SharePoint folder uh in my SharePoint, you know, subscription. And so that folder's on my computer. And I pointed Cowork at that folder and said, okay, here's your folder, and we're gonna start talking about TVs, uh TV shows and movies that we like. And I'm gonna tell you a bunch of things TV shows and movies that we like, and you're gonna write them down. And then you're gonna generate recommendations for us and write those down. And when you write them down, they're being stored in this SharePoint folder. Right? And so, and I'm also, as we go, I'm also telling it instructions on how I want it to interact with me in the future. And it's storing those instructions in this other file that it reads every time that tells it sort of how it's supposed to interact with me. Like I don't ever want to be handed back a list of 15 recommendations and just have the titles. Right.

Speaker 125:19

Yeah.

Speaker 225:19

I need I need two sentences next to each one that explains why I might be interested in it every single time. Like don't don't cut corners, you know. Um, so I'm tuning its behavior as I go, and it's storing, and we're not waiting for the memory feature to kick in. You know, the the long seasoning process you're talking about.

Speaker 125:40

We're forcing it.

Speaker 225:41

Yeah. We're forcing it, and we're like, I'm deliberately like curating it as we go. And I did the same thing um writing my book. Okay. So I I wanted lots and lots of real-time feedback and editing and proofreading of my book. And um, I've never had a human editor before. I've written books in the past. Yeah. Um, but it just it just the the workflow was too slow. Yeah. Yeah. Um, you know, and uh, but I created a cowork project that had all of my word docs in it that were for every chapter of the book I was writing. And I was writing all those by hand, but I would periodically circle back to cowork, which had access to this folder, and say, hey, take a look. I just wrote some, you know, a couple of paragraphs or half a chapter, or go take a look and then let's talk. And I slowly trained this project, which I who I who I then then nicknamed Eddie, Eddie the Editor, yeah, on how to be my editor, how to interact with me. Um like everything should come back to me in numbered lists, so I can respond in, you know, but by I can just reference a number and say, hey, I'm gonna ignore that. Yeah. Uh teaching him him, see how it goes? Right. Teaching him uh not to correct certain kinds of of informal grammar because that's just the way I write. Yeah. And ended up with like I think like a 16-page instruction manual that that Eddie wrote for himself based on our chat.

Speaker 127:22

Yeah.

Speaker 227:23

Um and Eddie the editor probably read my book 30 to 40 times cover to cover in the end. I did some statistics at the end, it was really fascinating. We Eddie and I to combined wrote about 10 times the length of the book to each other. The book was like 200 pages, 250 pages, but we produced 2500 pages of back and forth, and 70% of that was him, yeah, was Eddie, and 30% of it was me. And um you just can't get that uh out of a human editor. And I did all of this without writing any code. Claude Cowork is a great, like sort of like gateway drug into this experience. Um, but when you want to do something institutional that multiple people on your team are gonna be, you know, on at your company are gonna be using, I think more often than not, it comes down to you need custom software. Um and like so the the the chat box, if if there, if there's a chat involved, um that chat box lives in a piece of software that you own rather than living in chat GPT. Yeah, yeah, or living in Claude. And in that way, you can control behind the scenes what information is handed to the LLM when it first wakes up.

Speaker 128:49

Through a security purpose.

Speaker 228:51

For security purposes, but also like for um uh you're absolutely right. Like that the the software does control the security layer. Um, but it also uh is responsible for both preloading the right information into the agent, yeah. Um, and also then potentially also giving it access to additional information that it can fetch on demand when it needs it. So in the book, I call these handbooks and backgrounders. So a handbook is something that you inject into the agent when it wakes up, no questions asked. The handbook is like what you would give to a new hire, you know, that explains here's your role as a call center agent, or here's your role as Rob's editor, right? Um, and then backgrounders are things or pieces of knowledge that it might not always need. So you don't want to load them all. But when you're talking about writing marketing copy, it should probably go get your brand guidelines. You know. Yeah. Um, anyway, so you're about to ask a question before I kept talking. You know, when did you start writing this book?

Speaker 130:00

I started really, I think like basically the first of the year, like January. And they may have existed, you know, last year, but I certainly wasn't using them. I I've started leaning really heavily on in this concept. They've made it easy by creating skills and using connectors and things like that. So in the book, like were were those even out yet? I don't even remember. Were skills files and and connectors as covering as they are now?

Speaker 230:27

All of those things are out, but I I I kind of deliberately stayed at a level, what I call the Goldilocks level. Okay. Of like so you need to be you know, at that at that same level, if we if we if we looked at what uh an average CEO knows about software and could extract that into a book, right? Uh I wanted what I wanted was that level of explanation, but for AI. Okay. And so there's there's two as a as a book Right. So well as a book author, there's there's two real advantages to doing it this way. Number one, um the types of people that are more valuable, right, like leaders, uh are more likely to read and understand the book. Right. I didn't want to write a book for developers. Um and if you wanted to read a book, if you wanted a developer-focused book, you should have someone other than me write it. Like I'm not, I'm not, I'm not that interested in doing it. I wouldn't be that good at it. Um But the other reason is that it also leads to a book that's more durable. You know, in a time where this technology changes so fast. Yeah. Um, I've been very, you know, as soon as I sent the book off to the publisher, there's this breath holding, right? Like of, oh no, what's gonna happen next week that's gonna make my book look out of date? Yeah. And I it's been, you know, I that's been many months now that I've since I've handed it off. I handed it off in May. And so far, everything in it has really stood the test of time.

Speaker 132:01

But these are still the things on the tactical part of like a specific skill, but you're more focused on the strategic or the thematic concept of what is context? Why is it important? How like what is good context look like as compared to crap?

Speaker 232:17

Okay, that makes a lot of sense. Yeah, and and who and who at your company uh might be involved in helping solve these problems with you. Like in it it might be different people than what you've uh who've been involved in solving other problems. Like how to solve this close to the business problem while at the same time it being your responsibility. Yeah, yeah. Very tricky, you know? Yeah. And that's not gonna change, you know, unless they could unless AI turns into something completely different than what it's been, you know, uh, you know, over the last five years. Yeah.

Speaker 132:50

Um capturing that nuance of institutional knowledge within your organization, that's a big challenge for every business. So by capturing that, if somebody leaves or takes the cruise or goes, you know, uh goes on retirement or whatever, that institutional knowledge has already been captured. And it's not just in some Google Drive folder or some scribbled notes. It's actually in a place that's accessible to anybody who comes in the future. And how would Jim have done this? Oh, well, yeah, we actually captured that. This is exactly how he would have done it. Yeah, I like it.

Speaker 233:26

Yeah. In in BI, we had this phrase, and you know, I think we still do, but it just hasn't it's called one version of the truth. Um and you know, when I first heard it, I was like, what? Um but uh what it really meant was uh companies were playing a game of my spreadsheet can beat up your spreadsheet. Yeah. Like we we we meet when we have a meeting, two different departments come with their own spreadsheets that are purportedly measuring the same thing and they have different numbers.

Speaker 133:54

Yep.

Speaker 233:54

And and of course, you know, the numbers are more favorable to each department that wrote the spreadsheet, you know. And um so uh centralized BI meant that we weren't arguing over what gross profit's definition was anymore. Yeah. Right. Uh not we we we we weren't arguing over its definition, we weren't arguing over what data needed to be fed into it, right? It was just the same number for everyone. Yeah, okay. That's right. Or or at least maybe one department did drive it, but everyone eventually had to sign off. And now we could at least say we could stop arguing. We had we we sort of like certified it, you know. No matter how it happened, it got certified.

Speaker 134:40

And that concept lent itself to your development of this framework of context. And what do you call what do you call what are you calling the framework?

Speaker 234:49

Well, okay, so like I think there's there absolutely is a parallel one version of the truth knowledge problem. You know, there needs to be certification. Uh you know, and and and certification is a both a technical thing, but it's mostly a human thing, right? Like human beings have to figure it out and say, like, like this is the um so for example, in the book I talk about um, I use as an example. I say, I just go to off-the-shelf Claude and say, write me a proposal for this client of ours, and it spits out garbage.

Speaker 135:26

Yeah. You know, it it it's which is most people's experience prior to customization.

Speaker 235:31

Yeah. Um and then we build internally uh an agent that we, you know, we're we're very creative with our naming. We call it proppy, at least originally. It's been folded into something else. Proppy is now part of a bigger system. But anyway, so um and and proppy has a handbook that, you know, and and if you're if you were you're nerdy about this, you might call it the system prompt. Okay. But I think you know, I I I prefer more Fisher Price terminology, right? So it's you if you think of this, if you think of Proppy as an employee, as a as a digital employee, a digital teammate, digital colleague, whose job it is to help you write proposals more effectively, um you would you know you would give a handbook to a new hire. Um so and and guess what? Handbooks are written in English, right? Uh and so um but there there shouldn't be any debate in the end. There might be debate there should be debate as to what his handbook should be, but once the handbook has been finalized, it's the source of truth. That's the that's the source of truth for how Proppy should write proposals. Um and then again, if Proppy does need to fetch other information in certain situations, um like let's say when we're writing a proposal for a government client, let's say there's a different set of rules or uh uh additional set of rules, it might not make sense to preload all of those all the time. Is the background because by the Yeah, that's that's that's the background. So you so the backgrounder gets loaded on demand, the agent can go fetch it, just like it can run a web search, let's say. That backgrounder also needs to be certified. Yeah and and also secured. You know, some backgrounders in particular could be very, very, very sensitive. Yeah. Like our strategic plan for competing with our chief competitor for this, you know, for the next six months can be a backgrounder, but boy, that is like uh a well-constructed backgrounder could be one of the most sensitive pieces of information at your company. And your the your custom software shouldn't allow that backgrounder to be loaded unless the person sitting at the at the at the app is authorized to have it. Yeah you know. Um because once you give it to the LLM, you can't trust that it won't spill the beans. Yeah, sadly. Yep. It just it just can't get there. You you can't rely on the LLM to to to keep secrets.

unknown38:07

Yeah.

Speaker 238:07

Yeah. So I don't really have a name for that system, right? I mean, I I I think and there's a very important distinction there between like there are some workflows and some parts of your business that are sort of like in steady state. Yes. Like, you know, we were sort of like well-worn, we we we know how we operate, and those are the easier ones to deal with. But when you're doing kind of like the the frontier development of new plans, it's much harder because last week's slide deck that someone proposed has already been superseded by some new thinking. And the old slide deck doesn't have a a pointer in it now that says, Hey, I've been superseded.

Speaker 138:51

You know, yeah, like we actually built that into our process today that with with agents. Because this is the reason I'm particularly interested in this, is this is something that we're going through now. This idea, and I've mentioned it many times on the podcast, but the idea from Jack Dorsey originally is where I saw it of the company brain, the intelligence layer, the context layer, the single brain, whatever that thing is. And totally get how valuable everything that you've said is, because now, as long as that's shared globally and accessed globally, anybody can write the marketing copy, really, because they're not writing the copy, they're making a request to the model that is leaning on, in your case, the handbook plus whatever backgrounders or that that context document on or the skill or the rule, this is how we do the things. And that's like concept, awesome. For one task, great. For a thousand tasks, wow. What's the hierarchy? What's the canonical structure? Like, like that's where it gets tricky. And that's why I was asking, like, if in the book, if you had and and so I'm I'm seeing why the book is going to have longevity because it's not giving specific instructions like that, it's giving conceptually, like, you need to be thinking about this as you're creating an LLM environment that you can trust that everybody in the company, they're not getting ad hoc information. A big part of their output is the sources of truth, the handbooks plus the backgrounders that we've put together. And I think that for any company, I mean, obviously, like we're just we're just scratching the surface here on this. Um, but I think for any company, if what you take away from this episode is A, wow, I understand this concept now and I see the importance of it. But B, this is not something that we should just like, ooh, let me make one for this and this and this, that there needs to be buy-in, there needs to be conversations about. I mean, little things, if it's just your workflow or process, maybe you got your way of doing it. But if it's something that's going to step on multiple roles or across departments and that sort of thing, it can't just be something where, hey guys, I built that file, let's move on. Well, who the hell are you?

unknown40:58

Right?

Speaker 140:58

Where what let me see it before we commit that to the global structure? So um the the idea behind the book, and I would encourage people, this is you know, uh the only book that I've seen out here on this concept. And as I mentioned just today, we're working on this. Uh, the book is called Fair Game: Customizing AI to your business is easier than you think. And I think that, you know, based on the conversation we've had today, you can trust that Rob has put uh a lot of uh elbow grease and and mental um uh cycles for sure into this concept. So I would encourage you to take a look at the book. Rob, you mentioned that you have a Substack. Um, where else can they, and and what what is that? And we'll put it in the show notes, but where else are you kind of sharing these discoveries from the the frontier of context engineering?

Speaker 241:48

So uh my substack is Rob Collie.substack.com. Um just launched it. Uh it's really only like five articles in, so it's it's pretty new. Um, but it's kind of meant to be the continuation of the book in a way. Nice. Um like a newsletter that kind of follows on after the book. Um I've been doing a podcast called Raw Data with Rob Collie. We are about to move that to YouTube because uh increasingly there are things I want to show. Um and it's it's more instead of being sort of like a a two-person co-hosted chat format, we're gonna be moving to more of an explainer where I can report from the frontier and say, so this is these are the things we're saying, and I can demonstrate things like that. So that's that's coming soon. And of course, there's our website, p3adaptive.com. Um, you know, we're we're builders um of BI solutions, but we are increasingly seeing that the one of the highest value places to start with AI is by AI fying or AI empowering your BI uh at your company. There's just a you can pick up like 3x ROI on all of your BI efforts if you bring AI into the story. And it's been really, really exciting. Nice. Um so uh yeah, we've there's a the there's a very exciting future here.

Speaker 143:04

Very cool. Yeah, so um for the listeners, again, just the main takeaway really should be this idea of this um codified, documented, agreed upon, certified sources of truth that the LLM can lean on. And I know Rob's suggesting custom software, but the easiest place to start for anybody might be just getting those documents, whether you use Claude Co-work or you use ChatGPT work or Codecs or Claude Code or whatever your people are using, knowing that there is one environment where this is our company, this is our you know, unit economics, this is our you know, industry competitors, this is our brand guidelines, this is our core value. All the stuff that we all do in all these plannings and those sorts of things, if they sit in a drive and nobody has been instructed or you haven't tuned your models to lean on those drives or folders, great that you have it, but you're not getting the benefit of it outside of the water cooler, chatter, or you know, the company retreat. The LLM on the day-to-day use just will continue to be like, meh, it's okay until you do these steps. So, Rob, thank you for taking the time to come on and share this concept because as I mentioned earlier, like this is something I'm working on today and have been working on pretty heavily since April. So it's um, in my opinion, the thing that will distinguish companies that adopt this, like their people, the team level adopts it because it's actually helpful as compared to, yeah, we got a license, yeah, we got about 2% of the people that are power users, 5% are using it, and the rest are using it like Google. So Yeah, that's right.

Speaker 244:45

That's exactly I think that's where we're at. Yeah. We're in that bot sitting phase as a world right now, and that's okay. It's a phase we have to go through. You know, it's like the awkward, the awkward teenage years. Like that's where we're at.

Speaker 144:58

Yeah. So everybody, thank you for for listening in on another episode of Using AI at work. Um, if this episode, you know, something landed in this and there's other people on your team, because I'm I know that a lot of people like you're you might be one person that's contributing to the conversation internally, but that there is a collective conversation happening in every company asking the question, what are we going to do about AI? If you found this episode or any of our episodes like a resource that you think would help the rest of your team upskill or kind of go, oh, aha, or I understand it better, please forward the episode. Uh and if you have the time and you enjoyed this episode or any of our episodes, we'd love it if you would leave um a review on whatever podcast platform that you listen to. Even if it's just to push the stars. Ideally, it's the five stars. Um but if you want to leave a comment, we'd love that as well. So thank you everybody for listening to the show. Uh, we'll be back next week with another fascinating episode of Using AI at Work. Thanks for tuning in to Using AI at Work. Don't forget to subscribe for more conversations about how to use AI at work. And a special thank you to our sponsor, Chief AI Officer, for empowering businesses with AI education and training. Visit their website for free AI readiness assessment and AI strategy guide to help you get started using AI at work. That's www.chiefaiofficer.com. Follow us on Twitter at the handle usingAI at work and visit www.usingai at work.com for free resources to help you harness AI in your role.