Lodemark  |  Chad D. Russell

The Trust Series: Building with AI

Part 1

Why I started paying attention again

The ground is shifting under our feet again. Every big change in technology works this way, and the people that tend to feel it most are the ones accidentally standing in the wrong place. I was in telecom when dedicated trunks gave way to IP switching. I was at Salesforce when software stopped being something you installed. Some of you were there with me, everyone has been impacted by it.

I was a functional solutions guy. Before the “architect” title came into play. When the work was about figuring out what the system ought to do and how people interact with it, and then handing the building to people better at it (much better at). I am not an engineer, a coder, a person that can follow A-Z step-by-step without veering and I have never claimed to be.

Then I spent eighteen years in mineral management, running Lodemark, in a corner of the world where the software, quite frankly, is not very good. I built a management platform for the firm because what I could buy did not do the job, and I have been tinkering with it for years, because the design part, where you solve problems, is the part I like most.

Over those years, I dug into most of the technologies that came and went. Some were big, like blockchain, but I never saw how they changed anyone’s day-to-day work. Others were flashes in the pan. None of them really grabbed hold. I was raising kids, coaching soccer, running a business, and everything else that goes along with trying to be present in your own life. But the designer in me kept an ear up, kept tinkering.

I’d had a vision for something once, but never the means or talent to build it. Then, like everyone else when the first consumer versions of AI came out, I played around: a soccer team logo, a recipe, a business problem. Clearly, something with this new wave was different, but I couldn’t put my finger on what I could do about it. So, I added technology conferences back to the calendar with the industry ones. Then a conversation with a good friend, and consultant worth your time, Rich Nazzaro, sparked my curiosity, my need to mess with things again. We talked about what was coming, and the parallel to the moment SaaS took hold.

It really wasn’t the technology that got my attention about language models or “AI”. It was that some of the same questions that followed SaaS or other technological shifts around for years were rearing their heads again, just with different syntax. The biggest one, as always:

How do you know you can trust what the thing tells you or what it’s doing?

That question is not academic where I sit, and not unique to my position. Our job is to hand a client an answer about their own property, their own resources, assets, and money, and provide them with the best options so they can act on it.

Since January, I have been diving into it twelve and fourteen hours a day. Not to show how hard I work, but because I am self-taught (always have been) and the only method I know is to find what breaks and what I can fix. Or in this case, find out what breaks, what works, or seems to work, only to find out it’s fool’s gold and fix it again.

The models got better as I was working with them and we all know they are moving faster than we can keep up. Some of what I ran into has since been closed by the models themselves. But, I know why those problems existed, not just what the workaround was.

Plenty of what I hit is obvious to somebody more technical, and that is fine, that’s not what this is about.

Next week we’ll look at what I really f’d up. Wonder if any of you share the experience, or what was just my own dumb mistake thinking I’m smarter than I am.

Part 2

Fifty-seven measuring cups.

There are fifty-seven measuring cups in my kitchen.

I like to cook, mess around in the kitchen really, see what happens when you change things a bit. Not bake, that takes too much precision. It started back in the olden days, as my daughter calls them, of the ’90s. I was going to school in Denmark and had no easy way to eat besides to cook for myself.

I have to be in a certain mindset to follow a recipe step-by-step since it slows me down and takes some of the fun out of it. Now, I don’t mind doing it the first run on a recipe someone else made, but after that I change it to my style and my palate, or I should say my family’s palate. An unintended consequence of this hobby is that it has turned my son into a food snob, and my biggest critic.

It’s also why when my wife follows a recipe I use, it doesn’t come out the same as when I cook it. It’s not the grandmother thing, leaving something out on purpose so yours is never as good. I tasted something new somewhere, wanted to try an alteration, and forgot to write it down.

Fifty-seven measuring cups isn’t literal, but it is where the real problem was noticed. I am going to explain how that happened, because it’s where I really f’d up and wasted my most precious commodity: time.

I have a big binder of recipes I keep covered in chicken-scratch. Picture one of these recipes with the handwriting, slightly legible, all in the same color. It says use the blue measuring cup. Somebody replaced the blue cup two years ago with a new one, same color and a different size. The recipe is not wrong-looking. It is just wrong, and you will never catch it trying to follow the recipe, only when you sit down to taste it.

At Lodemark we run a monthly process, a monthly process that was perfect for automation. We had all the components, and we knew how they all fit together. So, we built it and we ran through it four or five times (at least), documenting as we went: RAGs, swim lanes, bullet points, you name it. And every time it felt like starting from scratch.

The written procedure named the program to use, the steps to follow, the decisions to make, normal process stuff. But when we analyzed it from end to end, it turned out to be the previous month’s work, hard wired to two finished files and date-stamped for that month. It never touched what the document said it processed.

So, we worked backwards from a leftover output file to find what happened, so I looked at what was in the drawer. Fifty-seven versions of the programs that one process runs on, and thirteen versions of the procedure telling you which of the fifty-seven to use. Every program versioned, every procedure time stamped so we thought we were running the newest and updating the newest each time. All of these programs at one time were the working program. Any one of them is sitting right there, it opens, it runs, and it is the wrong one, and nothing about it looks like an error until we’re two hours down in the process trying to figure out what the heck is happening. What was broken was one sentence in a document: use the blue measuring cup.

Confused? You should be. The best way I can picture what we found is one of those movie scenes where someone is tracking down ‘who done it’ and their evidence board has turned into a Crazy Board, red string running all over one wall then spilling onto the next ones. Outputs giving us answers that looked right on sight, but something smelled and tasted off.

I don’t believe the number is the problem. You get that number when every individual fix is correct and nothing ever retires, when you keep your old pots and pans or measuring cups “just in case” one of them was the trick that made everything work.

The most frustrating thing is we had the check that would have caught this, we had approached it six months earlier, and set it aside. But now we have four blue cups in the same drawer… son of a…