EP 16 - Dr. Kristen DiCerbo - Khan Academy’s Chief Learning Officer and the creation of Khanmigo

In this episode, John and Jason talk with Dr. Kristen DiCerbo about how Khamigo was born, how it works, and how it might help transform and humanize online learning. 

Join Our LinkedIn Group - Online Learning Podcast

Resources:

• Kristen DiCerbo
• Check out the Khanmigo page
• Khan Academy / Canvas Instructure Press Release where they announce their partnership
  • Video from Instructurecon
• More information about the ICAP Framework

Transcript

We use a combination of computer-generated transcriptions and human editing. Please check with the recorded file before quoting anything. Please check with us if you have any questions!




Pre-Banter 


[00:00:00] Jason Johnston: Was Khanmigo able to join us on the call today? 


[00:00:04] Kristen DiCerbo: Khanmigo right now does not have text to speech, so we'll not be on the podcast. 


[00:00:09] Jason Johnston: All right, probably too busy. There's a lot of people to support out there. And they're a bit of a rock star. So they're probably at the White House or something  


[00:00:17] Kristen DiCerbo: Most likely, yeah exactly. Some world leaders. Yes. 


[00:00:20] Jason Johnston: Leaders guiding policy across G20 or something like that. 


[00:00:25] Kristen DiCerbo: yeah, exactly. 


Start 


[00:00:26] John Nash: I'm John Nash here with Jason Johnston. 


[00:00:29] Jason Johnston: Hey John. Hey everyone. And this is Online Learning in the Second Half, the Online Learning Podcast. 

[00:00:34] John Nash: Yeah. We're doing this podcast to let you in on a conversation we've been having for the last two and a half years about online education. Look, online learning has had its chance to be great and some of it is, but a lot of it still isn't. And so how are we going to get to the next stage?  


[00:00:50] Jason Johnston: That's great question. How about we do a podcast and talk about it? 


[00:00:54] John Nash: Perfect, what do you want to talk about today? 


[00:00:58] Jason Johnston: Well, I'm very excited about our guest today. We're going to be talking with Dr. Kristin DeCerbo, the Chief Learning Officer at Khan Academy. Welcome Dr. How are you? 


[00:01:11] Kristen DiCerbo: Good, good. And please call me Kristen. It's great to be here today. 


[00:01:14] Jason Johnston: Well, it is great to have you here. And we just wanted to start off by just getting to know you just a little bit. Tell us a little bit about your current role at Khan Academy. 


[00:01:25] Kristen DiCerbo: Yeah, so a chief learning officer can mean lots of different things and lots of different organizations. So, At Khan Academy, I lead our content team, our product management team, our design team, and our community support team. So most of what you see, on the Khan Academy site is built and created by a lot of the folks that are on my team. 


And I do not lead the engineers, they're a whole 


[00:01:51] Jason Johnston: Okay. That's good. 


[00:01:52] Kristen DiCerbo: group, not them. But I come from an educational psychology background. So my PhD is in educational psychology, and so I don't have the traditional kind of product background that some educational technology folks who lead those kinds of teams do. 


And instead, I bring a lot of, you know, experience and insight about how people learn. And we try to build that into then the articles, the exercises, the videos, and all of the experiences students have on the site.  


[00:02:24] Jason Johnston: And so, you mentioned about your education, you've got a PhD in educational psychology. Is that what you said? 


[00:02:30] Kristen DiCerbo: Yes. I actually thought when I went to grad school that I was going to be a school psychologist. And so did a research practitioner program where I was doing. All of the work and training to be a school psychologist and doing all of the work on how you diagnose learning difficulties and all of that and did a whole bunch of research and kind of fell in love with that side of things too. 


But then did end up being a school psychologist in a school in Arizona where I live. One of the schools I worked at had 1, 200 kindergarten through third graders, if you can imagine in a school, 


[00:03:05] John Nash: No, I can't imagine. 


[00:03:07] Kristen DiCerbo: And 80 percent of them had a home language that was not English which makes them figuring out why are kids struggling to learn quite a challenge , but also really drove home for me some of the Potential for education technology and could see, you know, there's, we have all of these kids who need extra support and extra help and all different levels in a classroom and how does a teacher help start to differentiate all of those. 


So that was a bit of my first thinking about how do we, how could we use technology to help support these kids? 


[00:03:44] Jason Johnston: So you got a lineup of kids waiting to see you at the school. And then did you just think one day, man, I'd like to learn more about doing this through a computer or through online or how did you get into Khan Academy? 


[00:03:55] Kristen DiCerbo: Yeah, so it's a little bit of a winding road. So at first I was, after a couple of years, was thinking I was going to make a career change, thought I might go be a professor and did academic job applications. In that process, I got back in touch with someone who had been one of my statistics professors at Arizona State and he had gone to work for Cisco, the networking company, which sounds crazy, but Cisco has something they call the Networking Academy Program, where they create curriculum and assessments that they give away to high schools and community colleges for free to help students learn computer networking skills. 


And so I got in touch with him to write me letters of recommendation for these academic jobs, and he said, of course, I'll write you those letters, but you should think about coming to work here. We're doing this online learning thing. And we have a whole lot of engineers that are starting to, you know, help us write this curriculum. 


But we need some people who know about learning and assessment to think about this. And I was like, huh, that's kind of interesting. And the kind of the kicker is, and you could create these things that touch hundreds of thousands of learners every year, and really have that kind of impact on the world. 


And so I was convinced and. So went to Cisco and spent a good amount of time doing some really interesting things around simulations and simulation based assessment, because it turns out that the high school students would practice on this expensive networking equipment, and they'd break it, and they didn't know how to fix it, so we said, well, let's create some simulation tools so they can configure this and try those crazy what if questions in a low risk environment and work through those. 


And then we said, wait. We are, by doing this, we're capturing all of the information about how they're configuring these devices. Why are we giving them a multiple choice test about networking when we can actually see them doing the skills that we're trying to create? And so, got into performance based assessments and thinking about how we can use that digital data to understand what students know and can do. And from there, I was presenting on some of these ideas and some folks from Pearson approached both me and my mentor, who was the one who brought me and said, Hey, we're starting something called a Research and Innovation Network at Pearson. 


And would you like to take some of these ideas you've been playing with and try them out in domains other than computer networking? And, you know, think about how this might fit across K 12. And that was pretty enticing. And obviously Pearson also has big reach and big opportunities. So went there and spent a good amount of time there in a research role. 


And then gradually over time, as big companies do reorganizations and changes ended up leading a team of learning designers and researchers who were working closely with product teams to. building based on what we know from the learning sciences into some of the products and places things that offerings that were being released at Pearson. 


And then about, oh, three and a half years ago or so, I was thinking about what my next step might be. And I was talking to some friends and said, I think I want something smaller. I think I want to go in the nonprofit space. And I think, you know, I was a VP at Pearson. I think I'm going to go like something like a chief academic officer, chief learning officer. 


And three weeks later, this job that I have now was posted. And I said, this is what I was just describing and it all worked out. And so that's how I got to Khan Academy. 


[00:07:36] John Nash: Did it feel, going back to your experience developing performance based assessments, and that was in an online environment, did that feel different and kind of exciting? Because that seems like that would have been a little cutting edge at the time. 


[00:07:51] Kristen DiCerbo: it was. So yeah, we're talking like 2006, 2007 at this time to date myself. And it was, and it still feels like I still look back on some of that stuff we were doing and was like, this is still kind of cutting edge. And in terms of what's going on in the world. So it did feel that way at the time. And I still, You know, I'm still hoping there's a place for some of that. 


And then at Pearson, I also was doing part of this group called Glass Lab that was a collaboration with EA, the makers of SimCity, the big gaming group. And EA gave us the code to SimCity and said, make a game based assessment. in SimCity, which was super fun. 


[00:08:37] John Nash: I bet. 


[00:08:38] Kristen DiCerbo: And we learned a lot and made, so we made SimCity EDU, which if you Google it, is no longer in existence and has some issues and never really got to large distribution, but was a lot of fun to make and a lot of fun to think about the issues around that. 


[00:08:53] John Nash: Yeah, that's cool. So you sought something that was maybe non profit. Okay, check. Maybe smaller, maybe not check. 


[00:09:02] Kristen DiCerbo: Well, so Pearson has like 30, 000 employees and Khan Academy has 250. 


[00:09:07] John Nash: fair, very fair. But certainly a footprint of goodwill in the place. What's it like working at Khan Academy? 


[00:09:14] Kristen DiCerbo: it, I have never been someplace where everyone is there. For the mission, a free world class education for anyone, anywhere, like that, people, lots of people who work there can be doing other things to, you know, make a lot more money to be in those space, but everyone is there thinking about how can we help. 


Kids learn more. And what does that look like? That's exciting and motivating to come in every day, keep things going. And now that we've, I don't want to foreshadow the rest of our conversation, but now that we've kind of jumped in with both feet into this AI space, it also is kind of exciting because it feels like we're a bit on the cutting edge of in the technology space, which is a fun place to be too. 


[00:09:57] John Nash: it is. And no, you foreshadow away. I'm dying to dive into that part. Yeah. Yeah. 


[00:10:02] Jason Johnston: Yeah, and I was thinking about that this morning in anticipation of our conversation, just how impactful Khan Academy was from the standpoint of the flipped classroom. And obviously Sal Khan is a huge personality and just incredible story. But, even as I was a high school teacher and technologist for a while and using Khan Academy and all those kind of things. And the reorientation of what a classroom could be for my students was in part because of Khan Academy. So it actually, it really surprises me that there's 250 people there because there are many ways and no disrespect to Pearson because they're huge in its own way, but Khan Academy has had a lot more impact on my life and my teaching life and my kids lives and my students lives than Pearson has so far. 


[00:11:03] Kristen DiCerbo: I think the phrase is, we punch above our weight. 


[00:11:05] Jason Johnston: Yes, like that. 


[00:11:07] Kristen DiCerbo: As you know, as a nonprofit, we get a lot of funding from philanthropists. And one of the things that our head of philanthropy says a lot is that for the budget of a large high school, we reach tens of millions of students every month. 


[00:11:18] John Nash: That's a fantastic tagline.  


So, let's talk a little bit about Khanmigo. Can you briefly explain what Khanmigo is? 


[00:11:27] Kristen DiCerbo: Yes, so Khanmigo is an artificial intelligence tutor for students and assistant for teachers. So, for those who haven't been part of the big AI conversation for the last year, last November an organization called OpenAI released ChatGPT, and that is what, in the artificial intelligence world, is a generative AI model, or a large language model. 


So, it, goes and trains on a whole, Enormous set. Basically, if you can imagine all the information on the internet, like that's a good proxy for the kind of what they either trained on and what they do is, as most folks know, they've played around with chat GPT is you can have a dialogue with it and it's producing text. 


You input text and it produces text back to the user. So that was last November. If we go back from there, last September, so just about a year ago from when we're recording this, Sal and I got to see what at the time was the model they were training that they called G P T four. What they released in chat G P T at the time was G P T 3.5. 


And so they were training their next version of it. Bill Gates had said to them, Don't come back to me and show this until it can pass the AP bio exam. So they needed a bunch of AP bio questions to keep testing it on. And we at Khan Academy have a whole bunch of AP bio questions. So, that was the impetus for them reaching out. 


But as we started talking and Sal, they gave Sal and I access to a Slack bot so we could talk to this new model over Slack. And we were blown away by what it could do. And said, okay, let's talk more broadly. Yes. We'll give you those AP bio questions, but let's talk about how we might be able to use this as a tutor and think about what it could be. 


[00:13:21] John Nash: Hey, can I be nerdy for one second? Cause that, I kind of got goosebumps with that. So in last September, you're having Slack conversations with GPT 4. 0 and fellow nerd, Jason Johnston and I are wigging out about 3. 5 and you're already, you're just going like, you guys wait. Is that kind of how it was for  


[00:13:42] Kristen DiCerbo: Yes. We had this super strict NDA, so we couldn't talk about it at all. So then chat GPT comes out and everyone's, Oh my God. And everyone wants us to comment on what we think it's implications for education are, and we can't say like, just wait till March. 


[00:13:58] John Nash: I know, you're like, "it's okay." 


[00:14:02] Kristen DiCerbo: "Seems interesting." 


So yeah, it was definitely a challenge to keep our mouths shut and not talk too much about what we knew was coming. 


[00:14:11] John Nash: That's something. 


[00:14:12] Jason Johnston: So how quickly did you move into kind of more formalized talks? Because I'm guessing, I don't actually know this for a fact. So I hadn't looked this up, but I know that Khanmigo is different than chat gbt, but I'm guessing that there is some basing upon that language model. 


[00:14:33] Kristen DiCerbo: Yeah, so, so I'll tell you kind of how Khanmigo works a little bit under the hood. So it is based on GPT 4. So what happens is when a student is on Khan Academy and working, say, on a math problem, they have access to Khanmigo. Icon, you go icon you know, can pop up and then the student can talk to it. 


And what happens is they might type in, say, "I'm stuck." And what happens is then the what we send, we take what the student sent. And then we add in the problem that they're working on. And then what's called a prompt. So about, in our case, it can be three to 500 words that tell the model how to act. So in our case, we'll say something like you are an uh, an empathetic Socratic tutor, you will do these things, you will not do these things, like you will not give the student the answer you will do these things, and we know from So you will the literature on what makes a good human tutor, the kinds of things that good human tutors do, so we prompt it to do those things. 


So all of that gets sent to GPT 4, and then, and that is the input to the model, and then the model sends back, "Okay, what do you think the first step in this problem might be?" Or some similar response to what's going on. So we are absolutely linked to GPT 4, but, That prompt that we have that tells it how to act like a tutor or the other activities that we've created based on how students learn. 


One way that we put a lot of guardrails in that if you just put that same problem into chat GPT, it'll just give you the answer. Like, it'll work through it and say, here's your answer. Whereas ours is going to act like a tutor and say, well, how do you get the variables on both sides of the equation or whatever that, you know, what that might be. 


We also have some things, for example, we store chat transcripts, which for students, if they had a good conversation yesterday, they can go back to it and review it. But they're also visible to the teacher or parent if they're under 18. And so there's another place where if teachers or parents are concerned about what they're saying to GPT, there's all that record there. 


And we have set everything that the student sends, we actually send through also a moderation API, so it goes into another... AI that's checking for instances of things like violence, hate, self harm, and sends up a flag. And for students, if something gets flagged, an email gets sent to their parent or teacher, and it's indicated in that transcript that there's a flag there. 


So there's some more safety in terms of what the conversations are that we're doing. So in terms of the, you know, how did that all come about? After Sal and I had seen this model, We coincidentally had a at Khan Academy, we have a tradition of hackathons. And they aren't just engineers. 


The whole company for a week takes on problems that aren't part of our everyday roadmaps. So, any team can work on something for a week and do something. But we pulled in about 25 people, convinced OpenAI to open our NDA to 25. People. And so we had a mix of our content creators, our engineers, our designers, our product folks, and Just spend a week like what could this be? 


How does that look? And so this is still like the last week in September of 2022 and those weeks can be pretty like you get a lot of stuff There's a thousand flowers that can bloom and try a bunch of things And then from there started, you know, okay, this will work. This doesn't work. 


Let's see if we can get this to work and Then we basically threw out our product roadmap for the rest of the year and said, all right, we're going to do this. And let's see what we can get done by March. 


[00:18:30] Jason Johnston: Amazing. There's actually so much there we could talk about. Thank you for that. I really appreciated your clear description of what it was as well for those that are listening that haven't necessarily seen it as well as kind of letting us see under the hood just a little bit in terms of the guardrails and the safety that's going on. 


So there's a lot there we could talk about. I'm curious to jump off of one of the things that you said. So the, okay. teachers are able to see these chats going on, which is a very unique feature. I hadn't really thought about that. Versus a teacher maybe sending them out to ChatGBT or to another chatbot, or even some, you can make some chatbots, but without potentially the way to see what's going on. 


And I also saw within the Khan Academy, because we've been checking out Khanmigo from a teacher side, as well as a student side, I really thought this class snapshot was interesting. Do these chats get put into that class snapshot? Cause I haven't been able to use this in a real world situation yet. 


[00:19:35] Kristen DiCerbo: Yeah. So right now they we have chat summaries, which is actually a separate piece. Sometimes we're moving so quickly sometimes I forget what's still in beta and what's actually launched, but so the chat summaries allow you then instead of having to read the whole transcript, it'll provide summaries of what were the big questions that students were asking and what did those look like. And then the class snapshot right now is Okay. pulling on the data from students interactions with the exercises and articles and videos and gives summaries of how much time have students spent. Our, one of our big metrics is skills to proficient, which in Khan Academy, we have a mastery learning framework. 


So the goal isn't to get a high score, it's to master skills. And so in our system, you go from familiar to proficient to mastered. And our research shows that if you get to proficient, we see the skills that students. Students who get to more skills to proficient show greater growth than expected on a lot of norm reference tests. 


And so, so we want to encourage students to get to proficient, encourage teachers to work to get those skills to proficient. It's better, in fact, better to get even fewer skills to proficient than more skills, just a familiar so that we're reporting to teachers in that in that class snapshot for how many students are getting to proficient with that looks like. 


But then we also want to give teachers things like, hey, what students, might be struggling on this particular skill, what students might get a "congratulations, good job" because they're moving quickly and what that looks like. So that we're pulling together that data. It's still, we're still, you know, nothing's ever as good as there's always opportunities for improvement. 


So when I look at it, I still think there's opportunities to even give teachers more help about and you could do this with them and you could, make suggestions for things for teachers, but at least we're summarizing the data for them and it can, and then the teacher can have a conversation with Khanmigo and ask some, detailed questions about what's going on in the data. 


[00:21:44] John Nash: and I forgot all of the hats you wear on the CLO side, but you've got everybody but engineering, I guess it sounds like. And so, if Khanmigo is what one might call a GPT 4 powered solution what kind of pain points did you discover amongst teachers that Khanmigo addresses that say Core GPT does not? I mean, we've talked about some of these features, but as you, as we think about the jobs to be done that teachers are trying to do, what did you discover out there? 


[00:22:13] Kristen DiCerbo: Yeah, so the first thing I come from the, you know, the learning background is what are the learning problems that teachers struggle with? So if I'm in a K 12 space, I'm in high school, I've got 150 students across my five classes, you know, that I'm trying to work at, and really difficult to give that individual attention to students. 


So I was in a classroom actually here in Phoenix that was doing some testing with Khanmigo and they're actually in our, Howard College Algebra work. So we have a partnership with Howard University and National Ed Equity Lab where students work through and get mastery on a college algebra course on Khan Academy, take the Howard midterm and the Howard final, and then get , three college credits from Howard for doing that work, which is a whole other podcast we could talk about. 


[00:23:04] John Nash: That's cool. 


[00:23:07] Kristen DiCerbo: But so there's one teacher and there's all these students and they're doing a lot of independent practice on Khan Academy. So what happens, of course, is they get stuck on a problem. And so they raise their hand and they're sitting there waiting, and Ms. Alvarez is doing as much as she can going from student to student, but you know, wants to give each student that she's with enough time. 


And so what happens is the students were able to Ask the question to Conigo. Wait I'm not getting to the right answer here. I don't understand this. How do I subtract, you know, two from both sides? All of those kinds of things. And so it's that piece when students are working, we go way back to, you know, our friends and talking about zone of proximal development. 


That, that place where you're most learning is a place in between where you can do things independently and where you need a lot of support. You like, you're really learning if you're working in a place where you. Just need a little bit of support to get over. And Khanmigo can really be that individual support right at that place where students just need a little nudge to get on to where they're, where they need to go. 


So that's a big use case for us in thinking about that. There's also some other things, just we know teachers are always asking me about how do I motivate my students, how do I keep them going, and so there's some things like we have an activity that's called Chat with a Literary Figure, and so you can of course go and read a Wikipedia page about Jay Gatsby, but if you can have a conversation with Jay Gatsby, That's, more engaging and more interesting and keeps some of that motivation and interest there where students are. 


So we're trying, a lot of those things, like chat, do that. We have a debate activity where students can debate with Khanmigo on a bunch of different topics that we have set. And all of those things are things we're trying out, seeing how they work, seeing how students react to them, how teachers react to them, and what that looks like. 


But so, there's certain, there's the individual, individualized tutoring piece, and then there's the engagement piece, I think, are two big problems that we think this can solve. 


[00:25:07] John Nash: That's really fascinating and I think they're overlooked use cases when so much right now in the spaces that I think Jason is in and I'm in and around LinkedIn and our colleagues who are doing well and at our universities trying to socialize professors and then I teach teachers to become principals, but it's, let's get used to chat GPT and here, maybe it's a, prompt workshop, or maybe it's just thinking about how you're going to reset your assessments in the wake of GPT, but it's all about core GPT use and not these powered solutions. 


I see some of these popping up and mostly in the P 12 space for teachers can go and get like, help me write a letter of recommendation. I click a button and then it's a Khanmigo like and sort of, you know, I fill in some blanks and it writes that. Do you foresee. These GPT 4 powered solutions evolving more and core GPT use or whatever's next diminishing what's your thought on that? 


[00:26:06] Kristen DiCerbo: Yeah, I think that it's a lot to ask to turn every teacher into a prompt engineer. And so I think these solutions from people that can generally understand the problems of teachers and students are going to be the way that this technology can actually get into the hands more widely into the education space as a, and so the, all of these solutions that are powered by these large language models, I think are going to be the way to go as opposed to raw. Raw GPT. 


[00:26:40] John Nash: Right, right, right. Yeah, 


[00:26:43] Jason Johnston: I'm looking at some of the activities that you have listed under Khanmigo, and, you know, I was just thinking about how helpful, as you call them, powered solutions, John, and basically to describe for those of you that have not seen it out there. There's a menu along the left. And you can chat with Khanmigo in the middle as a teacher. 


But the menu starts with kind of my classes. And then that's where we already talked about the snapshot and summarizing chat history. And then it gets into teaching aids with a number of things about developing a lesson plan and lesson hook and learning objectives and rubrics and so on. And then a tutor me. 


Math and Science and Humanities Refresh, Practice My Knowledge, which was a nice way to, to put it for teachers. Refresh 


[00:27:31] Kristen DiCerbo: Well, and we also know there's teachers that are out there that are being asked to cover classes for the sub in, that they haven't taught that in who knows how long, and like, oh, I need to brush up on this, or I haven't taught this in 


[00:27:45] Jason Johnston: all the 


[00:27:46] Kristen DiCerbo: there's lots of things where you just need a quick brush up. 


[00:27:49] John Nash: that hadn't struck me yet. The, what value this has for a sub coming in to maintain continuity with what's been going on. And yeah, that's, yeah, that's good. 


[00:28:01] Jason Johnston: Then the rest of the activities are Write, Crafting a Story, Debate, Chat, Play, and then extra things like Ignite My Curiosity uh, Navigate, College Admissions and Financial Aid, which I thought was great. And I mean, I mean, seriously, it's a mess. And. any kind of help to navigate this, you know, my my kids are a few years out still and we're already stressing about all of this, right? 


Getting all the forms right at the right time and getting through this and then academic and career growth and personalize my interests. So I think this is a really fascinating menu that I'm assuming will kind of grow, but probably some concern about making this menu too long as well. 


[00:28:46] Kristen DiCerbo: yeah, and what I actually see happening is that some of these activities will end up embedded in our courses. So if you go to a traditional Khan Academy course, there's videos, articles, and exercises, and I can imagine, so, think about, the particular math topic, maybe you're doing Pythagorean theorem there could be and then an activity, actually in the course, it's like, talk to Pythagoras something that where they actually then they're not just hanging out by themselves, but are more embedded into the activity of the courses and then so some of those would come off that activity and the new ones that we're trying would come on to this activities page and land there and see how they go. So that's kind of what I'm seeing right now. They're separate, but I'm imagining in the future they actually become more embedded into our courses themselves. 


[00:29:39] Jason Johnston: Yeah. I love that idea. One thing that we wanted to talk about before we ran out of time was. Because both John and I are in the higher ed space we're you know, at the University of Kentucky University of Tennessee, we're both Canvas institutions, so Instructure Canvas. This summer at InstructureCon, it was announced that there would be a partnership with Khan Academy. There hasn't been a lot more details about that and even searched before this conversation to see, and there's really not much else except for that press release to talk about that. It kind of sparked our interest and actually part of the reason why we're contacting you. 


You know, we're fans of Khanmigo as well, but because this hits home in terms of our own context here. What else can you tell us about this or what are some of your hopes and dreams for this partnership? As I know it's just kind of in development stage at this point. 


[00:30:35] Kristen DiCerbo: That's right. So it is early days and we are working with Instructure to define what that, what we're actually going to create and when. So that's our, I have two meetings this afternoon about this very topic. So, so I can't tell you anything concrete about what we're doing, but I'm happy to say, you know, we can imagine, we know that Canvas is where a lot of teachers, professors, students are, and if some of these tools can be available on Canvas, it can make things then a lot easier for people to use them. They don't have to go to ten different places, everything is right in one central location. What does that look like? So, some of the things that are on our what if list, or could we do this list are, one, things like including these teacher assistant kinds of tools, and what would that look like? Second is thinking about, you were talking before about what is kind of the conversation around GPT, and a lot of it we know is around cheating and plagiarism and all of that. Well, what if you could use Khanmigo to help you draft and write your essay, would but that all is recorded, like, in the transcripts that we have, and so it's clear what the student did and what the model did and how that fits together, and we already have the rubric creation tools, can we then help the teacher create the rubric that they want to use to score this, and maybe even suggest to teachers, hey, we think this is a three on organization and a four on tone and voice and a five here, go ahead and, you know, review and create your own score.  


So, we think there's a lot of potential for some kind of a writing solution to help address a lot of the concerns that we hear around using these tools to just write your essay for you. So we'll see how, we'll see how that goes. 


And then, of course, there's a tutor. So, is there, you know, a Tutor Me in STEM and a Tutor Me in Humanities that could be part of Canvas? So, again, none of that is guaranteed, that is not an announcement, a product, that is the things that we've been thinking about. 


[00:32:48] Jason Johnston: Yeah, that's, that sounds very exciting and interesting. And I would expect that Canvas was a good choice for you all. And I'm not sure who approached whom, but because they're actually pretty 12 space as well. 


[00:33:06] Kristen DiCerbo: right, they have both the K 12 and the higher ed folks that are on their platform. And so that's a good piece. I will say my sister's a high school math teacher in Massachusetts. And when we were thinking about this I say, Hey, what learning management system do you use? 


Oh, Canvas. I was like, great. We're gonna maybe do something for you there. So, yeah, I think it's a good potential partnership behind both sides. 


[00:33:32] John Nash: Nice. I have a thought . I get the sense that Khan Academy and you, Kristen and Jason and I, we're kindred spirits, at least around this idea of where Khanmigo comes from as being an empathetic tutor, we're on a bit of a march to try to humanize online learning. And so as you start to think about the ways in which Khanmigo has been positioned as a support for teachers, and also I'll be honest, I mean, Jason and I are not really pleased with the level and quality of the instructional design that most people Professors bring to their online experience. Learner input is really not a part of course design as much as we would like. I'm not saying it doesn't exist, but as you think about our mantra of humanizing online learning and then thinking about potentials where Canvas and Khan Academy could come in, what are your hopes for that, and how do you see your tools helping advance that? 


[00:34:35] Kristen DiCerbo: It's a good question, and there's I have a number of thoughts. We'll see if I can get them out clearly here. So first is, there's a lot of research in the ethical aspects of AI about not anthropomorphizing it, and making it clear for someone that they are talking to artificial intelligence. 


They're not talking to a human and being careful about that. And we've done a number of things in our language is very careful with Khanmigo. It's never said. And things that make it seem too human and it doesn't have a human name on purpose and you know, some of those pieces. So we don't want the AI to become your best friend. 


Like that's not a healthy relationship. But what we do hope is that it can do some of the things that let the humans do the human part. And so if the classroom teacher or the professor isn't spending all the time on some of the things that Khanmigo can help with, can they do more of that? Help understanding the student as an individual, understanding what they need, understanding their goals and their progress towards those goals and working with the students on those kinds of things. 


That then can help the student actually build that relationship with a person as opposed to the AI. And, I mean, we know in the K 12 research world that if a student has someone that they think really cares about them in the school, their graduation rates are significantly higher, their going on to college is higher. 


So, having, being able to free the humans up to do the human things seems like a really important goal. 


[00:36:18] John Nash: Yeah, I think I would concur. I think that's the, and that's the conversation I've been having with other educators as they think about how to present even just raw chat GPT is what's the value proposition beyond just, Oh my God, how do I keep them from cheating? And it's sort of that it's how do we advance the human intelligence side of things and free up time for that with the artificial intelligence models. 


[00:36:47] Kristen DiCerbo: right. 


[00:36:48] Jason Johnston: I was at a tour of a Toyota plant in Kentucky, and they were showing us the robots, and they were very clear about emphasizing that the robots are here to assist humans, not to replace them. Which I agreed. I mean, I think that they believe that and I want to believe that, but then I also looked down the rows and I saw a lot of robots. How do we move forward with this to make sure this idealism that I believe that you believe what you're saying, but how do we move forward with this, that this idealism continues to ring true. 


[00:37:29] Kristen DiCerbo: Yeah, I think we can't be idealistic to the extent we don't see the potential problems. And you can absolutely imagine a future where, hey, the AI can do this, that human stuff, it's not that important. We don't actually need that many humans. To do all of this. That's on all of us together to push back on that and to say, no, the human part is the important part. 


And we do need that many teachers. And that means we need to support paying that many teachers and all of the things that go in that policy sphere, which is. It's not out of my realm of expertise, but I think because what we're doing has the potential to have impact there, we would be remiss if we weren't thinking about how we need to make sure that we are maintaining those commitments to how important all of those human things I was just talking about are, otherwise they're going to get pushed to the side. 


In favor of technology. So we all have to continue to hold hands and agree that, yes, these are important things that are worth pushing for committing to as the technology advances. 


[00:38:40] John Nash: Because Khanmigo is a G P T four powered solution, and if it arrives in a classroom to help a teacher in a busy situation where students have their hands up, that implies that generative AI has landed in that classroom. Have you or your colleagues been having conversations with schools or districts about their own guidelines for AI and whether they should use it at all? 


[00:39:06] Kristen DiCerbo: Yeah, as you can imagine lots of people want me to come talk to them. And so, we've been having a lot of that. We also are part of a coalition called Teach AI, which is Code. org, us, ISTE and ETS. The World Economic Forum are kind of the steering committee, and we have a whole bunch of... 


Other folks on the advisory committee and our charter is to think about how to teach about AI and teach with AI and a lot of the first things we're doing are setting out and will be, stay tuned, releasing some guidance for school districts on what should policy look like around this and whether some examples that folks might use if they're looking for those or just some guiding principles to, to work with Thank you. 


How those fit together. So we definitely are hearing that. And then we also have the past spring we were working with a couple of districts to pilot Khanmigo and then continuing this fall. And so those districts that are working. closely with us specifically on Khanmigo are also, giving us a lot of feedback on how it's working, what's working, what's not working, and how, what can we continue to build and change, and how are they developing their policies. 


So that's been really informative for us to see what happens when this is tried out in classrooms. 


[00:40:25] John Nash: Yeah. Really nice.  


[00:40:27] Jason Johnston: So what do you, what are you hearing from either classes and or Konamigo because it's in the pilot right now and I'm assuming a full rollout will happen in the next. 


[00:40:39] Kristen DiCerbo: We're kind of waiting 


[00:40:41] Jason Johnston: Okay. 


[00:40:42] Kristen DiCerbo: to see Before we release, we say, yes, this is, you know, releasing what the other piece we haven't talked about this much. The other piece that is a concern and that we need to continue to work on is the cost of this. Because GPT 4 actually costs, and so right now for Khanmigo, we are working on this through our district programs, but it is a cost add on to what we have, or individuals can get it for 9 a month or 99 a year, but you know, we would like that to come way down and be able to offer this to as many people as possible at as low a cost as possible, but that involves The cost of the computing power to come down, which everyone forecast will happen when GPT 4 got released. 


The cost of GPT 3 went way down. So we're hoping to see, you know, those similar kinds of patterns, but we shall see sorry, that was a little tangent to the question of, 


[00:41:42] Jason Johnston: that's helpful. Yeah. 


[00:41:43] John Nash: No, it was good. 


[00:41:43] Kristen DiCerbo: . So in terms of what we are seeing now from districts, there's some things around just specific features. 


So when we talked to district administrators, they said they actually wanted to also be copied in on those flags I was talking about on moderation. So they go to the teacher now, but the district said, you know what, if there's violence or hate or threats of self harm, we actually want to get those warnings too. 


So things like that. Okay. Let's. Yeah, make that happen. And how those work. We heard from teachers when we first released for teachers. We just had a couple of activities and was really through talking to teachers who said, Hey, could this help me unpack a standard and make learning objectives? Hey, could this help me write a rubric? I can never figure out how to differentiate level two and level three in a rubric. Could it help me with that? I said, sure, let's give it a try. So a lot of what you see now in the teacher activities are the results of some of the early feedback we were getting from teachers that then we turned into new activities that they could do, which is fun. 


And then we look at, for the students, we are doing, we do student interviews and all that. We also look at our data and we see which activities are the most used activities, which Tutor Me STEM is far and away the most commonly used activity. But craft a story does pretty well too, which is set up so that the AI writes two sentences and the student writes two sentences and the AI writes two sentences and you write a story kind of together like that. 


And students seem to. catch on to that one quite a bit as well. 


[00:43:17] John Nash: I was playing with the teacher version this morning before we met and I did notice that Khanmigo asked me to put the standard in when I was going to do some outcomes work. And so I think that's a nice aspect for teachers because they're so wed to that, but then how do they know they've really tied things in? 


And so that just handles that. 


[00:43:37] Kristen DiCerbo: Yeah, absolutely. It is the case that if you just put in the standard code, so in, for instance a state standard usually has some code that's, you know, MA. 3. 2 for a math. 3rd grade thing. It does not do a good job of just reading those codes and does not seem to have those. So, we do ask that you type in, put in the whole standard language. 


So there's some things that are, you know, the models still don't do well and we need to work around. 


[00:44:07] Jason Johnston: One of the things I really like, too, about Khanmigo versus just directly working with ChatGPT is this kind of prompting you forward. And so, if you started down the road, like, say, by, creating a lesson around a particular standard. It continues to prompt you forward to help you, help guide you, give you some example of some next prompts that you might do, might ask some questions about what you want to do next, or how about doing this next. 


And I really I appreciated that kind of guided Inquiry that you don't get with all the other LM and LLM models. Unless you pre prompt it to be that kind of model. So... 


[00:44:48] Kristen DiCerbo: yeah, that is definitely something we've designed into it. I refer a lot to this framework by Mickey Chee and Ruth Wiley, who are at Arizona State. Not just because I'm an alumnus and appreciate people from ASU. But it, they call it the ICAP framework. And basically, as students move from passive to active to constructive to interactive activities, learning increases. 


But it's the same for all of us as we're designing lessons or what that is. And so constructive is, you know, when you're creating something yourself, but the interactive is when you're actually working with, the research is on people, so it's research with another person. When you're working together and both of you are making constructive contributions to that final product, that actually leads to the better product. So it's not trying to do it itself, it's not asking the user to do it themselves, it's actually trying to collaborate. 


And that's what we've designed into the prompts that are those prompts behind the scenes is asking it to ask another question to finish with a question to lead you, the user along in whatever you're working on in the activity. 


[00:45:54] Jason Johnston: I manage a number of instructional designers at the University of Tennessee, and I mean, I've got a great instructional designers, and that is what great instructional designers do, right? They prompt, they guide, they help things move along they're listening, but they're also co constructing. 


And that makes for a really exciting dynamic, and I think that there's some space for that with a a chat bot or with a a chat GPT kind of tool like Khanmigo. 


[00:46:27] Kristen DiCerbo: absolutely. That is definitely what we're going for. So I'm glad that came across. 


[00:46:31] Jason Johnston: Yeah. You know, I've got a, I've got kind of a final question here. You know, Khanmigo is super cute. They're... Always there for you. Helpful, supportive, smart. What's Khanmigo like in real life? I mean, you've got the inside scoop. Is this really all there is to Khanmigo? Or is there like, you know, what are they like? 


[00:46:56] Kristen DiCerbo: Khanmigo is whatever you make Khanmigo to be. And so 


[00:47:01] John Nash: Wow. 


[00:47:01] Kristen DiCerbo: this is an important point. We are. are still in control of the machines. 


[00:47:07] Jason Johnston: ha. 


[00:47:08] Kristen DiCerbo: we are telling Khanmigo to act that way and to interact that way. So it is driven by the personality that we at Khan Academy have given to Khanmigo and that's a, it's definitely important to know that that's because of the work we have done, not anything Inherent in the large language model. 


[00:47:30] Jason Johnston: Yes. Well, that's good. And I think it's another great distinction about what makes Khanmigo different than just sending your students to chat GBT, certainly on any level. And I think certainly, I think as a student gets older, I think they're going to need to figure out ways to wield that wide open space, right? 


But as they're younger, there are lots of ways in which you want to guide it a little bit more and make it a safer space. And I really appreciate I appreciate that. And I appreciate that approach. 


[00:48:04] John Nash: I do too. I appreciate the, I've just even in this call, I mean, I've come to appreciate more the effort going into what do we call it, Kristen? It's a multi layered prompting scheme that's going on in the background to bring back those responses, but it's the, some of the best. prompts I've seen in sort of raw GPT use for say, guiding teachers interactively to set a clear measurable learning intentions and then create activities to do that, they're four page prompts. They're long and they work, but they're really beyond the realm or scope of most teachers to think, Oh, I need to put in this four page prompt and then go through this process. Whereas you guys have built in the four page prompts. I don't know how long. I don't know how long all that is back there, but to bring really high quality and interchanges between learners and in the machine, but also between teachers and learners, 


[00:49:04] Kristen DiCerbo: Yeah, absolutely. And yeah, some of the prompts are pretty long. And sometimes if they get too long, the model stops reacting to all the parts of them. So we do have to do some things. Called prompt chaining and some fancy new things that are, were develop, you know, just being developed. But yeah, all of that is the idea that the teacher doesn't have to be an expert in all of this. 


I do think in the future, as you were saying, that people are gonna have to learn some prompt skills. But it takes a lot. It has taken us a lot of experimentation to get to where things are. That's not something your average teacher has the time to do. 


[00:49:40] John Nash: right? 


[00:49:41] Jason Johnston: That's great. Well, Kristen, thank you so much for this conversation. We will in our show notes online learning podcast. com, we'll put in links to Khan Academy, of course, and Khanmigo and where you can sign up and give it a whirl. And yeah, we just really appreciate you taking the time to talk with us about this. 


This has been a great conversation. 


[00:50:03] Kristen DiCerbo: My pleasure. Thanks for all the good questions. 


[00:50:07] John Nash: You're welcome. 


[00:50:08] Jason Johnston: Yeah. Thank you. And for all of you listening, I hope you jump in with us on the conversation on LinkedIn and we'll put that link in as well. We'd love to hear what you think about this episode and what else we should be talking about and thinking about in regards to this. Thank you so much. Thanks, John. 


Thanks, Kristen. Great talking to you. 


[00:50:26] John Nash: Thanks, everybody. . 


[00:50:28] Kristen DiCerbo: bye.