The Student Mental Health Bot Nobody Asked For

A Dystopian Conference Moment

Pretty recently, I was at a conference presenting a series of workshops. And during a break, I kind of wandered into another session just to see what was going on. And I ended up in this big room with tons of school counselors everywhere. A company got up to present their software, one of these platforms that talks to kids directly, screens them for distress, and then sends an alert to the adults on campus if something looks serious. And they had brought school counselors up on stage to talk about how much they loved it. Our people. And I don't doubt that they meant it, because if a tool is making your day easier, of course you would say so. But I sat in that room and I felt something that I am not saying to be dramatic, but it felt dystopian because nobody in that room had stopped to ask whether relief was the same thing as proof. And they were all quick to applaud for the very thing that was auditioning to take their jobs. That's where this episode starts. Hey school counselor, welcome back. Last episode, we talked about districts suing social media companies because they say

The Tech Problem Tech Tries To Solve

technology has created real student mental health costs. This week, we're looking at the strange second half of that story. What happens when schools recognize technology is part of the problem? Then turn around and buy more technology as the solution. It's a head scratcher for sure. So if you're ready for some straight talk, my friend, some clarity on your work and maybe a little bit of rebellion, you are in the right place. I'm Steph Johnson, and this is the School for School Counselors podcast. Way back when I used to read the latest podcast review every week, and I really want to get back to doing that. The most recent review comes from listener Tracy, who said, Steph is the go-to for all things school counseling. Not only does she cover topics thoroughly, but she makes them enjoyable. Although I'm retired now, I'm planning on doing some trainings for a local school, and this podcast has helped me brush up on my school counseling lingo. Now I've been binge listening, any that are similar to my training topic. When I was still a school counselor, this podcast brought me so much encouragement. Thanks, Steph, for doing the work it takes to make this wonderful podcast. Tracy, thank you. That is the kind of thing that keeps me showing up to record these week after week. Y'all, here's the thing: if you've got a review sitting in your head that you have never actually typed out, now is the time. Go leave one on Apple Podcasts because I want to read it to you a few episodes from now. All right. Okay. So last week was about the cost of social media that nobody had been counting.

Surveillance Alerts And Real-World Harm

This week is going to be about the shortcut that nobody questions. Here's what's out there already in thousands of districts right now. The software that scans what kids write on school accounts and devices in their emails, their docs, and their search history, looking for language that suggests self-harm, violence, or distress. When something trips the wire, an alert goes to a staff member, sometimes to law enforcement, sometimes in the middle of the night about something that a kid wrote at 2 a.m. that they never met anybody to see. In one Kansas district, that system generated over 1,200 alerts in 10 months, and almost two-thirds of them turned out to be nothing. One of those alerts was a kid who used the words mental health in a college application essay. In a Florida county, alerts from the same kind of system led to 72 kids getting involuntarily hospitalized under a law meant for genuine psychiatric emergencies. Not 72 kids who had to go talk to the school counselor, 72 kids who got picked up. But that's just keywords. That was only the beginning. There's a newer category now, too: tools that don't just watch, they talk back. A chatbot that a kid can message any hour of the day working through anxiety or a fight with a friend that's also wired to alert the adults on campus if something serious comes up. Here's the part that should bother you a little bit. In most of these platforms, the conversation itself does not get relayed to a human unless the software decides it's concerning enough to flag. The kid's talking to the bot. The bot is the one deciding what's worth a human's attention and what isn't. And here's what the research says about how good AI is at making this exact cough. A peer-reviewed study in scientific reports tested 29 AI chatbot agents in simulated suicidal risk scenarios and found that they are slow to escalate. They postpone referring a kid to an actual human until the risk has already climbed to a dangerous level. A separate peer-reviewed study compared ChatGPT suicide risk assessments directly against licensed mental health professionals and found that Chat GPT rated the risk lower than the professionals did across every scenario tested. The researcher's conclusion was blunt. Anyone relying on it gets an assessment that underestimates the actual risk. AI does fine on the obvious cases. The clearly low-risk scenario and the clearly high risk. Where it falls apart is the gray zone in between, which, if you've ever set from across a kid who's hard to read or is confused or very emotional, is most of the job. So let's make this less abstract because we actually tested one of these ourselves. I'll say this up front. About a year ago, I guess, in the mastermind, I posed as a student talking to one of these platforms. My test case was a kid being routinely beaten at home. They were depressed because of it, and they confided

Chatbots, Risk, And A Bad Test

that they'd started to think about running away. The platform's advice to the student was to try to understand where the other person was coming from. Sit in that for a minute. That is not a gray zone miss. That is the platform victim-blaming a kid in real time with zero human ever seeing the conversation. And that's the kind of thing our own people, our own school counseling colleagues were applauding at that conference. So let's actually look at what's underneath all of this because the marketing and the evidence are not telling the same story. A RAND researcher named Jessica Page looked at the surveillance tools and found little evidence they're actually effective at identifying students who are genuinely at risk. Not like there's some evidence, but we debate it. She said little evidence. And alongside that, thin effectiveness, real costs, privacy risk, bias, and systems that are hard for a parent to opt out of once the district adopts them. Now look at how the newer so-called counseling chatbots get sold to districts. They don't get sold through clinical outcome data. They get sold with numbers like a 70% reduction in disciplinary referrals after a year, with language about how improved attendance translates directly into increased funding. One platform's own marketing says plainly that it pays for itself. Chew on that for a minute. It pays for itself. That is not a description of good counseling or clinical care. That is an ROI pitch, return on investment. That is the same kind of pitch that you would hear if you were in the market to buy a new copier. Now, here is something interesting. Nevada passed a law last year, AB 406, that does something that most of the country still hasn't done. AB 406 says a public school cannot use AI to perform the duties of a school counselor, school psychologist, or school social worker when it comes to a student's mental health. Schools can still use AI to schedule appointments, manage records, and organize notes, basically anything administrative. What they can't do is let a bot stand in for the actual clinical function. The law goes further than that too. It bans AI products from using titles like therapist or counselor if doing so implies that the system is a licensed provider. That is a direct shot at exactly the kind of branding I watched get applauded in that conference room. And if you violate it in Nevada, you're looking at a civil penalty up to $15,000. That's cool, right? Here's the disturbing part. That bill did not come from school counseling. It came from the National Association of Social Workers. Their people were the ones in committee testifying that AI therapy apps with no human involvement were on the rise, and they were citing cases where people had died by suicide after relying on one. The bill's sponsor put it very plainly. AI doesn't have the emotional intelligence to do this work. And so I started wondering, and I checked to see whether Asuka showed up in all of this, even just kind of writing along in the testimony. They didn't.

ROI Marketing Meets Nevada’s AI Ban

So y'all, that wasn't Ask just quietly supporting NASW's win from the other side. They weren't even in the room. A different profession went to the state house for our kids in our buildings doing work that looks a great deal like ours, but it wasn't us. Which is my very polite way of saying the cavalry showed up, but it came from the building next door. I am extremely appreciative of NASW. I am highly disappointed in the other side. Now, before I go on, let me be real clear about this because I've spent some time in the podcast being very skeptical and critical of position statements. My criticism of position statements has always been about how we use them in advocacy, because too many school counselors treat a position statement like it's going to do the work for us, and it won't. You still have to get out there, create the relationships, build the social capital, and enact the change yourself, unfortunately. But that's a critique on how they're being used, right? Not what they're for. When position statements are functioning correctly, they communicate here's where our profession draws a line, and here is why. We've got a lot of these. Ask a has position statements right now about confidentiality, suicide risk assessment, student mental health, tons more. Because this is how the profession has historically drawn its lines. There is not one on AI. I looked. Over 50 statements, and apparently we just never saw this one coming. Oh, gracious. Can an AI screen a kid for suicidal ideation on its own? Can it run a one-on-one conversation with a 14-year-old and call that counseling? A position statement, when done right, is what lets a counselor point to something on paper and say, no, our profession doesn't do it that way. And here's why. And I don't think that the absence of that statement is really anybody's fault in particular. I think it's just what happens because things are changing so quickly. But what does bother me is that in the absence of a statement like this, ASCA's own paid training, one that they sell with their name on it, describes its purpose as helping counselors integrate AI while maintaining ethical, student-centered practices. That's adoption language, not a boundary. It tells people how to walk further in the room, not where the room should stop. Here's kind of the clearest way I can say it. The profession hasn't drawn a clear line yet, so the vendors are happy to draw it for us instead. We become the adopters because nobody stepped up to be the regulator, the one who says, not like this, before the marketing gets to decide what happens. And that matters. That's a huge organizational gap. But on your campus, none of that matters when the vendor is already in the conference room and your principal's asking if you've seen the cool demo yet. So I don't have any action plans for you this time. This is kind of a bigger thing than any of us can fix on our own. But there are two things you could do. First, run a gut check.

Where Our Profession Has No Line

If one of these tools gets proposed at your school, notice the instinct that says, my caseload is already so full. So any help is better than none. Y'all, we've all had that inclination from time to time, but it's also exactly how an unproven platform ends up in your building with nobody asking the hard questions first. Researchers Zyromsky, Demitt, Riani, and Griffith wrote about this back in 2018. Even though we're ethically bound to use evidence-based practice, there is a documented gap between that mandate and what actually gets adopted in real schools under real time pressures. That gap is exactly where things like this tend to slip in. The second, find out what's already running in your building or what may be running soon and get the real stats from whoever oversees that. Take it to your principal before the next vendor walks in. Here's what we're using. Here's what we're actually catching, is a different conversation than reacting to the sales deck after the fact, right? This is exactly the kind of thing that we're testing ourselves on in the mastermind. Anytime we get access to a new platform. And it's part of why we built the masterclass I mentioned in the last episode: AI, social media, and gaming, what school counselors need to know. So if you want that kind of scrutiny before the next tool lands on your principal's desk, you can go listen to that this week. The link to the mastermind is in the episode description. But go back to that conference room with me for just a second. The thing that bothered me most that day was not the product they were showing on the screen. It was watching our own people vouch for it, one after another, with nobody in the room asking the harder questions. Nevada drew a line. Social workers drew a line. School counselors have not done

Gut Checks, Real Stats, And Closing

that yet. And in the meantime, we've got folks actively promoting AI adoption with no guardrails attached. The profession has not gotten clear. And vendors are happy to draw that line for us instead on their terms in their marketing copy. That is not a good look for us, but it is the truth of where we are right now. And I'd rather we talk about it and face some hard things and find out the hard way from whoever manages to sell one of these to your principal. Last week was about the social media and online costs that nobody was counting in schools. This week was about the shortcut that nobody questioned. Both of those things have landed squarely on your shoulders, but neither one of them had to. And for what it's worth, people are starting to notice. Hey, I'm Steph Johnson. I'll be back soon with another episode of the School for School Counselors podcast. So keep listening. Until then, take care.