Whenever you hear an AI researcher dismiss criticism as pearl clutching, you should hold onto what matters and start paying attention. I saw this play out during a recent panel where concerns about using GenAI for mental health support were categorized as old-fashioned thinking. We don’t worry when someone sits alone journaling for mental health support. Why should we worry if that same person types into a GenAI prompt instead?
Your diary doesn’t care about your data. The pages you fill won’t be sold to a data broker or used to build new AI models.
Many companies specify standards of conduct to prevent conflicts of interest. While preventing an actual conflict of interest is good, it isn’t always enough. A higher standard is to avoid any conduct where there could be a perception of a conflict of interest. Until there is more regulation and governance around the use of AI for mental health support, there will be a clear possibility of conflict of interest between user privacy and company profit.
The AI proponents claim that people are ok with trading off their data for the benefits that they receive in return. Look at all the deeply personal information put into search queries for decades. This data trade is a lot like smoking cigarettes. Most people don’t really understand the potential harm that they are exposing themselves to. Once laws and research brought more transparency to the effects of smoking, there was a dramatic decline in the number of people who were willing to continue the tradeoff. Nothing changed with the actual risks; knowledge of those risks drove a behavior change. It is naïve, perhaps intentionally so, to assume that most people know the potential harms sharing their personal mental health concerns with general-purpose chatbots.
I am only talking about the data exposure risks here. I will leave other harms and benefits for another time.
Leave a Reply