How to Interrogate a Corporate Chatbot: A Primer and Demo for Journalists

Corporate shopping chatbots are marketed as sales tools, but they might also be one of the only public-facing ways to query things like a retailer’s product catalog, business rules and metadata.

Join us for a hands-on demo featuring Erie Meyer, former chief technologist at the Consumer Financial Protection Bureau.

Meyer will walk through a new study she co-authored at the Columbia Law School’s Center for Law and the Economy, which found that Amazon’s and Walmart’s shopping chatbots can detect false “Made in USA” claims in their own product listings — but were built not to protect shoppers.

In this demo, Meyer will show how she and her co-author approached the research, including cases where a bot claimed it couldn’t do something — and simple follow-up prompting got it to do exactly that.

She’ll walk through the specific techniques used and how to adapt them to interrogate other corporate chatbots for your own reporting. By the end of this demo you’ll gain a better grasp of:

• Why corporate chatbots can be a uniquely useful reporting tool for accessing information that may otherwise be opaque.
• Concrete prompting techniques for getting underlying information.
• How to design your own tests to see what other companies’ chatbots will — and won’t — tell you.