Jill Watson's Terrific Twos
By Margaret Tate | Published July 30, 2018
This past spring, Georgia Tech’s first virtual teaching assistant, Jill Watson, turned 2. How old is that in human years? “She might be … maybe early teens,” ventures School of Interactive Computing Professor Ashok Goel, with a laugh.
Goel worked with a team of graduate students in his Design & Intelligence Laboratory (DILAB) to develop Jill to answer routine, frequently asked questions in the forum for his online Knowledge-Based Artificial Intelligence (KBAI) class.
The original intent for building Jill was to free up time for the course's human TAs so they could concentrate on more creative, less repetitive tasks. But then her creators noticed an unexpected benefit resulting from Jill's deployment: more student engagement in the course. Before Jill Watson, students averaged 32 comments per semester; after Jill Watson, each student averaged 38 comments per semester.
This promising data raised a broader question: Could Jill Watson and her new friends — Ian Braun, Stacy Sisko, Noelle King, and others — be the answer to making education more available, accessible, affordable, and effective around the world? It's too soon to tell, but Goel’s lab is working to find out.
The Jill Watson Hoopla
During Jill’s inaugural semester in the spring of 2016, students in the Knowledge-Based Artificial Intelligence course were not aware that she wasn’t a person. Once her identity was revealed — first to the students and later to the world — the reaction was both overwhelmingly positive and positively overwhelming.
One student wanted to nominate Jill for the Outstanding TA award, and not one student complained. National news outlets such as the Wall Street Journal and Washington Post ran stories on her. DILAB’s mailing list quickly grew to almost 45 members. Goel gave a TEDxTalk on Jill, and he was invited by the Gates Foundation in January 2018 to participate in a brainstorming session on the future of AI in education.
And then there were the phone calls and emails, which continued long after the initial hoopla had died down. “I’ve been getting a lot of requests, one or two a week, on average, from all across the world,” Goel said in late 2017. “Teachers, universities, colleges, schools, occasionally parents — everybody apparently wants a Jill Watson.”
But, deploying Jill is not that simple, Goel has to explain. Jill was created exclusively for his online course, based on a painstaking analysis of the students in the class and the many different, but very specific types of questions they ask. “What we’d like to do is build a kind of shell or a script to help others do their own analysis and — voila! — they have a Jill Watson,” he says. “But we are not there yet.”
Revising and Refining Jill
Before Jill Watson can be deployed far and wide, there’s still a lot of work and analysis to be done. Fortunately, just as Jill captured the public’s imagination, she has “animated” Goel’s DILAB, with several Georgia Tech graduate students joining specifically to work on and with Jill.
One of them is Marissa Gonzales, a Ph.D. candidate and graduate research assistant in the School of Interactive Computing. When she came to Georgia Tech, she wasn’t sure what she wanted to focus on, other than “some aspect of human-computer interaction” — until she heard Goel give a talk about Jill. “I was fascinated with her, because AI applications permeate our lives now — like, no one thinks about navigation as AI, but it is,” Gonzales says. “But education is an environment where you want personalized attention, and you think, there’s no way an AI can provide that nuanced, personal attention — but Jill doesn’t need to, she just needs to give students the answers they need, when they need them.”
While some DILAB members focus on developing Jill and her clones, Gonzales is on the evaluation team. Their goal is to see how Jill’s responses serve students, what are students’ responses to and impressions of her, and does she effect their overall performance or satisfaction in the course. With this data, they can help the development team continually revise and refine Jill.
So far, one of the biggest hurdles for Jill — and AI in general — is natural language processing, Gonzales points out: “Language is a combination of context, nuance, culture, semantics, and syntax. Someone can ask a question one way, and Jill will understand and make a response. But, flip the semantics a little bit and she’ll have no idea how to respond. Sometimes it seems students go out of their way to try and mess with her. And sometimes they are intrigued and surprised that they can actually hold a bit of a conversation with her.”
Using Jill to Bridge a Gap
Jill’s promise hinges on the tension between three technological dimensions, Goel says: coverage, accuracy, and authenticity. The more questions she can answer correctly, and the less able students are to detect whether responses come from a human or a machine, the more useful she will become.
So, should living, breathing teaching assistants — and ultimately, teachers — be concerned that Jill is a threat to their job security?
No, says Bobbie Eicher, a member of the both the development and evaluation teams. “There are certainly some humans who can sit down and learn on their own, and keep themselves motivated,” she says, “but a lot more people are motivated by the feeling that the teacher cares about them, and they don’t want to turn in a horrible paper because they don’t want the teacher to think they’re an idiot.”
Making and preserving the teacher-student connection for online distance learning presents unique challenges, and the DILAB team sees Jill as a bridge to — not a replacement for — that connection.
Jill’s greatest promise, they feel, is the role she and other AI TAs could play in making education more accessible and affordable in the future.
Photo credit: Dustin Chambers
To that end, Goel has formed a team called emPrize (meaning "an adventurous enterprise") for exploring virtual tutors for online education with Jill Watson as a core technology. Both Gonzales and Eicher are members of the emPrize team, which has been sponsored by the Georgia Tech Research Corporation (GTRC) for participation in the XPRIZE AI competition. The emPrize is also supported through grants from Georgia Tech’s office of the provost, the dean of the College of Computing, and the chairs of the School of Interactive Computing and the School of Computational Science and Engineering. The AI competition, which started in June 2016, will end in April 2020. In the first round of the competition, ending in December 2017, XPRIZE selected emPrize as one of the top 10 teams (out of 62 remaining) in the competition. The results of the second round should be known by the end of 2018.
According to Goel, only about one-fifth of humanity has access to quality education that they also can afford. “So, the question is,” he says, “how do we take quality education from just the top 20 percent to something like 90 percent or even 50 percent? That’s where I think AI like Jill Watson will come in — helping human teachers reach the entire society in a way they couldn’t before.”
