Prof. Sebastain Thrun's experience of teaching 160,000 students AI for free!

[Extracted from June 2012 mails, so a little dated.]

In Jan. this year, Prof. Sebastain Thrun, Stanford professor (may have left Stanford by now, I think) gave a talk on his "profound" experiences with a wildly popular internet medium teaching course on AI that he taught late last year, and how he feels that it will revolutionize education. There seems to be some amount of marketing hype in the talk. Free Internet medium teaching for higher education in the form of video lectures has been around for long now (MIT OCW, Indian IITs NPTEL etc.). But perhaps the way the internet was used for this course was far more effective. It somehow managed to engage stupendous amount of students worldwide by bringing in some amount of interactivity (online quiz), forcing the students to think, and having a regular assignment/exam pattern that made students work hard if they wanted to continue with the course.

I am a great fan of on-demand learning as that is what I have followed quite successfully in my software industry career. However my knowledge is not certified. It did not matter for me while I was in the software industry as the industry has the means of assessing knowledge and capability of persons. But for Indian academia and Indian government type of organizations my international industry experience is almost non-existent! For aspiring students of today, no matter what age or what country, certified free higher education from top notch Professors like Prof. Thrun would be like manna from heaven, especially given the debt trap that many students fall into while pursuing higher education not only in developed countries but even in developing countries like India.

Prof. Thrun and other initiatives like MITx seem to be making on-demand higher education with certification of some kind a reality TODAY or in the very near future (MITx perhaps is behind Prof. Thrun's Udacity as of now). Here's the youtube video link of Prof. Thrun's Jan. 2012 DLD talk. While all of us may not agree with all of Prof. Thrun's views I feel we may benefit by knowing about his experience of moving from a Stanford classroom of 200 students to the Internet medium classroom of 160,000 students. I am awaiting permission before I put the partial transcript of the talk on this blog post. Meanwhile I am putting down some comments from me below. The comments will make sense only on reading it after viewing the video/along with viewing of the video.

-----start comments ----

Prof. Thrun got inspired by listening to a TED talk by Mr. Salman Khan of Khan Academy. At that time Prof. Thrun was a tenured Professor at Stanford University.
...
Prof. Thrun mentions that students would get a certificate at the end.
[Ravi: I think what separates Prof. Thrun's AI class from MIT OCW and IIT&IISc NPTEL is the exams and certification. Further the exams were the same that the Stanford classroom students would take! That would have made it as close to a Stanford course as possible. Perhaps that's what drew the 160,000 students. I mean, which student interested in learning software would want to miss up on attending a Stanford course for free from the convenience of an Internet connection anywhere in the world and the chance of receiving a certificate on successful completion.]
...
Thrun shares his amazement when 160,000 students sign up!
[Ravi: The POWER of the Internet is mind-boggling! I think it can be such a fantastic force for good if wielded properly.]
...
They use just a camera, a pen and a napkin for course recording!
[Ravi: As simple as that, huh! I think one just needs the will and the knowledge for Internet based offerings. The budget is not really an issue.]
...
He talks of far more engagement with students in this medium.
[Ravi: From my industry and academic teaching experience I know that interactive sessions for teaching software technology go a long way in better learning of students. So I agree with Prof. Thrun's technique here to be more effective in teaching a wide group of students. Perhaps if one is having an elite (knowledge-elite) group of listeners the one-way lecture style is better.]
...
Paying, in-class students prefer to watch him on video!
[Ravi: That is a very telling experience. Students who have paid money prefer to use his free internet teaching ways than his classroom!]
...
He shares the Afghanistan student experience.
[Ravi: Higher education for all no matter which country you come from, which economic class you come from, which religion you profess - and for free. Wow!!! I think that's the way it should be. Education should ideally be "universally" accessible. And the vastly accessible and cheap medium of the Internet world wide is a fantastic carrier. It can even penetrate a war zone!]
....
He shares the experience of a student who is a lady with a teething infant and facing significant life-problems.
[Ravi: This melts one's heart. I mean, the lady has all the odds stacked against her. But she wants to fight her way to get out of her problems. And free higher education is giving her a chance to come out of it. Even if the AI course did not land her a secure job, if it gave her a sense of accomplishment I think Prof. Thrun and team would have done a fantastic good Samaritan job here.]
...
He talks of Stanford having weeder classes but that this medium can do it differently.
[Ravi: I think we have an elite vs. commoner education system issue here. The elite education setup weeds out the guys who can't come up to the expected elite levels. Perhaps that's necessary in an elitist setup like a Stanford, an MIT or India's IITs and IISc. But "universal" education has to cater to both the elite and the commoner. A "universal" education model cannot have weeder classes as then it will fail to be "universal" and become an "elite" education model.]
...
He mentioned that students felt more connected with him even though it was an internet video medium course!
[Ravi: This feedback was quite a surprise to me initially. But as I think about it, having seen a lot of youtube and other videos over the past few years, I find that I too develop quite some intimacy with the characters in the video. I have replayed some videos which really impressed me (like this one) and so have noted so many aspects of the main speaker/characters in the video like their facial gestures, their voice inflections, their accents, their 'mistakes' etc. In a regular classroom I am a back-bencher by choice and so would never have observed the lecturer/instructor that carefully.

Of course, the human contact aspect especially the vital eye contact aspect between the instructor and the student is not available in this Internet classroom. Further, the student cannot raise his/her hand and ask a question directly to the instructor during the course of a class, at least. Asking the question in some forum may result in a 'wrong' response - the forum is not as 'authoritative' a knowledge source as Prof. Thrun.]
...
He states that Profs teach the same way as it was done a 1000 years ago.
[Ravi: I think that's quite unfair. I have used PowerPoint slides and demonstrations of programs projected on screen in my over 9 years of teaching in a deemed university in a rural area of India.]
...
He says universities are rather slow in innovation and that people should reconsider this new medium.
[Ravi: Perhaps there is some truth in this part. Having taught both in the Indian software industry and in Indian academia I must say that I feel Indian software industry teaching techniques and practices (pedagogy) for software technology topics are far superior to Indian academia. But then Indian software industry did not have to worry about funds for teaching equipment. Indian academia has to deal with massive funding issues and all streams like Literature, Commerce, Pure sciences etc. have to be treated largely on par with sunrise fields like Computer Science or Information Technology. Indian software industry teaching/training departments don't have to worry about such problems.]
...
He talks about launching Udacity on that day.
[Ravi: He was charged in this talk, no doubt. I guess it is such guys who start off revolutionary movements. But will it stand the test of time? I wonder how Udacity's other courses fared in the first half of this year. They did not seem to make the waves that Prof. Thrun's AI course did. I think then he had the Stanford brand name. Now the name is Udacity. That's very, very different for students worldwide, I guess.]
...
He talks of a course which will teach a student with zero programming background to build a Google in seven weeks!
[Ravi: Building a search engine in seven weeks with zero programming background - That itself is a real tall claim. But, Okay, I am willing to take that. Maybe they are fantastic teachers and are going to teach just the bare minimum needed to know in programming to do a small search engine.
But building a Google in seven weeks with zero programming background - this will convey a completely wrong impression to many naive students that in seven weeks they can have their own search engine which can kind-of compete with Google. I feel, this statement is way, way over the top.]
...
[Ravi: I agree with Prof. Thrun that "programming is really important around the world right now in this time and age."]
...
He ends the talk asking support for Udacity.
[Ravi: Overall, I think this is a wonderful development. I hope Udacity is able to succeed in its endeavour to provide free higher education in the software field. Its success, even if partial, may result in big guns like MIT and India's IITs to try to modify their free higher education offerings to this perhaps more effective and popular model.]

--- end comments ---

Some remarks I made in June to an academic
[slightly edited as it is not appropriate for me to share that academic's private views - I have stated and modified only my response to be readable standalone].

Students need to put in sustained effort. Perhaps by giving the "same" exams that were given to the Stanford classroom students, Prof. Thrun ensured that students put in sustained effort. One of my past students, who now works in the computer department of a super speciality hospital, took the course. He was pretty squeezed for time and found doing the assignments/exams to be quite demanding. Like the 'daughter' who dropped out of the course due to the level of Statistics (Probability, I guess) knowledge needed, my ex student had to struggle a bit on it. But as he has a Maths background he could refresh his knowledge and pick up whatever additional knowledge that was required to be able to handle the Statistics part of the AI course.

So the impression I have is that students did have to put in sustained effort even in this Internet classroom AI course.
...
I agree that there seems to be a lot of hype ("star turn") to this AI class - especially the 160,000 number and world wide spread of students. So maybe there are quite a few drawbacks with their first such effort. But I believe for this particular course, perhaps due to the marketing hype, the support forums played the role that TAs perhaps play in regular classroom courses.

Were the internet forum support groups as good as TAs? Did they even misguide which would be a terrible thing for a student? Don't know for sure. But it seems it was not as if the students did not get any support at all besides the instructors' lectures.
...
I would say face-to-face teaching is preferable instead of essential. For those guys who are not in a position to avail of such a facility, these kind of courses will be very attractive.

I think it is a fairly simple issue of supply and demand, and ever rising costs of brick-and-mortar education. Perhaps we will have both models, brick-and-mortar and Internet-based co-existing, with students making a choice based on how much money they have, the quality of education they are willing to settle with, geographical and linguistic issues etc.

Maybe it will be like Proprietary and priced software, and free and open source software. Both models co-exist today.

Some input (edited) from a friend whose kin took Prof. Thrun's AI (Free on internet) course

Agree -- from personal experience (that students had to put in sustained effort for the course). Also, there was a fair amount of gap between the lecture & the assignments/test; much akin to the class room mode. So, yes. The class was definitely not spoon-feeding. I can vouch for the fact that the class needed a fair amount of thinking & hard work -- IMO, the class is worth the effort.
...
The support fora (for student queries) were not really general fora. They are like discussion boards for the class (need to be registered for the course), much akin to the current day scenario in the US universities. Lot of times, every live class has an online discussion board where students ask questions regarding the class, lectures, assignments & tests. TAs/professors handle some questions on the fora, and some others that need more involved discussion in the class.

While the online classes from Udacity did not really address the latter part completely, the professors held virtual 'office hours' where they fielded some of the harder questions from the forums. Not exactly like in the universities, but they did make a fair attempt to bridge most gaps.
...
As I had predicted in an earlier email, Udacity now offers accreditation for a fee.

Even at a public school like --- where I work, the annual fee for foreign students is about 56,000 USD. This model of free online courses with (hopefully) a small fee is perfect for us. Who knows, in these tough economic times, perhaps some tech savvy parents might encourage their (hopefully dedicated) kids to take a stab at online education & accreditation for a much, much, much smaller fee that 56,000 USD! (even for citizens from within the state, the fee is about 30,000 USD). Perhaps the cost of community college at the quality of Stanford/Harvard/MIT. Thats a bold dream -- one that seems perfectly poised to come true very soon.
...
An aside: I suppose you are aware that not all the 160,000 students completed the (AI) course; just about 23,000 did, IIRC. But still, even that is overwhelming -- when compared to the largest classes I have heard of -- about 200 students tops.
...
I suppose you are aware that MITx is now called edX -- a joint venture with Harvard!! [Ravi: I did not know it then as I had switched off these topics for a few months. Thanks to the friend I visited the site and saw the edX announcement video.]

NYT article on Massively Open Online Courses

Read a very interesting article from the New York Times News Service, printed, a few days back, in The Hindu on Massively Open Online Courses. Some points from it, including some quoted phrase(s), are given below.

1. It gives a current-state picture of Coursera as growing in university partners as well as courses offered with over 100 MOOC courses to be offered this fall.
2. It states that MOOCs were unknown till last year, but now are "likely to be a game-changer, opening higher education to hundreds of millions of people."
3. Revenue stream does not seem to be an immediate concern.
4. A Prof. was thrilled with thousands of downloads of his videos.
5. Online cheating and quality of peer-to-peer grading are concerns. Paid examinations conducted by global education companies may help.
6. Concern that MOOCs may be a danger to universities - however, one Prof. thinks that MOOCs will provide diplomas (informal type of certifications) with in-class universities provide degrees.

Some additional thoughts of mine

Profs. happiness with large downloads of their teaching videos is a fascinating human aspect of online learning. I mean, the Profs. are the main guys. They need to be interested. The thrill of teaching orders of magnitude more students seems to be almost an irresistible draw. The 'altruistic' teacher sharing his knowledge with the world-community - fascinating human aspect of online learning for me.

There are some serious concerns. A high-dropout rate has been reported by some sources - but even if 10% of 160,000 passed out, it is still a great number!

Rampant cheating is reported by one source. I am not surprised by that. However I feel that once exam services like what Udacity is reported to have tied up with Pearson Education comes into play, then it may work out. Those students who cheat with assignments during the course will know that their ignorance will make them trip up on the paid exam - so they will make the effort to learn.

Eventually they will have to find a revenue stream. But I think the Internet has lots of time-proven business models where the customer gets his service for free but his usage of the service is converted to some gain. E.g. gmail usage resulting in advertising money for Google. The impression that I am getting is that venture capital is pushing this model first into respectable delivery. Once that is done, they will find a way to make money from it. Due to the scale, like for iPhone/iPod Touch apps, the price can be really affordable. I mean, I have bought iPod Touch apps. for 1$, 2$, 10$ equivalent - unthinkable for PC business model. Once the online education guys want to monetize they may make huge money from millions of Indians, Chinese etc. besides Western market students who may be very willing to shell out, say, 20$ equivalent per course/subject. 

I think there is tremendous potential for new education providers who provide a college experience but use mainly MOOC teaching and combine it with good examination services like what Pearson is supposed to have got into. Such education providers will save costs of most faculty - they may need some administrative kind of faculty - and pass on the saved cost to students. These education providers may get accredited by international academic accreditation agencies and also tie-up with professional certification organizations like the IEEE CSDP. That would make for a pretty strong software development professional education provider.

Of course, elite education may still mainly be done in residential education environments. Face to face interaction with inspired & knowledgeable faculty along with shoulder to shoulder interaction with sharp peers is usually vital for excellent learning. But for those who are not that elite or have money problems or geographical location problems, I feel MOOCs based education providers may be a very attractive education solution which could also become quite scalable.

It is less than a year since I read about Stanford's AI class offering. In such a short period of time so much change in higher education circles in the US is amazing. I would not have believed it if somebody had told me that all this is going to happen in so short a time, a year ago.

Of course, it is still a 'Wild West' area. But, IMHO, surely something substantial will emerge once things settle down - the exact "settled down" form or forms it will take are not clear now but that is not so big an issue. The main thing is the movement. I remember the free email wave when I came to know of it - hotmail was the first IFIRC - but I got onto the yahoo bandwagon. It was unbelievable then. After gmail it has become almost a part of modern life, a given.

Is MOOC going to become a given like free email?

Wall Street Journal article on "Higher Education's Online Revolution"

A few weeks ago, I read this Wall Street Journal article, "Higher Education's Online Revolution"  by Mr. John E. Chubb, Distinguished Visiting Fellow at Stanford University's Hoover institution and member of its K-12 education task force, and Mr. Terry M. Moe, Professor of political science at Stanford and a senior Fellow at Hoover. BTW both Mr. Chubb and Mr. Moe have a PhD in Political Science (wonder why Wall Street Journal does not prefix Dr. against their name?).

I feel this is a vital article on online education (dated May 30th 2012) as it appears in a leading and very respected media outlet and is written by people with very strong education/academic credentials and associated with one of the well known technology and education innovation universities in the world, Stanford University. I have given some comments of mine as points below. They may make better sense when read in conjunction with the Wall Street Journal article. A few phrases and sentences from the article are quoted below.

1. Initially when I viewed the edX announcement video my feeling was are they overdoing it? Yes, it was Presidents of MIT and Harvard, and top-notch Profs from MIT and Harvard on the panel with, IFIRC, Anant Agarwal referring to the printing-press (and others not rebutting it). But still, "biggest change in education since the printing press"? That's saying one heck of a lot. I mean I recall OCW made quite a splash a decade ago in the newspapers (I was in industry then but still came to know of it as it got so much publicity). But it did not really 'revolutionize' higher education. But as I thought about it and factored in what I have read (and shared some over email) about Stanford's AI class I felt that new factors like high-cost of higher education, student-debt-trap, powerful collaboration tools (students among themselves, students with Prof./teacher and/or Teaching Assistants etc.), *massive* affordable Internet penetration worldwide now as against a decade ago (in Puttaparthi, AP, India - a remote/rural town - where I live, a decade ago we did not not even have broadband access to Internet!), video medium teaching techniques having evolved (Sal Khan Style Videos - ksv's are what edX's Anant Agarwal said edX was using as a model for some aspects of these teaching techniques as Khan Academy of Sal Khan has become the poster-child of this new avatar of online education), grading and assessment (OCW may not have had that) and, last but certainly not the least, certification. MIT & Harvard are talking of great educational research possibilities from the data they will be collecting on the courses they run. So, maybe this time around, this new avatar of online education may really be a 'revolutionary' game-changer of education. I learned a new term, 'flipped classroom' from the edX site - I had read about this approach but did not know the jargon of 'flipped classroom'. I think education seems to be in for quite some 'flipping' in the coming years.


2. How will they make money to sustain themselves if they give everything away for free? That's the big question on everybody's minds, I think.


3. "One Nobel laureate can literally teach a million students, and for a very reasonable tuition price." That's some line.


4. I guess that's what IT did to lot of industries (substitution of technology for labor). I have seen it happen in India and the world, while in industry for nearly two decades, from a pretty 'insider' kind of view, and, later in India, while out of industry but as a user of services in a rural Indian town (banking, railway reservation, post etc.) for nearly a decade.


5. Having been involved in academic classroom/lab. teaching for nearly a decade and fair amount of software industry classroom/lab. teaching/training before that, I find these points (about advantages of online education technology) to be very impressive advantages over traditional classroom teaching model.


6. I think this ("elite-caliber education to the masses") is the real winner point due to which institutions like MIT, Harvard, Stanford with backing from grant-giving foundations and perhaps political support from the US government may be pushing online education even if this is going to bring lot of disruption in the education system. Orders of magnitude more students benefit - so the pain of disruption/change is worth going through, perhaps. Another point is that in the competitive, meritocratic, capitalistic USA if MIT, Harvard, Stanford, Princeton etc. don't then Udacity (for-profit company providing only online education and funded by venture capital) will. And Udacity may then get so ahead of the race that it may become the Google of higher education! Perhaps therefore MIT, Harvard, Stanford etc. don't have a choice if they want to retain their leadership in higher education. This is one of the fantastic benefits of the competitive market-driven model as against a bureaucratic government-regulated model like Indian academia which may choose or even be forced (due to community/political pressure) to just kill any disruptive change as it will be treated as a 'disturbance to existing system'.


7. Interesting about skeptics worrying about "college experience" being destroyed by online learning and how the authors disagree with the skeptics. 'Flipped classroom' seems to be what they are talking about.


8. "College X" mixing online courses with local Professor taught courses is a cool idea!


9. That's a significant business number (doubling their share)! Hmmm. Seems to be making a strong case for for-profit education models "embracing technology" aggressively and profiting from it. I plan to read about the University of Phoenix foray into this area.


10. Apple or Microsoft can do 'wonders' with such content if made available to them!!! The world has seen and experienced what they can do! Of course, the big concern will be whether they will lock away this content into some 'walled garden' that only they can profit from and, worse, only they can deliver.


11. So humbly and clearly said ("The MITs and Harvards still don't really know what they are doing"). That's what even the edX announcement video conveys. One of the news reports talks of Anant Agarwal mentioning this area as a 'Wild West' area. I really appreciate this candour.


12. I think the last paragraph of the article, quoted below, is worthy of real hard consideration by education policy makers, education administrators and educators. Perhaps they tell us of what is in store in this area in the coming years, worldwide - that's what the Internet is - a global phenomenon, and coming years not coming decades. "But like countless industries before it, higher education will be transformed by technology—and for the better. Elite players and upstarts, not-for-profits and for-profits, will compete for students, government funds and investment in pursuit of the future blend of service that works for their respective institutions and for the students each aims to serve."

Google's Hybrid Research + Development Model

Last Updated on July 23rd 2012

Here is a very interesting recent paper from ACM flagship, CACM, July 2012 on "Google’s Hybrid Approach to Research". The page also has an embedded video of around 5 minutes where the authors give their views - worth watching, IMHO.

Some important points of the paper from my perspective:

1. "Research results come not only from universities, but also from companies, both large and small. The way research results are disseminated is also evolving and the peer-reviewed paper is under threat as the dominant dissemination method. Open source releases, standards specifications, data releases, and novel commercial systems that set new standards upon which others then build are increasingly important."


2. Google does research + development together with R&D (or R&E) teams usually writing production or near-production code from day one! [Ravi: That's awesome!]


3. "Typically, a single team iteratively explores fundamental research ideas, develops and maintains the software, and helps operate the resulting Google services— all driven by real-world experience and concrete data."


4. Google's CS Research follows "Hybrid Research Model" where research teams are encouraged to have the right balance between research and engineering activities. The right balance can vary greatly. [Ravi: That's quite fuzzy. But the message that they give importance to engineering/software development as a vital part of its research model comes through clearly.]


5. The paper has some information about Google's research efforts e.g. Google Translate, Google File System.


6. Google publishes research work in academic publications "at increasing rates (from 13 papers published in 2003, to 130 in 2006, to 279 in 2011)."


7. Google feels that academic publications are "by no means the only mechanism for knowledge dissemination: Googlers have led the creation of over 1,000 open source projects, contributed to various standards (for example, as editor of HTML5), and produced hundreds of public APIs for accessing our services."


8. Google has "chosen to organize computer science research differently at Google by maximally connecting research and development. This yields not only innovative research results and new technologies, but also valuable new capabilities for the company."

--- end Google Hybrid Research + Development Model Paper - my perspective points ---

Some additional points regarding Google from a friend

Please note that some of the points mentioned below may have been covered in the above paper itself.

1. Google seems to care about applied research, not pure — a Googler needs to be able to articulate why his/her research will substantially benefit millions of users.
2. Google research is short-to-medium term: a few years at most.
3. Google tries to break research down into a number of intermediate deliverables that each have commercial value.
4. A research project may impact users, or it may advance theoretical knowledge, or ideally both.
5. They don't build elaborate research prototypes. Focus is on real systems with real data and production-quality code. So research is often a component of a production-oriented larger project rather than being a separate research project in itself.

Necessary components of the Google model:

1. Smart engineers
2. Ability for individuals, or entire teams, to transition to or from the research organization
3. Distributed computing infrastructure that lets a small team use tens of thousands of servers, which enables large-scale experiments
4. A billion users
5. Lots of money

The friend reiterated that publishing (academic) papers is only one way to distribute knowledge.

Indian CS Academic Research vs. Google Hybrid CS Research

I (Ravi) find Google's hybrid approach to be very interesting as it is in very great contrast to what I have seen in Indian academic CS research. Very often, the craze is to produce a 'paper' and the 'research' stops there - I have not come across many instances of research efforts from Indian CS or IT academia which went beyond 'paper' to get translated into semi-real-life stuff which can then be handed over to interested software companies for real-life implementation. Maybe I am not that well informed. If Indian CS academia does have a hybrid research + development model then perhaps such models should be given publicity. Anyway I got put off Indian academic CS research due to this 'paper' publication limited goal mind-set - I took a decision to steer away from such 'paper' production oriented research.

The big problem with this kind of 'paper' production research is that, most times, it is out of touch with real-world-software. So many academic conferences are around that getting such an out-of-touch-with-real-world-software paper published is no big deal. Now I am not saying that such  papers have false information in it - No, not at all. They are certainly valid within a very-small-prototype world. The question is whether the approach used in the very-small-prototype world makes sense to be considered in real-world-software. Most readers of such papers would have such questions and may just note the approach used in the paper. Instead, if academic CS research is able to combine research with development in some small way then academic CS papers would have a lot more value. Just imagine such academic papers having a reference to its open source software code + data download link. A reader who is interested in the approach can just download the software + data, and if he likes it, can even consider to do further research on top of this software + data.

I think Indian CS academia should carefully study Google's Hybrid Research + Engineering model and see if it can pick up certain practices of Google and adapt it for use in Indian CS academia.