The Gillmor Gang talked with FriendFeed co-founder Bret Taylor, who’s moved to Facebook as Director of Products. The discussion centers around Tornado, the new open sourcing of key elements of the FriendFeed realtime back end. Full transcript from SimulScribe below:
Mr. STEVE GILLMOR: Hi, this is Steve Gillmor. This is the Gillmor Gang for Thursday, some day and some month for some year. It’s the day after the Beatles Day and to my delight I walked in to Starbucks today for a meeting and Beatles music was all over the stereo. So, I immediately thought, well, that’s interesting that they’re playing that. But of course, they’re not going to have any records. Whoops! They were selling Sgt. Pepper, the While Album, the Help! Album and Rubber Soul at the counter at Starbucks. So, we’re in a new thing. This guy, I think you recognize. What’s your name again?
Mr. ROBERT SCOBLE: Loren Feldman.
(Soundbite of laughter)
Mr. SCOBLE: I’ve got a puppet.
(Soundbite of laughter)
Mr. GILLMOR: Yeah, so that’s Robert Scoble and we have another regular, Kevin Marks. Welcome, Kevin.
Mr. KEVIN MARKS: Hello. Hi there.
Mr. GILLMOR: We may be joined by some other folks on Calliflower and if they get there, rather than do this – so that we hear that music, if you call in to Calliflower please let us know through Twitter or whatever. And our special guest today is Bret Taylor.
Mr. BRET TAYLOR: Hi.
Mr. GILLMOR: Hi. Thanks a lot for showing up. We got the notice this morning via Kevin Marks’ tweet or FriendFeed of an important announcement, right, what I believe is an important announcement from you regarding – well, you tell. What’s Tornado?
Mr. TAYLOR: Tornado is an open source version of the web server we built at FriendFeed. It currently serves all of FriendFeed’s traffic. Advertently, we built the entire site in Python which is a pretty popular language but it hasn’t been a huge – the Python community hasn’t converged on sort of single web development framework. It’s really taken off sort of like, there’s no equivalent to Ruby on Rails in the Python community. Probably, the closest thing is a project called Django. And so we started developing FriendFeed and we are looking at the large number of projects or doing web development in Python. None of them really met a lot of our technical needs so we ended up building our own from scratch. And over the course of developing FriendFeed, we started adding more and more real time features to the site and as a consequence, our web serving infrastructure changed quite a bit to very honed for these types of real time services. And so, we had realized we’ve created something that we thought would be probably useful at FriendFeed and we had been planning an open source in it, just too busy. I know when we got to FaceBook and we talked to the people here about the technology we developed and the possibility of open-sourcing it. They were extremely supportive and excited to do it. And so, in the past few weeks since we’ve arrived, we’ve just been spending time kind of re-factoring the code to make it more useful for other developers and we really (unintelligible). So, essentially, Tornado is a web server and it’s written in Python and it’s ideal for Python developers. And it’s particularly unique in the sense that it can support tens of thousands of simultaneous connections to the server. And the reason that that’s important is real time web services like FriendFeed uses technology called long polling to enable a real time user interface with the end sever with a page of html and to enable long polling on a site like FriendFeed it means that every active user of the site maintains an open connection to the server waiting for new updates to be received. And to maintain an active connection for every active user of the site turns out to be a technical challenge. It’s really not addressed well by existing frameworks but it’s addressed pretty directly by Tornado.
Mr. GILLMOR: Robert Scoble, you have some questions?
Mr. SCOBLE: No, other than, do you think that this will be used inside FaceBook’s existing service in some way?
Mr. TAYLOR: You know, it’s a little hard to tell at this point because FaceBook’s a very large product. It has a lot of bells and whistles and I think, you know, it’s a very non-trivial task to incorporate a new piece of core infrastructure like this. I doubt it will be used in the short term. That said there’s a lot of analogous services that FaceBook does. It’s all similar technical problems. For example, the FaceBook chat feature that you probably have seen on the bottom right part of your screen lets you do real time chat with your friends. That uses similar types of real time technologies to enable chat. And we’ve been talking a lot to the chat team about how they solve some of the interesting engineering problems that they face and how we solve them. And we’re doing a lot of sort of engineering exchange at this point, but, it’s hard to tell how it will be used concretely here in the future.
Mr. GILLMOR: Kevin Marks, what do you think? Do you think this will be adopted quickly?
Mr. MARKS: I had a quick look at it, it looks very interesting. It reminded me a little bit about not finding what you want in existing frameworks. So, I was interested to know what you thought of Twisted and how much that met your needs, because some of the concepts seem similar in here.
Mr. TAYLOR: Yeah, Twisted has this unique property where – there’s a lot of people contributing content to it but not as many large websites actually using it. And that particular attribute became abundantly clear when we tried to use it, which we did when we first created FriendFeed. A lot of the protocols implemented within Twisted are extremely buggy. There’s about three different implementations of HTD piece servers within the Twisted framework by default all of which are broken in different ways. And this is, I think, if you go and Internet surf for people who actually try to use it, that frustration is not, we’re not unique in our experience. So, Twisted has some interesting underlying concepts but it definitely just wasn’t, you could tell it wasn’t in wide use. So, the HDT piece server part of it wasn’t in wide use by like a real company. We’ve had much better success with the open source projects that were developed by a company that saw the specific real purpose. Just as an example memcached to you is developed by LiveJournal and it was serving tons and tons of traffic. And so when you use it, it was obvious that it was being in real world use because it worked. The obvious bugs, it didn’t break when you did something slightly outside of the bounds of normal behavior. Twisted, we sort of had the opposite experience. And so, it has a lot of similar concepts. Our hope is it twists because Tornado is actually in use on FriendFeed. It’s serving all of its traffic today, that hopefully, developers can use it right away and not encounter some of the same types of headaches that we’ve encountered with Twisted. Certainly, a lot of the goals are similar of the project.
Mr. GILLMOR: What about –
Mr. TAYLOR: I don’t think it’s distinct. Go ahead, I’m sorry.
Mr. GILLMOR: I’m sorry, I didn’t mean to interrupt except that I was interrupting twice.
Mr. TAYLOY: I just got to say the other thing that’s distinct is that we are focused on web developments not generic network programming. As a consequence, Twisted implements sort of an HTP server but it’s not optimized for high-level web development. On Tornado, it shifts with a lot of building blocks that you need to build like real websites, like secure cookie sign-in and templates and all of the things you would expect in the web development framework. It’s also a little more accessible at a higher level for programmers which I think is pretty important.
Mr. SCOBLE: What do you think the impact Bret is going to be on the other real time infrastructure plays that are going on PubSubHubbub and RSS Cloud? How do you think those systems will use Tornado?
Mr. TAYLOR: You know, I’m not sure, our hope is that if this can enable a lot of people building real time services just to make their user interfaces better, one of the examples that we shipped with the product is a chat room. And using that demo, you could build on top of it to make a real time messaging service and connect it with all these open protocols from FaceBook stream API to PubSubHubbub and hopefully we’ll enable a new class of apps just by making it a little easier to get off the ground.
Mr. GILLMOR: I’m just going to say to somebody who just was talking about coming in from, I believe, Africa and that their bandwidth is limited. There is, a few people who are already taking the audio of this feed and re-mastering it, an homage to the Beatles, by the way, and putting it on some server that some guy named Jack on FriendFeed has been doing this. So, there will be a version of this probably open-sourced shortly. And soon, we’ll be delivering the video version of it to YouTube so that you can download that and it is going to be made available for download as soon as I can figure out how to be able to use the RSS part of the feed. And so, following up on Bret on your comments about the use of Tornado, is the source code going to be a good example of how to address the API?
Mr. TAYLOR: Yes, so there’s a couple of things there. One is it is really a generic piece of infrastructure so it’s not tied to any particular API. It’s just a building block you can use to make any rote(ph) application. But I think most modern web applications interact with APIs as a part of their service so we have a built-in support for FaceBook Platform and FaceBook Connect, FriendFeed’s API, Twitter’s API and Google’s APIs just to make it a little easier to get off the ground particularly in areas of authentication. And so included with the product is a FaceBook Connects demo and another demo just demonstrating how to authenticate with Google OpenID. And so, we’re hoping to make it easier for people building FaceBook apps or just incorporating third party authentication in to their sites. It should be a little easier to get off the ground with Tornado.
Mr. GILLMOR: But what about the chat implementation? Basically taking, you had examples that you released with the API too –
Mr. TAYLOR: Yes.
Mr. GILLMOR: …that supported pretty much everything except the real time stuff. So, could you take the two pieces of code and cobble it together into something –
Mr. TAYLOR: Now that you bring it up, I’m kind of asked me to do that, Steve. It’s a good idea. Yes, it will be possible and we should have had that demo because it would have worked really well. And we will probably make that demo now that we just talked about it. It’s a great idea.
Mr. GILLMOR: Excellent.
Mr. SCOBLE: Bret, if you were on the Google Wave team, how will you be using this technology.
Mr. TAYLOR: (unintelligible) just like I mentioned FaceBook having a large, you know, complex product. I imagine, Wave is fairly well-developed at this point, so I imagine, you know. This is kind of a core piece of infrastructure so it’s to like remove your core piece of infrastructure and insert a new one without having it break your entire site. And so, I imagine that Wave is pretty far down the path of choosing their real time infrastructure. That said my very limited understanding of Wave is that it’s an open and federated protocol and perhaps Tornado could be leveraged for people federating with Wave to enable some real time features on top of it. Certainly, because it’s written in Python, I think it’s accessible to a very large number of developers and certainly a lot more of them – well Java has a lot of usage. I think it’s a lot faster than developed web apps and Python and so hopefully this will enable that type of functionality to a wider range of developers.
Mr. GILLMOR: Have the puzzle hopefully, probable integration of the FriendFeed API into FaceBook API going?
Mr. TAYLOR: We’re still figuring stuff out. I mean, you know, part of the first couple of weeks we were like figuring how to find our desks and figuring out what projects we were going to work on.
Mr. GILLMOR: Well, how’s that going?
Mr. TAYLOR: We actually do have desks now, (unintelligible) the whole thing. It’s pretty cool. We have learned how to connect to our e-mail systems or communicating field for now. But, yeah, I think it’s a little too early to tell how things are going. The great part is here and I think as I said when we first started (unintelligible) is that everyone here sort of shares our vision for real time communication and openness and so, what’s great is, I think the teams are really gelled in terms of philosophy but there’s a huge number of details just in terms of talking about how FaceBook’s platform is going to evolve and our existing API, how they relate. So, I really can’t give any specifics right now. It just gives me, how we really figure it out ourselves. We’re working on it.
Mr. GILLMOR: Go ahead, Robert.
Mr. SCOBLE: Since you had a fresh look at the code, was there anything that while you were going back through it that you said, man, that was ugly or damn, I wish we had done it differently?
Mr. TAYLOR: Yeah, it turns out like lifting up the Komodo you start wondering like, whew, do I really want someone seeing this code?
(Soundbite of laughter)
Mr. TAYLOR: So, we ended cleaning up our own code in the process of doing this. We sort of re-factored it. We also wanted to reduce dependencies in the code base. So, essentially you can use the modules that we’ve included as a part of Tornado, sort of an all in part basis. If you just like this one module we have produced but you don’t want to use the rest of Tornado, it should be very easy to do that. So, we did a lot of work to kind of clean up and re-factor the code base just with external developers in mind because it certainly was you know, internally focused previously.
Mr. GILLMOR: Kevin, I’d like to take this up, level and sort of broaden the questions here to include one of the things that you’re working a lot on in your new role at British Telecom which is the – and I’m going to let you pronounce it. But I think it adds up to activity streams?
Mr. MARKS: Right.
Mr. GILLMOR: So, the question that I have is, what’s going to happen when this kind of abstraction layer that’s being developed, the kind of thing FaceBook is doing with FriendFeed which is basically absorbing it into some degree of interoperability with what they have done or what they are about to do. Robert and I were over at FaceBook a couple of days ago talking with Chris Cox and some of the people on the mobile team. One of the questions that I had related to, if you developed the notion that is called everyone, the everyone status inside of FaceBook, you essentially have a competitive stream of status updates that are public in nature. They could be expressed in a similar context to the Twitter streams. And that seems to sort of vector into the same kind of activity streams discussion that you’re involved in. Do you agree with that?
Mr. MARKS: Yes, broadly. The activity streams staff is mostly about trying to converge the way we express these different activities from different places. So, in some ways it’s doing this kind of thing that FriendFeed already did internally where they normalized all the activities from different places into a common format. We’re trying to – anyway, there’s interoperable. So, the group we’ve got on that from FaceBook and MySpace and Google and Yahoo and a bunch of others who are trying to converge their stuff. So, from what I see, the Tornado stuff looks fairly complementary to that as it’s handling a different piece of the problem. It’s handling the web serving and polling stuff. It’s not dealing directly with the data structure of the streams and the things that are flowing through it. Is that fair, Bret?
Mr. TAYLOR: Yeah, it’s sort of lower level essentially. Hopefully, if you wanted to develop a real time system using activity streams, this would be a building block but it is at a lower level. So, essentially, it’s a piece of infrastructure more than it is a standard.
Mr. MARKS: As compared to using the FriendFeed API which is already parsing a lot of stuff and integrating it for you, but then, normalizing it into FriendFeed’s interpretation of it.
Mr. TAYLOR: Yeah.
Mr. MARKS: Yeah. So –
Mr. GILLMOR: Go ahead. And I just wanted to welcome Mark Canter on the phone from somewhere in some part of the country that I’ve never been in.
(Soundbite of laughter)
Mr. MARC CANTER: Yo, yo, yo.
Mr. GILLMOR: You’re at the Rock And Roll Hall of Fame, right?
Mr. CANTER: Yes, I’m heading out to the airport and I wanted to congratulate all the FriendFeed people who were really now faithful people. And I want to congratulate all the Google people who are really VP people who’ve gone back to his British roots. And we totally love people from Texas who pay for radio shows.
Mr. GILLMOR: Yes, we do. Rackspace rocks. So, I didn’t mean to interrupt, Kevin. So, let’s get back to Kevin. And join in if you want to, Mark. Just no rolling of buds because we don’t have you on video.
Mr. MARKS: So, from what I see of Tornado it’s an encouraging alternative to other frameworks that we’ve got because being able to do long polling things like that has always been a bit fiddly with the ordinary web-based frameworks. I notice, Bret, that you have an integration with App Engine there. I haven’t a chance to dig through that yet. Have you been able to get the long polling pieces working there or not? Or are you still constrained by the App Engine timeouts?
Mr. TAYLOR: Yeah, we’re not – essentially, explicitly, it does not support a lot of the long polling in real time features. But we wanted to support WSGI which is the Python web serving standard to, so people could deploy at least simple apps on App Engine. But if you want to use some of the nonblocking and long polling features, it just won’t work in App Engine right now. Unfortunately, the standard on which Python web serving is currently based called WSGI, doesn’t support a lot of modern web serving techniques. It’s a pretty well-known problem in the Python community. It’s just an older standard. And if your Python program is beyond the complexity of a simple CGI script, WSGI really doesn’t serve you well. And so we added support for it but only a subset of the features that we support in Tornado are available if you use App Engine or WSGI.
Mr. MARKS: So, to host it properly, you need a server that will let you do role Python hosting not just CGI hosting.
Mr. TAYLOR: That’s correct and we have in the documentation, a recommended set-up for people running in production. It’s pretty comparable to the way people run a (unintelligible) production. I can go to the technical details but it’s probably beyond the scope of this conversation.
Mr. MARKS: Well, certainly, we should just flag up people like rent space and the other hosting services to pay attention to because it’s something we’ve had trouble with before with hosting Python as you end up having to run your own box.
Mr. TAYLOR: Yeah, that’s true.
Mr. GILLMORE: Well, when you say that it’s beyond the scope, I’m already lost.
Mr. TAYLOR: Sorry, (unintelligible) at a very low level technical details. I apologize for that.
Mr. GILLMOR: I’m not asking you to apologize. I just (unintelligible). If there’s something that you think is beyond the scope of this conversation, I think you’re already well there. So, feel free to keep talking.
(Soundbite of laughter)
Mr. TAYLOR: No, it’s fine. I think that Mark’s point was very good though. The challenge with (unintelligible) web service is that they (unintelligible) real time connections like FriendFeed, current hosting providers, a lot of them won’t work properly for it. But a lot of new ones, like you know, for all of the modern hosting providers whether it’s something like your App Space or something more generic like EZ2, now a lot of them are going towards virtual machines where you can run your own custom set-up and it should work pretty well. It just depends on how flexible your hosting provider is.
Mr. GILLMOR: Robert.
Mr. SCOBLE: Can you tell us a little bit about what FriendFeed was, what the infrastructure FriendFeed was hosted on?
Mr. TAYLOR: Yeah, sure. We actually built our own servers and we run in our data center that we leased space out. So, it’s pretty basic. We didn’t use any managed hosting though. We partly because we wanted to build our own servers, we keep – the work inside of data and FriendFeed as captured memory so we can serve a large number of server quest and we managed to do that at a very cost-effective basis by building our own computers.
Mr. SCOBLE: Right, very cool. Why didn’t you choose the cloud hosting solution? Why did you have to feel you have to build your own servers?
Mr. TAYLOR: Cost essentially. In particular, high memory machines are at a very high premium with most hosted and managed serving providers simply because a lot of them run lots of virtual machines on single physical machines. And so, memory is a premium. With our own machines memory prices are low enough. We were able to scale up our service at a lower cost simply because we were buying the memory directly.
Mr. SCOBLE: Yeah, how many machines were you hosting on?
Mr. TAYLOR: I don’t know (unintelligible) I had and probably that’s private too.
Mr. GILLMOR: Well you seem to be pretty open about – I’m surprised by the speed with which you were able to pull us together. It’s essentially two weeks. I mean, Paul was on vacation last week.
Mr. TAYLOR: Yeah, Paul is at (unintelligible) while I was doing all this.
(Soundbite of laughter)
Mr. TAYLOR: I came locked up. No, I’m just kidding. The good thing is, it was mainly the FaceBook support, they were recording, they started here about a week ago actually. And he’s leading FaceBook’s open source initiatives. And there is just really wide organizational support for openness. So when we sort of had this idea in our head and we came here. Not only, we didn’t think having any roadblocks. People that are here just chipped in and helped it along. It’s just really great to see how committed this organization is to openness and I think that’s the reason it happened so quickly.
Mr. SCOBLE: What are you hoping that other programmers are going to learn by looking through your source code?
Mr. TAYLOR: Hopefully, they’ll find problems with it and submit fixes. That’s one of the benefits of open source. The other thing is I really hope, I’m going to start working here at FaceBook on the FaceBook Platform. And one of the things I think a lot about is how FaceBook as an organization can make it easy for developers to create really great applications whether it’s on the FaceBook Platform or just fit in social services in general. I think that really benefits FaceBook and the FaceBook ecosystem of applications. And to me, my main hope – if I can look back a year from now and say, wow, look at all the great FaceBook apps that this piece of infrastructure enabled that wouldn’t have been possible before. That would be a really great thing. And so I’m just hoping that one of my favorite apps six months or a year from now is built on Tornado and I can just feel good that we helped those developers get those apps along. And that would be definitely our metric for success.
Mr. GILLMOR: Mike Canter, you are going to develop on this platform?
Mr. CANTER: Yes, absolutely. In fact, I was just been to class at Paul’s alma matter at Case Western and I had 25 students who are all chomping at the bit. I’ve got them working at Google Maps and Google Calendar APIs now and we’re also implementing some diesel(ph) trending so we can do distributed trending because we wouldn’t want all our social graph sitting on the FaceBook servers. We do want to bust open those data silos. But anytime we hear about an open source problem like this, also we’re going to be talking about this stuff in Amsterdam next week at a Digital City Summit where Vince (unintelligible) has been doing very similar work in Amsterdam on shared servers and open source servers for municipal services. So, have you guys talked about the waiting in for Laconica with Evan?
Mr. GILLMOR: Who are you asking?
Mr. CANTER: You know what I’m talking about?
Mr. TAYLOR: Yeah, I’m not sure I understood what you said.
Mr. GILLMOR: He’s talking about gate waiting(ph) into the open source project called Laconica or Identi.ca.
Mr. TAYLOR: Yeah, yeah. It seems like a really interesting idea. Right now, those services I believe are written at PHP so that’s from a technical perspective – this is written in Python so it’s not entirely obvious how applicable it would be. But that said, inter-operability would be something that would be very easy to do and as an open source project, I would expect someone would develop that which is great. The great part about an open source project like this is third parties can introduce support for any service and hopefully it will be in an interface that a lot of developers can leverage because they’re sharing the same underlying framework.
Mr. MARKS: Something that you did very nicely at FriendFeed was the ability to log in to multiple services so you could bind in through your FaceBook and Twitter and Google accounts and draw the stuff in. It seems like, I haven’t actually looked in the code yet, but it seems like some of that is in the Tornado framework as well. So, the basic model is not that the user has an e-mail address, the user has a bundle of different identities. Is that right?
Mr. TAYLOR: So, we actually didn’t abstract at that high of a level. We built, we packaged the code so that you can easily connect with FaceBook and Apps Google, Twitter, FaceBook and OpenID and (unintelligible). So, we have these building blocks in there and a couple of demos at how you can use those building blocks to have your users log in. But, we didn’t abstract a way user identity in general. We just made it easy to like implement the authentication but how you store and think about users is not included in the package. It’s up to the application.
Mr. MARKS: I see.
Mr. TAYLOR: We did that on purpose because it’s a lower level framework on purpose. Our problem with a lot of the existing frameworks we’re developing FriendFeed is there is often an impedance mismatch between the assumptions of how you run your site and how you actually want to run your site. And particularly with multiple identities and things like that, there’s a lot of applications semantics that you have to assume when implementing something like that. And we thought, just providing the building blocks would be infinite. It takes away 90% of the problem and how you want to store that in your own data base is really up to you as an application developer. So, certainly the building blocks are there for a system like the one that we built but it’s really up to the applications how they want to incorporate third party authentication like FaceBook and App into their apps.
Mr. MARKS: Because I mean, what I’ve seen with existing frameworks is they still have the e-mail password, send you an e-mail out, click to get out your spam folder registration model baked in. And one of the –
Mr. TAYLOR: Yeah, in fact if you visit the chat demo that is linked to from a variety, so from a blog post MyBlog, it logs you in via OpenID. So, another service. I mean all of our demos encourage using third party modules like FaceBook Connector, Google OpenID. But that said, we don’t dictate how identity works on your application because we feel like that that’s an application level decision. You know whether you want to let people log in through multiple methods, how you want to store that in your data base, how your user interface works. So, it certainly encourages that model but it is at a lower level than I think a lot of frameworks are implemented.
Mr. SCOBLE: Bret, FriendFeed was designed for a user base of what – one to five million people. What would you have to rewrite to make it work on FaceBook for 300 million people, 400 million people?
Mr. TAYLOR: A lot. I think that application performance is generally downed by architecture. What I mean is, is your data storage mechanism horizontally scalable? Meaning, can you scale it by simply adding more machines to your data center? And FaceBook is a very sophisticated piece of infrastructure. And it turns out that things like Tornado are a building block for large systems like that and it probably wouldn’t need to be changed much to accommodate lots of traffic, you just make more front end servers and you’re done. But the hard part of building scalable wide applications is the back end, the data storage. So, that’s something that we would have, if we were to actually, the stuff we built at FriendFeed it pales in comparison to the infrastructure engineers have built here simply because they built it for 300 million users, so, in it a very evolutionary manner obviously as a site group.
Mr. GILLMOR: But in the case of many of the, what I would call affinity groups on FaceBook, those are a lot smaller than the entire pool of users. People seem to work in work groups, in groupings according to interests, you know, et cetera. So, is the scalable issue all that different than what you already were dealing with?
Mr. TAYLOR: It is very different, yes. I think, actually, if any software developers are listening, there’s a FaceBook engineering blog you should check out. There are some really interesting blog posts about some of the problems FaceBook engineers have solved. They’re very unique scaling problems, given the number of users. There is a great post actually about the FaceBook chat service that one of the engineers did about scaling it. It turns out that even though people interact in smaller groups of people, the whole point of FaceBook is the global set of connections, you know. You can interact with anyone in the world that you happen to know. And so, you really can’t fragment it too much or it loses a lot of the value of the network and so is the consequence. It is a really significant engineering problem. But I think one of the most fun parts about coming here, not just as an engineer, I mean it’s so much more fun solving a problem for 300 million people because (unintelligible) it’s the most mundane product feature in the world. It’s a really interesting engineering problem and that’s a really fun thing. I mean, it’s just everything is sort of more of a challenge and more infrastructurally interesting.
Mr. GILLMOR: But you come from Google which had similar kinds of scale issues. So, are you just more comfortable at that kind of scale?
Mr. TAYLOR: Yeah, I think we’re comfortable with it. The interesting thing about FaceBook and the reason I think it’s a really fun set of engineering problems here is that it is kind of unique because of how personalized it is. Google has a lot of interest in infrastructure but the problems here are quite different in the sense that everyone viewing FaceBook is viewing a different thing because it’s personalized to them and their social network. And the infrastructure problems that are presented with that unique problem it turns out just being a different engineering problem than giving the same 10 search results for everyone who types in a particular query. I think that to me is one of the most interesting challenges here as an engineer.
Mr. SCOBLE: The Cassandra Project which was developed at FaceBook was just used by Digg. And it’s just starting to be used by Digg to manage its servers. Have you played around with that at all and though about how you would join some of these technologies into Cassandra?
Mr. TAYLOR: The good thing is that they’re very complementary. Cassandra is a data storage service and Tornado is a web server so you could very easily write a web application that uses Cassandra as a back end and Tornado as a front end. So they solve two distinct problems of building websites. And the good thing is, Cassandra when I mentioned horizontal scalability that’s sort of its focus. I mean, it was really, really need to see the (unintelligible) of that. I personally have looked into it in great detail though.
Mr. SCOBLE: So, now that you’re done with this, what are you going to do at FaceBook? What caught your eye already just being inside the FaceBook world?
Mr. TAYLOR: I’m going to be working on the FaceBook Platform. My specific role we’re still up but that’s where my passions lie. So, I’m very interested in – yeah, I always just sort of start up, you know, and our constant struggle is how do we get users using our product. And FaceBook’s Platform is like really unique in its ability to get applications users. There’s a large number of companies that built their businesses on the FaceBook Platform as a consequence. And so, to me that just solves one of the most fundamental problems in technology and for a web business as a whole. And so, I’m really excited about working on it. But since my third week here I’m still learning everyone’s names. I don’t know exactly what I’ll be doing yet.
Mr. GILLMOR: Just rewinding a little bit, how would you describe the impact of your exploration of real time technologies and how do you see that as progressing at FaceBook?
Mr. TAYLOR: I’ve always been of the opinion that real time is kind of for a long time, there are social products and there are non-social products and now as the web is evolving, almost every product on the web is becoming social. I think that’s what is so exciting about products like FaceBook Connect as you can take a website that previously wasn’t social and make it social. I think real time might have a similar type of ubiquity in the near term future meaning, sites that were previously very static are slowly but surely going to incorporate more and more real time features because I think people’s expectations are going to change. Just as they did with social products, they want to see a personalized view of the web when they browse it. I think people want to see a real time web now as their browser. They want to see what’s changing in real time. And so that when you’re in your web browser it doesn’t feel like you’re just reading the static page. You’re watching a real time system evolve in front of your eyes. That obviously doesn’t apply to every product but I do think it applies to a large number of products we use on a daily basis and I’m hoping that some of the work we’re doing and just some of the thought work that people in the community are doing will all influence that trend.
Mr. GILLMOR: What about- I don’t want to get into the fight aspect of these adoptions. Robert mentioned it earlier PubSubHubbub and RSS Cloud. Can these two technologies live side by side do you think?
Mr. TAYLOR: I think, in general, engineers are a little more practical than maybe all the people arguing on blogs who probably use both if both exist. So, what I want to do is fix the site that want to integrate with. I do think that there are some interesting scalability problems. We are the first site to actually integrate PubSubHubbub when Brett Slatkin and Brad announced it at the real tech and real time conference. Since then, we get millions and millions of PubSubHubbub pings a day on our service. And it was actually impacting our site reliability for a while. We had to sort of quarantine net traffic to the site of servers just so that it wouldn’t impact end users. I think it has a lot of promise but I do think that there’s a lot of important details that people are working out of all these discussions about how to make real time and pit polling versus pushing and all that, they all have different infrastructure requirements. So, the standard that we introduced to FriendFeed has gotten some of the option called SUP, Simple Update Protocol, instead of an alternative approach. I think it’s very typical at this stage of technology if there’s a lot of competing approaches because people are still kind of figuring out what will work best in the long term. And I think, you know, maybe it will take a couple of years but a couple of years from now, we’ll probably have a more cohesive picture what technologies work best and where. And this is a really healthy process in the meantime of people having arguments and engineers encountering these technical problems and discussing them openly. And it’s really just a natural process of engineers figuring out what works and what doesn’t.
Mr. SCOBLE: Jesse Stay in the FriendFeed chat room is asking, when are we going to see real time features that we’ve got and gotten used to in FriendFeed move over to FaceBook? And I guess that leads to a deeper question, have you talked to Mark Zuckerberg and the other people on the teams about their philosophy of real time and whether, why haven’t they gone to it more real time stream that flows down your screen?
Mr. TAYLOR: So, actually I haven’t been involved in a lot of those discussions here at FaceBook. I mean, I certainly have talked to the home page team a little bit. But I’ve been focusing a lot on just keeping FriendFeed people integrated and trying to figure out our roles here. I can say just from a personal standpoint, real time is very complex in the sense that it can be kind of overwhelming. I think I’ve heard a lot of people describe FriendFeed as a fire hose in the past. Perhaps some of you actually used those words. And so, you know, FaceBook is available to 350 million users and I think making real time accessible to that broader range of a user base is a really challenging product design problem. And so, as much as I want to see real time features integrated to FaceBook, I do think that FaceBook, we’re solving a much harder problem just because of the breadth of the user base there. And I think that I take that really seriously, but I don’t actually know of the details of their philosophy, just because I haven’t been focusing on it since I arrived here.
Mr. GILLMOR: And Kevin, do you think the activity stream of group who I’ve been discussing. I believe (unintelligible) has seen it called open FriendFeed kind of set. Do you think that we need that now?
Mr. MARKS: Oh, definitely. I think that the point of it is the kind of thing that Bret was just saying. We had a lot of people experimenting with how these bits fit together and booting(ph) saddle and solutions. And we’re at the point now where we got enough practice that’s working, that we know how to do things and we can start agreeing on standards on there. It’s often a mistake to come up with standards before you’ve got implementations. This is kind of like representing about Twisted where they boot a bunch of stuff that seem like a good idea but didn’t have any practical implementations. Whereas once you’ve built something, you’ve got a set of people who’re doing something very similar, you can then drawer out the common facts to say, OK, these are the pieces we standardize. And if you look at the sort of the open web standards that’ve come out over the past few years, it’s mostly being getting people together who built fairly similar things and converging them. If you look at the (unintelligible) for example, that was very much taking the best of the existing (unintelligible) protocols coming out with the common way of doing it. And similarly with portable contacts where we said, OK, we’ve got six contacts formats. They’re all basically the same thing. Let’s merge them into something that we can use and then have a straightforward transform between them. So the activity streams can be looked at in that way as well. To say there’s a whole bunch of different sites that are generating updates about the things we do on the web and off the web that have time stamps, that have people attached to them. And FriendFeed is a great existence proof of the value of converging different streams from different sources in one place. So they’ll see if we can find a way to express this in a way that we can easily manipulate them back and forth without having to have one site do the work of all that processing.
Mr. GILLMOR: So, in other words we throw away all the connectors in effect that FriendFeed laboriously coded and switched to? How does this work?
Mr. MARKS: Well, hopefully it means that hopefully we can use the connectors that FriendFeed built till we can map the output of the FriendFeed API into activity streams would be a sensible thing to do. But also it means we don’t have to build the after(ph) products of all the possible connectors. We can say, if there’s a lingua franca in the middle, we can have end connectors (unintelligible) and square connectors being built. And that best of value to say, oh, I have the inner walls to support Twitter format and FriendFeed format and FaceBook format and open social format and whatever else format. You have to then build something that translates everything into everything. We can have a standard thing in the middle, we can translate them through. We can as much more straightforward to extract the pieces you want or just build a new piece that you need to add.
Mr. GILLMOR: Bret, what do you think?
Mr. TAYLOR: Yeah, I’m actually really excited about all the activities here in standards. I mean, actually, it turns out FaceBook stream (unintelligible) uses the activity stream of the standard already. And so, you know, the challenge I think as Kevin said is making standards that are both practical and open. I think some of the standards have suffered on the practical aspect. And so as long as there’s actually engineers in the room who are actually building these products that are using the output of these systems, I think it’s destined for success. The ones that haven’t been successful are sort of a committee of people who don’t actually write code every day. And I think that those are the ones that end up with sort of a bad state. Just see that I have about a minute left. I’m sorry, I just looked at the clock and I have somewhere to be in, well, it’s actually at 1:15 but I can stay on for another minute or two.
Mr. SCOBLE: It looks like FaceBook just has been tripping an app reply feature that’s similar to Twitter’s app reply feature. What has kept them from matching some of the other features that FriendFeed implemented like the like feature. FaceBook has likes but I can’t see all my likes. I can’t go back in time and search. I can’t see all the likes that Steve Gillmor did. When do you think we’re going to see an advancement of the platform in that area?
Mr. TAYLOR: You know, I really don’t (unintelligible) to tell you. You have to ask me in a few weeks. It’s not so fair enough with the current state of things to answer your questions accurately. So, I really (unintelligible) the stuff, just kind of still tweaking here basically.
Mr. GILLMOR: Well, I just think in general, we’ve come a long way from what I used to think was a sort of a FaceBook disease where people who have been very, very open and clear suddenly had this kind of murky layer of – well, that’s an interesting idea and we’ll take that feedback back or you know, kind of approach. I think that it’s obvious, given the speed with which you’ve rolled out this open server and I think that the pressure that’s been brought to bear by groups including the activities stream group to keep FaceBook honest and keep the FriendFeed architecture available until you guys figure out what you need to.
Mr. TAYLOR: Yeah, that was the goal. I think the nice part is that we are entirely sure you know, exactly how our product is going to evolve yet, but, our hope is that this is a real statement to our commitment to be open and transparent about the process. And so I’m really, really happy that you saw that too.
Mr. GILLMOR: All right, I think this is a good note to wrap this on. This is Steve Gillmor. This has been the Gillmor Gang. I want to thank everybody who showed up including Mark Canter from somewhere in Ohio. I don’t know where you are Kevin but it’s close by.
Mr. MARKS: San Jose.
Mr. GILLMOR: Robert Scoble over my shoulder and Bret Taylor. Thanks so much and we’ll hold you to that. We’ll talk to you in a couple of weeks.
Mr. TAYLOR: Thank you.
Mr. GILLMOR: Thanks again. Bye-bye.