Re: Online RSS Reader Update

1

I tried Google Reader after the last time this subject came up and I don't think it's ready for prime time. It's slow, oh so slow, so very slow; I hate seeing that damn Erlenmeyer flask. And the left panel is often very wrong about how many posts have yet to be read. I'll stick with Bloglines, which doesn't update often enough but is better than it used to be.

Of course, if I didn't read my feeds from two different places, I wouldn't use a web-based reader. They all have issues.


Posted by: Sean Flame | Link to this comment | 10-23-06 10:48 AM
horizontal rule
2

Huh, it didn't seem too slow to me. I saw the flask for about one-two seconds for each new feed.


Posted by: ogged | Link to this comment | 10-23-06 10:53 AM
horizontal rule
3

I'm not sure if I've understood this correctly, but I think you can sync two GreatNews readers through bloglines. That obviates the need for a web reader, no?


Posted by: SomeCallMeTim | Link to this comment | 10-23-06 11:00 AM
horizontal rule
4

Uhh, I seem to keep up fine with Bloglines, if Unfogged Comment sections are any indication.
Correction:Hmm, I am usually a few comments behind in the pane, and open up the site for real-time commenting. I probably should open up notepad and do a study

I would be shocked it the delay was more than a half-hour, and surprised if over 15 minutes.


Posted by: bob mcmanus | Link to this comment | 10-23-06 11:03 AM
horizontal rule
5

I think you can sync two GreatNews readers through bloglines

Sorta kinda. If you have a bloglines account, you can subscribe to the bloglines feed in Great News. But you're still constrained by bloglines's update schedule.


Posted by: ogged | Link to this comment | 10-23-06 11:07 AM
horizontal rule
6

I read Unfogged the way God intended: by compulsively hitting refresh. You people may think you can reach enlightenment without putting some effort into it, but you're wrong.


Posted by: apostropher | Link to this comment | 10-23-06 11:12 AM
horizontal rule
7

I would be shocked it the delay was more than a half-hour, and surprised if over 15 minutes.

Thirty minutes might be fine for the bi-theoretical, mcmanus, but people on one side or the other need it more often than that. We have less to do with our time.


Posted by: SomeCallMeTim | Link to this comment | 10-23-06 11:14 AM
horizontal rule
8

2. Two seconds is too long; it gives the impression of sluggishness. I'm a demanding consumer.

But my biggest problem is that my dodgy workplace net connection (hooray for public universities!) seems to interfere with what I have to assume is wacky-cool Google-magic. Bloglines has no magic; it just works.


Posted by: Sean Flame | Link to this comment | 10-23-06 11:17 AM
horizontal rule
9

I read Unfogged the way God intended: by compulsively hitting refresh.

Yes, this is the only way to read Unfogged. I mean for other blogs.


Posted by: ogged | Link to this comment | 10-23-06 11:22 AM
horizontal rule
10

test


Posted by: ogged | Link to this comment | 10-23-06 3:32 PM
horizontal rule
11

icles.


Posted by: apostropher | Link to this comment | 10-23-06 3:33 PM
horizontal rule
12

We all do our part, apo.


Posted by: ogged | Link to this comment | 10-23-06 3:35 PM
horizontal rule
13

I found it amusing that pair.com posted a test message more or less simultaneously with my #11 there.


Posted by: apostropher | Link to this comment | 10-23-06 3:39 PM
horizontal rule
14

I missed that. But I'm sure I would have found it testiconormative, and not the least bit funny.


Posted by: ogged | Link to this comment | 10-23-06 3:43 PM
horizontal rule
15

I just decided that my individual resistance is going to mean nothing when google's servers become sentient and know everything about everyone else through gmail, google calendar, google chat, etc. So I got my friend to send me an invite and created [real].[name]1@gmail.com (simpler versions were taken). I also want [pseudonym]@gmail.com, just as an alias. I assume it isn't taken. Can I do this? How?


Posted by: washerdreyer | Link to this comment | 10-23-06 3:44 PM
horizontal rule
16

E-mail an invite to yourself.


Posted by: SomeCallMeTim | Link to this comment | 10-23-06 3:47 PM
horizontal rule
17

Quick, someone take w/d@gmail.com so you can sell it to him!


Posted by: ogged | Link to this comment | 10-23-06 3:48 PM
horizontal rule
18

This compulsive-reloading thing makes me cranky - because the technology could be better, and in fact is in areas that aren't based around web pages. Some blogs and comment sections feel like they'd be better modeled as newsgroups (blog author posts, commenters are all followups of one kind or another), others maybe more like a chat room or BBS.

(and there's some irony that this comment was delayed a couple of hours by the burps that blog software seems so prone to)


Posted by: Nathan Williams | Link to this comment | 10-23-06 4:44 PM
horizontal rule
19

blog software seems so prone to

It really hasn't scaled up very well.


Posted by: apostropher | Link to this comment | 10-23-06 4:53 PM
horizontal rule
20

Isn't it time for some "Web 2.0" commenting modules? Gmail and Yahoo! mail beta both refresh content without refreshing the screen, so why can't blog comment threads do the same thing? You'd just leave the comment window open, and new comments would fill in as they were posted. Seems like the technology exists.


Posted by: ogged | Link to this comment | 10-23-06 5:00 PM
horizontal rule
21

I wonder if the quality of comments would go down, and this would become even more like IM or IRC.


Posted by: ogged | Link to this comment | 10-23-06 5:01 PM
horizontal rule
22

Seems like someone worked on it, but I can't make sense of the page.


Posted by: ogged | Link to this comment | 10-23-06 5:03 PM
horizontal rule
23

That was easy. I thought there may be some way to do it within my first new account, instead of having two new accounts wherein one forwards to the other, but this option is fine as well.


Posted by: washerdreyer | Link to this comment | 10-23-06 5:03 PM
horizontal rule
24

Ah, here's the explanation. Coding wizards?


Posted by: ogged | Link to this comment | 10-23-06 5:05 PM
horizontal rule
25

Now I'm not sure that page does what I described.


Posted by: ogged | Link to this comment | 10-23-06 5:09 PM
horizontal rule
26

I'm pretty sure that the quality of comments would go down, ogged. IM software just seems so expectant.


Posted by: Jackmormon | Link to this comment | 10-23-06 5:11 PM
horizontal rule
27

Ok, nice first step here. But maybe this is a solution in search of a problem? Somebody stop me.


Posted by: ogged | Link to this comment | 10-23-06 5:26 PM
horizontal rule
28

20 and following: You could probably do it on the cheap with a bunch of iframes. The problem would be that you'd potentially be introducing a ton of load on your server, causing your ISP to get angry with you.

People who go catatonic when confronted by tech babble, skip the following: There's no callback mechanism in HTTP, so you'd be constantly polling the server -- it'd be just like hammering refresh all the time, but your browser would do it for you. There are experimental techniques for doing push technology over HTTP by keeping single connections open -- google on Comet AJAX HTTP -- but I believe the server end of things is not quite ready for prime time. It'll probably be a while before versions of Apache that support this make it into production use. If you were super 1337 haXX0r d00d 0gg3d, you could probably slap something together using Dojo's cometd server or the Twisted Python framework or something and cackle about being Web 2.1 compliant.


Posted by: Steve | Link to this comment | 10-23-06 5:52 PM
horizontal rule
29

I don't even use a reader. Your solution is at least five steps ahead of any problem I'm having. But if you find an elegant solution to my non-existent problem, I'm sure I'll come around since I'm the kind of person who thinks cameras? on cell phones? hmmph, that'll never take off.


Posted by: Jackmormon | Link to this comment | 10-23-06 5:53 PM
horizontal rule
30

Ogged: down, boy.


Posted by: Becks | Link to this comment | 10-23-06 5:56 PM
horizontal rule
31

Thanks, Steve. Thanks for killing my dreams.

So how does Gmail work? You can leave the gmail window open, and the page changes when new mail arrives (ditto Google Reader).


Posted by: ogged | Link to this comment | 10-23-06 5:56 PM
horizontal rule
32

Gmail works by pretty much hammering the server. Thing is, they have tens of thousands of them.


Posted by: mrh | Link to this comment | 10-23-06 6:25 PM
horizontal rule
33

Gmail is using Ajax with polling* (which is what it sounds like -- every twelve seconds or something, it's sending a request back to Google SKYNET asking if you have new mail), but Google has more money than Powerball Jesus and as much experience with running large-volume servers as any company on Earth, so they're rather more able to handle it than a lot of places. If you guys were paying for your own server at Pair, you might well be able to handle the load for Ajax-enabled comments -- I had assumed it was a classic shared host sort of deal, which would almost certainly result in nastygrams given the amount of traffic here.

* I believe Gtalk, on the other hand, is using some of the tricksiness I mentioned in 28. The problem is that web servers (or at least Apache, the most commonly used one and the one I have the most experience with) generally have overhead for each connection they're making. When you're dealing with a gajillion Unfogged readers waiting breathlessly to hear about Tia's Newsies outfit, each with an Ajax process sending a request to the server every fifteen seconds, that creates serious stress on a server. The better way to do it is to never hang up the phone, as it were, and keep spitting out new information as it becomes available, thus avoiding all the overhead. The problem is that neither HTTP nor the current generation of servers is really designed for this, but you can fake it if you're willing to risk damaging your flux capacitor, etc.


Posted by: Steve | Link to this comment | 10-23-06 6:34 PM
horizontal rule
34

Pwned by 32, but I at least got to mention Tia's Newsies outfit.


Posted by: Steve | Link to this comment | 10-23-06 6:39 PM
horizontal rule
35

Yeah, we're on a dedicated server, but we're ditching it. First, because it sucks, second, because the new server plan promises to work at least as well for a fraction of the cost. We'll see.

Just musing now, but given how often some portion of the readers here actually refresh comment boxes, I wonder if we really would increase the load on the server by giving up the refresh and just polling for new data every 60 seconds. When we refresh the comments now, doesn't the server have to actually serve up all the comments each time? Whereas, if you just poll every minute, presumably there will be times when you don't have to serve up any new content, and (maybe, if I'm understanding correctly), when you do serve it up, it would be just new content, rather than the whole thread, which can be a lot of data.


Posted by: ogged | Link to this comment | 10-23-06 6:43 PM
horizontal rule
36

Is it less work for the server if we view the comments in an RSS reader?


Posted by: SomeCallMeTim | Link to this comment | 10-23-06 6:46 PM
horizontal rule
37

36 - Probably slightly. Well-behaved RSS readers use Conditional GET, which means Unfogged will spit back a standard error code (304, "Not Modified") if there's no change since the last time you requested the RSS feed.


Posted by: Steve | Link to this comment | 10-23-06 6:55 PM
horizontal rule
38

When we refresh the comments now, doesn't the server have to actually serve up all the comments each time? Whereas, if you just poll every minute, presumably there will be times when you don't have to serve up any new content, and (maybe, if I'm understanding correctly), when you do serve it up, it would be just new content, rather than the whole thread, which can be a lot of data.

Congratulations, you've just reinvented the concept of the caching browser.


Posted by: Josh | Link to this comment | 10-23-06 6:59 PM
horizontal rule
39

35 -- you're right that the client could poll, ask if there's new data, and only download if there is some; but for the server to only serve the new data wouldn't really work with a browser client. It would work with an RSS feed but I guess that's not what you want. With browsers you need to serve the whole page or nothing. (Or have an ultra fancy dedicated Unfogged client app like GMail -- that would be fun to write.)


Posted by: Clownæsthesiologist | Link to this comment | 10-23-06 7:16 PM
horizontal rule
40

Clowny, you could write a quick script to return new-posts-only via JSON or something like that, then use JavaScript to slap it into the page. It's doable, just likely to be a little crufty and resource intensive.


Posted by: Steve | Link to this comment | 10-23-06 7:22 PM
horizontal rule
41

May I ask a question of everyone here? Sorry if this is out of place.

I don't know much about these newfangled feeds, but when I search for my URL on bloglines, for example, it shows five feeds. None of which have updated since the end of August 2006, and three of them not since August 2005. The only feed that seems to work is the rss.xml one but that one does not appear on feedreader searches. I do not understand this...please advise.


Posted by: Stroll | Link to this comment | 10-23-06 7:34 PM
horizontal rule
42

Not quite sure what you're asking, Stroll. The feed linked from your front page seems to be current.


Posted by: ogged | Link to this comment | 10-23-06 7:42 PM
horizontal rule
43

40 -- that's what I meant by a fancy dedicated Unfogged client app. Seems to me like a huge amount of work to develop and maintain. But, I haven't done a lot of (or indeed any) Java development, I might be overestimating the effort involved.


Posted by: Clownæsthesiologist | Link to this comment | 10-23-06 7:44 PM
horizontal rule
44

43 - Ah, fortunately there are a ton of tools out there that speed the process. Someone who's done one before could do write an in-browser one using an off-the-shelf free JavaScript library (Dojo, Prototype, Rico) and PHP or another scripting language for returning the results, probably in a matter of a few days. Not sure how easy it would be to do in Perl and integrated with Movable Type directly, though.


Posted by: Steve | Link to this comment | 10-23-06 7:55 PM
horizontal rule
45

Screw that, I'm thinking we establish a completely new transfer protocol UFTP native to port 90210, and build client and server from the ground up in C.


Posted by: Clownæsthesiologist | Link to this comment | 10-23-06 8:04 PM
horizontal rule
46

Ack -- just realized UFTP already denotes UDP-based FTP. But nobody ever uses that, I think we can colonize the acronym.


Posted by: | Link to this comment | 10-23-06 8:05 PM
horizontal rule
47

O-or, maybe UOTP -- UnfOgged Transfer Protocol.


Posted by: Clownæsthesiologist | Link to this comment | 10-23-06 8:06 PM
horizontal rule
48

My gripes here relates a lot to the use of a web browser as a dumb-ish display engine. 39/40 is the closest to what I had in mind... AJAX can let the page update itself (really, it's appalling that this constitutes an *advance* in technology...), and while it would probably still have to poll the server (per 28), once it knew it had N comments, it could just ask for "any comments on thread X past N?" which should be less load than "give me all comments on thread X". Since this is, more or less, what a newsreader does to a news server, it seems like it ought to be a solved problem.


Posted by: Nathan Williams | Link to this comment | 10-24-06 1:35 PM
horizontal rule
49

Wow, I think the Unfogged program-related community is hitting untold heights (or, probably, depths) of nerdiness here. How is that even possible?

That said, the goggle-tanned Iranian seems to have become my guru of computer programs, since I useOpenOffice and, now, GreatNews. All hail the wise man from Iran, the vertiable Xerxes of blogdom.


Posted by: peter snees | Link to this comment | 10-24-06 2:06 PM
horizontal rule
50

Ah, geeks! While I have you here, a question:

I use Thunderbird for RSS -- one instance at work and one at home, which means I'm constantly scrolling through the same damn posts twice, which is a pain in the ass.

However! One thing has kept me in Tbird, that I can't find replicated in any online reader, namely: Tbird doesn't display the content of the RSS (xml?) file. Instead, it displays the content of the referring page. In other words, it always displays the whole post. That means I don't get stuck reading the first two lines in an RSS reader and then clicking over to the site to read the rest. I just can't stand that. As it is now, I do about 90% of my total web media reading inside Tbird, without ever clicking out. It's too good to give up.

Does anyone know of an online reader that will do this? Or -- better yet -- a way to sync OPML files between two instances of Tbird?


Posted by: Realish | Link to this comment | 10-24-06 10:33 PM
horizontal rule
51

Funny, that behavior is one of the reasons I don't use Thunderbird for RSS. I just did a little googling and checking at the mozilla site, but I don't see anything to sync what you've read across multiple computers in Tbird. And I can't think offhand of an online reader that does that either, and I've just recently looked at a bunch of them. Sorry. (Though it sounds like you're pretty satisfied, overall.)


Posted by: ogged | Link to this comment | 10-24-06 10:44 PM
horizontal rule
52

does that

By which I mean "shows the page, rather than the feed."


Posted by: ogged | Link to this comment | 10-24-06 10:46 PM
horizontal rule
53

Hm, really? Can I ask why you prefer not seeing the whole thing (along with comments)? There are several blogs where I pretty compulsively read every post. If my RSS reader only showed the first two or three lines, I'd be clicking back and forth from my browser to my RSS reader (or between browser windows) constantly -- with a little one-second delay at every click.

Curious why you have that preference, and wonder how many people share it. It must be a lot, since I can't find a single online RSS reader that does what I want.


Posted by: Realish | Link to this comment | 10-24-06 11:40 PM
horizontal rule
54

Most blogs publish full posts, so I don't have to click through anyway, and in addition to being faster to scroll through feeds than to pull up pages, I like the uniformity of the formatting in my reader; I can make the font big enough to read comfortably, keep the column size small, etc.

I only read the comments at a few other blogs, and I just go to those pages directly.


Posted by: ogged | Link to this comment | 10-24-06 11:51 PM
horizontal rule
55

AJAX comments! Yes!

I'm on a slow connection and I can barely participate in 100+-comment threads because it takes like a minute to reload the hundreds of comments I've already seen.


Posted by: neil | Link to this comment | 10-25-06 5:39 AM
horizontal rule
56

O-or, maybe UOTP -- UnfOgged Transfer Protocol.

Why not just use Asynchronous Transfer Mode?


Posted by: My Alter Ego | Link to this comment | 10-25-06 7:33 AM
horizontal rule
57

I hate to be a pessimist, but I don't think that AJAX comments are a particularly great idea. We had them introduced to us over at DCist against our will, and they've been a pain in the ass. MT wants to rebuild the comment page when a new comment is submitted. Do you do that when a new AJAX comment is submitted? I guess you could, but then you lose the performance gains you're shooting for and have just introduced eye candy. Instead you just display the comment using Javascript in the submitter's browser and send the comment to some sort of queue from which the page is periodically rebuilt. It's doable, but you have to tear apart MT and build a PHP comment buffer that runs off a separate database. I was on the verge of writing this a while back, but never got around to it.

Anyway, I'm rambling a bit. My point is just that I've managed an MT system with AJAX comments, and I didn't like the experience. It has lead to lots of double-posting, weird browser errors and people talking past each other. But maybe there's a better implementation out there than the one Gothamist used.

But! If there's an MT plugin that purports to do it, it couldn't hurt to drop it in for an hour or so and see what people think.


Posted by: tom | Link to this comment | 10-25-06 8:25 AM
horizontal rule
58

I hate to be a pessimist, but I don't think that AJAX comments are a particularly great idea.

Yes! tom is now my own personal tech Jeebus. (And Becks, for her technical skills plus being willing to be the wet blanket, is Mary, I guess.)


Posted by: SomeCallMeTim | Link to this comment | 10-25-06 8:29 AM
horizontal rule
59

But! If there's an MT plugin that purports to do it, it couldn't hurt to drop it in for an hour or so and see what people think.

Wait, I hadn't seen the above. Apparently he's actually one of the false tech messiahs that we have been warned against.


Posted by: SomeCallMeTim | Link to this comment | 10-25-06 8:31 AM
horizontal rule
60

That's right! Barack Obama will be the one delivering you to the fiery lake — I'll be providing wifi access there at a ridiculously high hourly rate.


Posted by: tom | Link to this comment | 10-25-06 9:49 AM
horizontal rule
61

Wait, the -ist sites are already doing this?


Posted by: ogged | Link to this comment | 10-25-06 9:55 AM
horizontal rule
62

Yup. Unless I'm missing something that's been proposed for comments here -- but Gothamist and DCist definitely have AJAXified comments.


Posted by: tom | Link to this comment | 10-25-06 10:09 AM
horizontal rule
63

Maybe the actual lesson here is that MT blows?


Posted by: mrh | Link to this comment | 10-25-06 10:10 AM
horizontal rule
64

Tell me that after watching Atrios' upcoming switch to Wordpress. I predict catastrophe (but am willing to be pleasantly surprised).


Posted by: tom | Link to this comment | 10-25-06 10:14 AM
horizontal rule
65

Yup. Unless I'm missing something that's been proposed for comments here

If you leave the comment window open on those sites, new comments automatically fill in, without you hitting refresh?


Posted by: ogged | Link to this comment | 10-25-06 10:20 AM
horizontal rule
66

Ah. No, they don't do that. I thought I had seen something about that, but didn't realize it was part of what everyone had in mind.

Nobody does that because the only way to accomplish this in the web paradigm is with constant polling, which becomes a big problem when a bunch of people forget they have the window open in the background and end up hammering the server for no reason.

I've thought about ways to accomplish this with a funky new interface -- an unfogged master command console of sorts, where you specify which threads you're paying attention to and maintain a lease telling the server to ping you immediately when they're updated. But the vagaries of client-side NAT make this impractical, unfortunately.

I'll think about this some more, though. A firefox extension hooked up to a PHP script might be possible.


Posted by: tom | Link to this comment | 10-25-06 10:34 AM
horizontal rule
67

Actually, forget I said that last part


Posted by: tom | Link to this comment | 10-25-06 10:36 AM
horizontal rule
68

Instead you just display the comment using Javascript in the submitter's browser and send the comment to some sort of queue from which the page is periodically rebuilt. It's doable, but you have to tear apart MT and build a PHP comment buffer that runs off a separate database.

Why do you need to send the comment anywhere? I envision something like this:
1. someone submits a comment; mt does its comment-submission thing and rebuilds the page.
2. you, on your out-of-date, unreconstructed page, send an ajax request to the server
3. server-side script responds with the new comment in some format or other
4. js on your side sticks the properly-formatted comment where it goes

Steps 2-4 don't interact with MT at all. If you want the "rebuilt" version, reload the page. That is, I was primarily understanding this idea as a bandwidth-saver for comment readers, not a processing-saver for commend submitters. You still get some benefit.


Posted by: ben w-lfs-n | Link to this comment | 10-25-06 10:36 AM
horizontal rule
69

65: Why do you persist in this, despite the advice of the technically-minded? Go on, then waste your time and build your Ark. It's not rained in months.


Posted by: SomeCallMeTim | Link to this comment | 10-25-06 10:37 AM
horizontal rule
70

Nobody does that because the only way to accomplish this in the web paradigm is with constant polling, which becomes a big problem when a bunch of people forget they have the window open in the background and end up hammering the server for no reason.

See, this is exactly what I said, but no one listened to me.


Posted by: ben w-lfs-n | Link to this comment | 10-25-06 10:39 AM
horizontal rule
71

I listened, young Benjamin.


Posted by: Clownæsthesiologist | Link to this comment | 10-25-06 10:43 AM
horizontal rule
72

Now we're discussing this on two threads. What I said to that, b-dub (following your own proposed solution) was, "stop refreshing if there have been no new comments in n minutes. (And maybe put a "refresh to see new comments message at the bottom of the thread.)"


Posted by: ogged | Link to this comment | 10-25-06 10:44 AM
horizontal rule
73

68: You're right. I came to this thread late and skimmed it, misunderstanding the planned benefit as being in terms of server load (this was a problem w/ DCist, too). Minimizing rebuilds prompted by comments can wind up being a priority on a heavily-used MT system. But if performance is okay and there's overhead for the additional load, then adding on a PHP/JS solution like you describe would work fine. I've done some exploratory poking around the MT database in the past -- the comment table layout is pretty easy to figure out and would be simple to process with PHP.

There may be some activity-sensing Javascript tricks out there that could be used to avoid the polling problem -- detect if the mouse is on the current window before sending the XmlHttpRequest, etc. But I've never written one, and different browsers would almost certainly require different approaches.


Posted by: tom | Link to this comment | 10-25-06 10:49 AM
horizontal rule
74

And, as long as we're discussing it here, I'll bring up the rss angle again: we already publish a feed (two actually) that contains each new comment; would it help/be easier to parse that and publish new comments in the comment window, rather than polling the server/database?


Posted by: ogged | Link to this comment | 10-25-06 10:51 AM
horizontal rule
75

Well, it's a good thought -- taking advantage of a resource that's already being computed -- but it definitely wouldn't be easier. It'd probably be harder on the server, too. PHP and simple, keyed database queries (which these would be) are really fast relative to parsing XML and Movable Type rebuilds. The MT rebuilds for the feeds are already represented in the site's overhead, but pulling the RSS apart and chewing through it in PHP would be pretty slow, especially if this host isn't running PHP5 (which has faster, built-in XML capabilities).

If you wanted to improve performance you'd actually take those RSS feeds out of MT and turn them into (cached) PHP. It'd take some work to avoid related weirdness, though, and probably isn't worth the effort.


Posted by: tom | Link to this comment | 10-25-06 10:57 AM
horizontal rule
76

Here's something I have wondered about for a while: What if instead of building and rebuilding the pages on the server, and serving them as HTML files, the main page was just a Java app that built properly formatted pages on the client? When the page loads, the app executes and asks the server for the most recent n posts and the most recent m comments.

The server returns for each post, the text and metadata consisting of Author, Title, Text, Date, ID, How many comments, and How many cock jokes in comments. For each comment, Author and Post Title -- the client app uses this information to build a properly formatted main page (or archives page if it comes to that). A similar client-server request thingy could do comments pages. Then you don't have to constantly be rebuilding formatted pages at the server level. Only drawback is it seems like a hassle to build and maintain.


Posted by: Clownæsthesiologist | Link to this comment | 10-25-06 10:57 AM
horizontal rule
77

What if instead of building and rebuilding the pages on the server, and serving them as HTML files, the main page was just a Java app that built properly formatted pages on the client?

I don't want to see a coffee cup for five minutes when I load unfogged and wait for java to get going.

Really, this is solving the wrong problem. Do we really have bandwidth issues? Not to my knowledge. All what we're talking about would alleviate is bandwidth. Who cares?


Posted by: ben w-lfs-n | Link to this comment | 10-25-06 11:02 AM
horizontal rule
78

76: that'd work, and you're right, it'd be a good approach. instead of rebuilding lots of indices within the dog-slow MT system, you'd just use it to build one parsable data source, which could be read by a faster presentation layer as needed. The reason MT rebuilds all these static pages is that it anticipates being less of an interactive platform than unfogged is -- it thinks it's going to have to rebuild its pages and indices infrequently relative to the number of times they're viewed.

But the system you describe is so far outside of MT's basic design that you'd be better off moving to a different blogging platform. No need to reinvent the wheel -- just use Wordpress, where the "parsable data source" is the database (which is very very fast to read). I'm not a Wordpress fan, though.

Here's a stupid question: have the unfogged powers that be considered using typepad? I have a feeling it was tried and didn't work out. But if not, it might be a good way to stop worrying about performance problems, while being able to keep your templates.


Posted by: tom | Link to this comment | 10-25-06 11:03 AM
horizontal rule
79

If I'm understanding 76 correctly, is that the same as MT's dynamic publishing?


Posted by: ogged | Link to this comment | 10-25-06 11:04 AM
horizontal rule
80

this is solving the wrong problem

Reason 1: I thought this would be cool: Constant refreshing is annoying, and being able to leave the window open and get new comments as they arrive would be better for the user.

Reason 2: If we can cut down on server load, that would be a bonus. But I'd even want to do it if it would slightly increase server load.


Posted by: ogged | Link to this comment | 10-25-06 11:07 AM
horizontal rule
81

79: sort of. MT's dynamic publishing loads the information from a database and builds the web pages on the fly on unfogged's server. What 76 is saying is to load the information from the database and then send that to the client, and have the client build the web pages on the fly.

I looked briefly at the MT source code last night, but it's Perl and PHP, which I don't understand, and it looks like comments are already db-driven.

What you want can certainly be done, and should actually reduce the DB load. The problem is that it'd take someone familiar with MT or WordPress all of four hours to do, but anyone unfamiliar would have to spend two days figuring out how it all worked.


Posted by: Jake | Link to this comment | 10-25-06 11:19 AM
horizontal rule
82

76: Well, you wouldn't want to do it in Java. You'd do it in JavaScript with MochiKit, or YUI, or dojo, or some other such thing.


Posted by: mrh | Link to this comment | 10-25-06 12:11 PM
horizontal rule
83

Let's not overlook Ogged's most important justification, which is that it would be cool. That's just about reason enough for me.


Posted by: mrh | Link to this comment | 10-25-06 12:11 PM
horizontal rule
84

But what about our readers who use lynx/links/w3c/etc?


Posted by: ben w-lfs-n | Link to this comment | 10-25-06 12:18 PM
horizontal rule
85

you'd probably actually want to do it in PHP. Or, if we're really just talking about reformatting an XML document, you could try to do it all with CSS. That's what it's designed for, after all.


Posted by: tom | Link to this comment | 10-25-06 12:28 PM
horizontal rule
86

Will PHP execute on the client? I thought it was just for CGI. My understanding of the mysteries of web development is limited.


Posted by: Clownæsthesiologist | Link to this comment | 10-25-06 12:41 PM
horizontal rule
87

But CSS sounds promising.


Posted by: Clownæsthesiologist | Link to this comment | 10-25-06 12:41 PM
horizontal rule
88

But what about our readers who use lynx/links/w3c/etc?

We'll tell him to cut it out.

I don't know if y'all followed the link in 27, but someone at least got the comment posting ajaxified. I realize that's not the hard part.


Posted by: ogged | Link to this comment | 10-25-06 12:42 PM
horizontal rule
89

86: no, it won't, but that's the point -- you don't wanna rewrite the web browser on the client side in the form of a java applet. you use CSS for layout and formatting as much as is practical, aided by whatever server-side scripting is necessary to make it easier.


Posted by: tom | Link to this comment | 10-25-06 12:51 PM
horizontal rule
90

But I thought CGI was more of a burden on the server than frequently rebuilding HTML pages, since the server cannot cache output.


Posted by: Clownæsthesiologist | Link to this comment | 10-25-06 1:00 PM
horizontal rule
91

But what about our readers who use lynx/links/w3c/etc?

GE executive: but what about our customers who light their houses with whale-oil lanterns?


Posted by: apostropher | Link to this comment | 10-25-06 1:12 PM
horizontal rule
92

We looked at using Typepad but it didn't seem possible to do that and preserve intra-comment links. Since MT's import/export function renumbers entries and comments, all of the comment links would be broken. To not hose everything, we have to do a database-level import/export to move our site to a new host.

I suppose this wouldn't be as much of an issue if we made all of the old stuff static and moved the new stuff to Typepad but that would limit our ability to go to a non-Typepad platform in the future because we wouldn't be able to get a database export with comment IDs.


Posted by: Becks | Link to this comment | 10-25-06 1:20 PM
horizontal rule
93

Who's talking about CGI?


Posted by: mrh | Link to this comment | 10-25-06 1:22 PM
horizontal rule
94

Didn't Tom say the server should run PHP scripts to build HTML pages? I thought that was CGI.


Posted by: Clownæsthesiologist | Link to this comment | 10-25-06 1:27 PM
horizontal rule
95

(And whether or not it is "CGI", I thought it caused a burden on the server by keeping it from caching.)


Posted by: Clownæsthesiologist | Link to this comment | 10-25-06 1:30 PM
horizontal rule
96

I thought Becks had slain this dragon. (Tom, nifty idea about turning reload on or off based on activity.)

VERY BRIEF TECHNICAL EXPLANATION THAT I WILL TRY TO DO WITHOUT ACRONYMS BECAUSE YOU DON'T CARE:

So when you submit a comment or make a post, it goes into the database and then Movable Type rebuilds a bunch of pages. It does this because there are good reasons to hand out static pages on high traffic sites -- server load, for example. Wordpress is another blogging package, although I share Tom's feelings about it; the main architectural difference is that whenever you view a Wordpress page it's rendered live through an [acronym] script. There's no rebuilding. (79: I believe that's how Movable Type's dynamic publishing works.) The architectures are better at different things; Movable Type basically frontloads the costs of showing a page, which can make things rough if you rebuild three hundred times a minute as everyone tells Ogged what a rube he is. (Clownie is confused about what CGI is, but right that dynamic pages generally move the costs back to when the page is served, rather than when they are produced.)

What Tom is suggesting in 85 is to use [different acronym] to transform raw data into a viewable page, but that's not the same as Ogged's Magic Asscab pages. The way to do that is to have a script returning "comments for page X in since two minutes ago" to requests which are sent Ajax-style every two minutes, then have some JavaScript on the comments page slipping those in. This could be done Tom-style, but there are other, simpler ways to do it. The Tom/Clownie suggestion is a perpendicular problem.

Other nerds can pipe up if they think I've misrepresented or misunderstood something.


Posted by: Steve | Link to this comment | 10-25-06 2:01 PM
horizontal rule
97

[different acronym]

Ooh, Mad Libs.


Posted by: redfoxtailshrub | Link to this comment | 10-25-06 2:09 PM
horizontal rule
98

(Tom, nifty idea about turning reload on or off based on activity.)

Dangit!!!


Posted by: ben w-lfs-n | Link to this comment | 10-25-06 8:47 PM
horizontal rule
99

the main architectural difference is that whenever you view a Wordpress page it's rendered live through an [acronym] script. There's no rebuilding.

This is, AFAICT, either a nonissue or an incredibly complicated issue. MT comment pages come via a script, and it's comment pages that (as I understand it) we're concerned with. If we're concerned about the rebuilding of the main page every time a comment is submitted, that's different, but honestly, that seems beyond our ken entirely. I just got two dolphins in a duration in which either no, or no non-dolphined, comments were submitted.

I'd like to point out that we already have a routine, disconnected from MT, written to convert the database comment data into the nicely HTMLized form we're used to. And it's almost accurate, too!


Posted by: ben w-lfs-n | Link to this comment | 10-25-06 8:53 PM
horizontal rule
100

Huh, looks like you're right, and rebuilding on comment submission can be disabled (which must be how you're running it). I'll look at the code more closely.


Posted by: Steve | Link to this comment | 10-26-06 12:08 PM
horizontal rule
101

101!

Hey Steve, could you explain to me how I am mistaken about what CGI is? I was looking at this page and some others on the same site, and the way they define it seems close to what I was thinking. I believe it is also the acronym for some kind of graphics system but that's not what I meant.


Posted by: Clownæsthesiologist | Link to this comment | 10-26-06 12:51 PM
horizontal rule
102

A-and plus, as long as I've got your ear (assuming I have): Could you combine dynamic assembly of pages with allowing the server to cache page data when no update has occurred, in the following manner:

Somewhere that's really easy and quick to access, store a static flag that will get set when the page being served needs to be rebuilt. The script that gets invoked to build the page should check that flag (which again needs to be a really fast operation) -- if it is set, { clear the flag and build the page, and save the page to disk, lets say at "/mysite/mainpage.html" } -- then output, "Location: /mysite/mainpage.html

". Then the server can cache the file and it will still get updated when it needs to. Also I guess you need a semaphore so that processes updating the file will not bump into each other.


Posted by: Clownæsthesiologist | Link to this comment | 10-26-06 1:27 PM
horizontal rule
103

It stands for "Common Gateway Interface", a standard for exchanging information between a program running on a server and a web server running on that same machine. (Basically, a request comes in to www.example.com/cgi-bin/someapp; the CGI standard defines how you can pass variables to someapp; the server then launches someapp with those variables and returns the results as a web page.) These are usually, but not always, written in scripting languages such as Perl (in which Movable Type is written) or Python, as opposed to something that compiles to an executable binary, like C or Java. The more common way to do it now is through compiling server modules (mod_perl, mod_php) that contain translators for the scripting language so you don't have the overhead of launching the translator every time the page is called, or FastCGI, which attempts to minimize the number of times the outside application is launched (and does some other technically wizardypokery to make things run faster).

So you're right that PHP is something that runs on the server, but while it's technically possible to run PHP via CGI, the vastly more common scenario is to run it as a server module (php_isapi.dll for Microsoft's IIS, I believe, and mod_php for Apache).

The stuff that happens on the client side for Ajax-iness is written in JavaScript; it passes requests, which are basically just form submissions, through the Javascript "XmlHTTPRequest" function (or an iframe) to prevent you needing to reload the page, then does something with the data that comes back. The browser is language agnostic about how the server deals with the information it just sent. The reason people are talking about PHP specifically is it's a widely used language, particularly for quick and (very) dirty web hacks like this.


Posted by: | Link to this comment | 10-26-06 1:31 PM
horizontal rule
104

The blahblahblahginger of 103 was me.


Posted by: Steve | Link to this comment | 10-26-06 1:32 PM
horizontal rule
105

Ah -- I did not get the distinction between CGI and server modules. I wrote a CGI program a couple of years ago, in C. I also wrote my blog, in ASP, and I had always thought it was a CGI program but now I'm thinking maybe it's a server module.


Posted by: Clownæsthesiologist | Link to this comment | 10-26-06 1:34 PM
horizontal rule
106

Do you know where the specification for server modules lives? Or is it server-specific?


Posted by: Clownæsthesiologist | Link to this comment | 10-26-06 2:04 PM
horizontal rule
107

Right, if you had a web interface to a C program, you would have strung it up to be web-accessible using CGI. I'm not sure whether there even is a standalone way to run ASP (which is why you would need to do it via CGI). (Okay, I just Googled -- you apparently can through Perl's Apache::ASP module, which sounds brainmeltingly insane, so there are probably other ways to do it.)


Posted by: Steve | Link to this comment | 10-26-06 2:09 PM
horizontal rule
108

Do you know where the specification for server modules lives? Or is it server-specific?

What do you mean specification? In Apache, which is the server I'm most familiar with, they get called out in httpd.conf. (In Apache2, they'd also get called out in conf.d/FOO.conf.)

If there's something specific you're working on, feel free to shoot me an email at my domain - ".org" at gmail.


Posted by: Steve | Link to this comment | 10-26-06 2:13 PM
horizontal rule
109

This may be OT, but am I the only one who finds the posts+comments feed on this site almost completely useless? I can't understand any scenarios or blog-reading habits that this feed would usefully service. I love the post feed, and the new comment feed, but having a feed that contains the entire posts/comment threads, and updates with each new comment? Who would use this, and how exactly?

Note that I'm not trying to complain, I'm honestly assuming there is some utility to this thing that I just can't figure out.

I feel like there was a recent post by ogged about the feeds that would have been a better place to put this comment, but I couldn't find it. I didn't search for it either.


Posted by: Brock Landers | Link to this comment | 10-26-06 4:53 PM
horizontal rule
110

Brock Landers is banned!

I love the post+comments feed so I can read the thread offline (in the airport, etc.) in one big page. And so I don't have to constantly refresh the page at work.


Posted by: Becks | Link to this comment | 10-26-06 5:02 PM
horizontal rule
111

Agree with Becks. Landers is in league with Al Qaeda.


Posted by: SomeCallMeTim | Link to this comment | 10-26-06 5:11 PM
horizontal rule
112

So, how does that work. You open the thread and read the post and, say, the first 78 comments (or however many were left up through that time). Then 15 more comments are left -- your feed shows an updated thread, and you have to open the whole thing over again and scroll to where you left off? And do this every time a new comment is added? For every thread?


Posted by: Brock Landers | Link to this comment | 10-26-06 5:19 PM
horizontal rule
113

My newsreader (GreatNews) has a pane in which I can select the post (+comments) that I'm interested in, and I can scroll fairly quickly to the end of the comments. It's not perfect--right now, I only get the comments through #95 on this thread--but it's a nice option.

OT-ish: I just came across Ziepod, which is the best podcatcher I've found. I like it quite a bit.


Posted by: SomeCallMeTim | Link to this comment | 10-26-06 5:29 PM
horizontal rule
114

112: That is exactly right.


Posted by: mrh | Link to this comment | 10-26-06 6:10 PM
horizontal rule
115

114: what a nightmare.


Posted by: Brock Landers | Link to this comment | 10-26-06 7:41 PM
horizontal rule
116

I'd like to take back all those nice things I said about the comment-only feed (bridgeplate). I only signed up for it a few days ago and while I loved it at first, I've since realized that it's skipping *lots* of comments. Not just behind on comments, which wouldn't be a big deal, but just skipping right over them. Like in the "Talk" thread, comments 150-152 didn't appear in my reader (which left me confused about some later comments in the thread, which is what led me to click through and discover this quirk.) And I now see that I'm missing so many comments from the "Gah" thread that I can't even bother to count them.

Does anyone else have similar problems? Is it something I'm doing wrong on my end? I'm using bloglines for my reader.


Posted by: Brock Landers | Link to this comment | 10-27-06 7:32 AM
horizontal rule
117

Yeah Brock, I have noticed that too. Though it seems more like "occasional comments" than "*lots* of comments".


Posted by: Clownæsthesiologist | Link to this comment | 10-27-06 8:27 AM
horizontal rule