Old forum link

[Oldschool]

Figured I'd post the old forum link although some of us may have saved it as a bookmark/favorite as it is a wealth of information and served the community well.

For anyone that wasn't a registered user of the old forum you'll need to register it to access it. Looks like new registration is still currently available but unsure if that will change.

Here's the link.

http://shadowwolf47.proboards.com/index.cgi?action=login

[Scarbrow]

Finally sick of the buggy searching engine of the old forum (increasingly unreliable as time passes), I've been playing around with WinHTTrack and I've been able to fully download (I think, it's still in testing) the entire old forum in HTTP form.

At least I'll be able to view it quicker, but it's a monster of 8829 HTML objects weighting 500 Mb (though it can be easily compressed to below 15 Mb, and I could upload that to Megaupload or something like that for others to see). Now I want to know if there is any way to build a database around it, or extract the posts from the pages, or as the last resource, build a custom search engine for offline use. Do any of you know how to do any of this? I have only the faintest idea myself, I'm just crossing the bridges when I find them

As the supreme option, we could also port the old posts here, but because of volume (there are 3,245 threads with a total of 57,442 posts), and since much of the useful information is metadata (who posted what, and when) that would be extremely hard to do.

Opinions? Hints?

EDIT: I've submitted some interesting stats for the old and new forum's member composition (http://www.srythforum.com/showthread.php?p=18531#post18531) in the reputation poll thread

[Young Ned]

It's possible that our current forum software has the ability to import the old forum data, if it supports importing from that particular forum system. I've seen some forum conversions where they automatically ported everything from an old forum when switching over to new software. Those were always done at the time of switchover, though; I'm not sure whether the fact that we've been using the new forum for several months now would complicate things.

At any rate, conga-rats for managing to extract all the old posts into a more searchable format. That must have been quite a job! :cool:

[Scarbrow]

I doubt that proboard's format is supported. Citing from their "support" forum (http://support.proboards.com/index.cgi?board=support&action=display&thread=30155) (it still enrages me each time I read it):


Deleting board (http://support.proboards.com/index.cgi?board=support&action=display&thread=30155&page=2#220357)
Can I have a copy of my ProBoards database? (http://support.proboards.com/index.cgi?board=support&action=display&thread=30155&page=3#220369) (in short - NO)
What if the Admin 'disappears' ? (Transferring ownership) (http://support.proboards.com/index.cgi?board=support&action=display&thread=30155&page=3#2196741) (In short - YOU CAN'T)


That is the worst customer service I've ever experienced. How different from Wikia, that even lets you "adopt" an orphan wiki if the original admins disappear. What wonders me is why we took so long to change (especially since Havoc was quite aware of the problems and was proposing for a forum change since December 2005 (http://shadowwolf47.proboards.com/index.cgi?board=general&action=display&thread=51)). That was thread number 51, at that time the change would have been a lot less painful (although as the Sryth community didn't have its current members, we couldn't have organized the new one so well :))

[Scarbrow]

Good news:

The preliminary tests show that (at least, on first sight) my (now considered) trusty HTTrack managed to extract all threads from the old forum (and as you've seen, I'm making a lot of casual necrothreading job as I proceed).

There are 20 more threads than informed by the forum's software, placing the number at 3265 threads (topics). I've manually confirmed that figure on the smaller boards.

I've managed to reduce duplications on files, too. Now there are only 4318 HTML documents, weighting 273 MB. As most of the removed content was duplication, the size of the compressed file has just reduced to 11 MB

About the thread numbering, while it seems like it is always related to thread's creation date, the system is not quite clear. It almost seems like at the start of the forum, the threads were only created with odd numbers, but there are exceptions. Sometimes the jump was just 1, or 3. After that, it seems like the threads started being numbered by natural numbers, but again there are exceptions and odd jumps. Conclussion: you can't just try to follow the threads by URL-picking: you had to suck it from the indexes like I did. Phew, I feel relieved there was no other way to do it.

The longest threads (by number of pages - 30 posts per page)

Ruins of Tarramyre - 85 (of course)
Stuck? - 56 (on Help)
Ruined city of Tarn - 42
Ildraria's Lair - 34
The Curse of Talderus Redborn - 29
Wurmwrit Farm - 25
Saarngard Isle expansion - 20
The Crimson-Helmed Rider - 20

Any doubts of why we created this forum with a special section for multiplayers and replayables? :p

Well, that's all for the moment. More statistics to come, now I'm going to start reverse-engineering the evil Proboards code (almost sounds like a quest :D)

[Scarbrow]

Progress report:

I've managed to salvage (extract from the HTML) a total of 56,400 posts (98,18%) through semiautomatic (ok, almost fully automated) parsing. There are still 21 threads that have managed to elude my efforts (around 300-600 posts on them), so I'll have to parse them manually.

For the moment, it weights 52 MB on Excel form, so I expect it will be lighter on text or HTML form.

The next step is to build an HTML/CSS structure similar to the old forum for offline browsing. Or alternatively, I could just browse the HUGE chunk of text by hand. If somebody wants to help with shaping up the old forum with simple CSS, (or, if specially masochistic, wants to write a search engine) drop me a PM.

[Oldschool]

Nice work Scarbrow and duly repped for it.

I'd help but I'm lucky to know what HTML and Excel are. As for CSS and parsing I've heard of them. I do however know and am fairly proficient with PM's and smart aleck replies ;) it would seem.

Seriously, excellent progress.

[Young Ned]

Yes, very impressive work, Scarbrow! Sorry I can't help with any of it.

[Oldschool]

Scarbrow your recent "Our missing comrades" thread and my subsequent foray into the old forum got me wondering.

How's things coming along on the archiving of the old forum (for lack of a better word) and have you gotten any help?

[Scarbrow]

Right now the project has stalled (partly because my main tool to treat large quantities of data, Excel, plainly and weirdly refuses to parse some of the threads). I'll try to retake it when I'm able, although if somebody wants the collection of HTML files, I'll be happy to email them.

[Scarbrow]

I've finally completed my parser, and I've been able to recover a total of 59,214 posts on 3265 threads. Yep, there are more than the forum engine informs of.

There are also some users who had way more messages than their post count indicates. Some have less, but that can be explained as deleted messages which are not deducted from the post count.

The text file with the contents weights 29 MB (4 MB compressed) and the Excel file with the metadata a little below 10 MB (1,5 MB compressed), so it's now definitely email-able.

Next step: finding a way to present the information as a forum, not as separate data + metadata.

[Oldschool]

You have given out too much Reputation in the last 24 hours, try again later.Grrrr..... repped when allowed Maven Three Hats ;) - Wiki and both forums.

Oh and to no one in particular following Scarbrow's lead I've been delving a little more into mod "powers" and although I haven't tried it I think I have figured out how to change names and titles.

Muhahahahahahahaha!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! http://www.freesmileys.org/smileys/smiley-devil12.gif (http://www.freesmileys.org/smileys.php)

EDIT: Actually it seems I could do a little more.

Oldschool

[Ban User] [View Profile] [Change Signature] [Change Avatar]
[Change Profile Picture] [Edit Reputation]

Change signature.....

Kilroy Oldschool was here.

Oh the lure..... ;)

[Oldschool]

Boy am I glad we have this forum, that old one is haunted.

And for you naysayers I say peek through the shutters...... if you dare.

[spencer]

Does it have things that go BUMP in the night?

[dman2life]

I think we have things that go stabbity stabbity in the night.

It seems I joined the old forums in 08... I was playing Sryth for a few months before then, in 07. Holy crap that's a long time. At lest, for a video game.

I have some good memories of that forum. Ah, such good times.

[scout1idf]

Scout saw and joined the ghost's over there for the first time in quite a while...

I think it may be time to go back and remember, just not now, work's a calling :(