Shipbuilding History

What did I do?

To give you some idea of just what a monumental work this website is, here are a few numbers:

Summary
pages	1,186
tables	1,242	not including this one
sub-headers	2,240
data rows	188,933
columns	10,791	(calculated per-table)
cells	1,761,091
external links	995	all
	897	Maps
database size	102 MB	( originally over 870 MB of text files )

The absolutely huge amount of data in this site was assembled by Tim Colton, who passed away in 2022. It must have taken half a lifetime to put this together. I did not know him. When I found out about his passing, I became worried that this tremendous work was in peril of being deleted forever by an unpaid web host or domain registrar, so I immediately took a copy, strictly for myself. I have since been in contact with his heirs, and that now seems less likely any time soon, but not impossible.

Once I got the whole thing downloaded, I found that it was almost a GIGABYTE in size. When I looked at the files, I found they were almost all Microsoft Word documents saved as html. That is probably the worst way possible to do anything. Mr. Colton was an expert in the shipbuilding industry, but he was not a web designer. The site was more than 95% invisible Microsoft Word junk.

So, almost as an exercise in writing "regular expressions" (look that up yourself), and to save disk space, I started to clean it up, all 1180+ pages of it. In the process, I flattened the directory structure to be easier to work with, fixed as many broken links as possible, and deleted links that were unfixable. I also fixed any internal inconsistencies and simplified the navigation tree. After multiple passes with link-checker and html-checker, as of now there are no more issues.

I brought over 600 broken Google Earth links up to the latest standard. Then I converted them all to embedded maps, and added the missing ones, for a total of just under 900.

Finally, having stripped out all of the old formatting, I had to come up with something new. Many of these pages are absolutely huge - both tall and wide - and need every pixel of screen space on a large display. Almost all of the data is in the form of tables, many with hundreds of rows to scroll through. There is no room for decoration, which is why there is none. The formatting is purely functional, but not bad-looking, I think. With the addition of navigation structures, sitemap, and some code for Gargle, the site's text now tips the scales at just over 40MB, up from a minimum of about 36MB.

I hacked at this on and off for several weeks, and it went through several morphs until what it is now. Anyone familiar with a 1980s computer terminal or a DOS-based database will find the presentation immediately familiar, even though it is pure simple html and a bit of css in your browser. The last step is to get Gargle to index everything so that it is searchable, and they are SLOW. After three weeks, they've done maybe 10%; at that rate it will take about half a year. I've done everything I can to speed that up, but with Gargle, nothing really helps.

Having done all that work, I figured I might as well make it public, so here it is.