Release New Fun

GCD Web Site Release: New Fun

This is the main page for the first release of the new GCD Web Site, code-named New Fun. This page will holds general release-specific information plus links to documents on specific aspects of the release.

[back to the Web Site Project Page]

Release Goals

• To implement the minimum set of functionality required to deploy to production:
  • Display capabilities no more limited than what we have now.
  • Modification capabilities similar to what we have now, but with as much as possible subject to an approval workflow.
  • Full edit access restricted by user permissions for anything that is not yet under an approval workflow.
• To support the existing data and schema, with only the following minimal modifications:
  • Drop unused columns.
  • Migrate user management to Django's authorization framework.
  • Additional tables/columns needed to implement the full workflow.
• To lay the groundwork for a new look-and-feel (including navigation and search capabilities) of the site which will support our growth into the new schema.
• To thoroughly support internationalization.
  • UTF-8 support throughout the code and display.
  • Localization for all countries/languages currently involved in the GCD which have native speakers available to provide text.
• To ship with a test suite that will support rapid but reliable future growth.

Release Timeline

Note that this list implies a rather strict separation between phases (such as feature specification and coding, for instance). It is likely that these will actually overlap in various ways, and this section will be updated to reflect that.

• Detailed Requirements: 2008-11-02
• Design Specification and initial round of Bugs in Bugzilla: 2008-11-30
• Test Plans: TBD
• Test Data: TBD
• Code Complete (includes automated tests): TBD
• Deployment: TBD

Detailed Requirements

Throughout this section, [REQ] denotes a hard requirement, without which the release may not ship. [OPT] denotes an optional requirement, which may be bumped to a later release but is included because it is thought to not add much time over the hard requirements, or because it is particularly desirable. [OPT] requirements will mostly either become [REQ]s or move to a later release as planning solidifies. [NON] designates a non-requirement- a feature that is explicitly deferred to a later release, but is mentioned here for clarity.

Django Middleware

The following Django Middleware functionality needs to be integrated:

• Authentication
• Sessions
  • Stale sessions are not automatically garbage-collected, so we need to implement that, or find an available implementation.

Look and Feel

Colors

• Color scheme will be adjustable through default and user-contributed style sheets.
  • Support for user-contributed style sheets is not part of this release, but internal code should allow it.
• Default colors:
  • Must work well with common forms of color-blindness.
  • Background: TBD
  • [other color roles TBD]

Navigation

The GCD will use a fairly standard layout for navigation, with the global navigation across the top and loal navigation down the left side (TBD: possibly flipping to the right side in RTL locales). The global navigation will include widgets for switching between sections, running basic searches and logging into the site. Local navigation will be specific to a given section or page type, for instance listing the sequences on an issue page. Supplemental navigation will include site maps and guides, and be in a separate area accessible through the Global Navigation.

Global Navigation

• [REQ] Global navigation will occupy the top of every page
  • [OPT] The front page may have a customized by stylistically consistent version of global navigation
• [REQ] The basic search box will be part of global navigation
• [REQ] The login form w/ registration link will be part of global navigation
• [REQ] The following options will be present (as menu items, tabs, etc.- TBD during design) Sections marked as [OPT] are so marked because the area to which they link is optional for this release.
  • [REQ] Home
  • [OPT] Browse
  • [OPT] Advanced Search
  • [REQ] Contribute
  • [OPT] Projects (see description under- um.. where does this go?)
    • Projects could go under contribute, but having specific projects is a good recruiting tool, so giving it a prominent place is useful
  • [REQ] Help
  • [REQ] About
• TBD: Should a "Donate" button be part of global navigation? Or possibly in the global footer?

Local Navigation

As is implied by the name, Local Navigation will take different forms depending on what section we are in and what is being displayed in the main content region.

• Local navigation will be implemented as a sidebar to the left of the main content region
  • TBD: Switch to the right when displaying in an RTL locale?

Defaults

• [REQ] If browsing is implemented, then the default local navigation for areas that do not have content requiring local navigation should be the browse navigation.
• [REQ] The following areas may not have need for local navigation, since they either have little content, or will link to documents on the wiki rather than documents stored within the web site application, or just don't have enough subsections on the page to need a menu:
  • Home Page
  • Contribute
  • Projects
  • About
  • Publisher view
  • Series view
  • Cover galleries

Browse

Browsing is an OPT feature of this release. A [REQ] here means "required if Browsing is implemented at all".

• [REQ] Switch between areas of browsing (publishers, series, etc.)
• TBD: Should the local nav implement things like selecting which letter of the alphabet one is browsing (assuming one is using an alphabet, which all current content does). Probably better to use the main content panel due to space issues.

Advanced Search

• [OPT] If advanced search is implemented, local navigation may allow switching between specialized search forms

Help

• [OPT] Links to various help topics for using the site. Most often, folks will end up on one of those pages directly by clicking a help link. Assuming help gets implemented in this release beyond the very basics.

Issue View

• [REQ] Sequence-by-sequence links, as currently implemented in the right-hand box. Moving this will make layout more consistent. Not to mention much easier to implement.

Search

The focus for search in New Fun will be the search box- the primary interface seen on the front page and as part of the global navigation of the site. This is because it is the search interface that is immediately available to the user, and the one the vast majority of users will use. Advanced search generally appeals to a much smaller set of users, especially if the search box is well designed. It will be present only in a basic form, or possibly even deferred to More Fun. It is important to take advantage of this major change in the UI to get the user base accustomed to a new way of using the search box. Changing its behavior drastically *after* the main UI change is likely to be much more confusing.

The Search Box

The ideal interface for the main search form on a web site is a single text box that by default searches the entire site, considering each term separately unless they are grouped by quotes. However, this works best when each sort of thing the user might want to search is represented by its own database object. This is true of some things in our site (publishers, series, issues, sequences), but not yet true of others (creators, characters, features).

Because of this and some other cases where specific searches are needed (i.e. reprints), we will continue with a zoned search system instead of a single box. However, we will move incrementally toward the single box model by reducing the number of zones in the menu and in exchange allowing more specific requests to be made through search terms. This follows the model broadly popularized by Google, whose search box behaves very simply, but allows a number of advanced features if one pokes around enough. Our search results will include hints to people as to how to better format their search, in addition to the traditional help page.

The operators proposed are not quite identical to Google, mostly because we are searching a structured data set rather than the entire web. The principle behind using seemingly cryptic modifiers such as "+", "-" and "^" is that they are quick to type, and do not clutter up the search interface for those folks who don't need them. Similarly, using qualifiers like pencils:Kirby is similar to how Google applies restrictions (i.e. site:comics.org). Use of quotation marks allows those symbols and words to appear in search terms (i.e. "pencils:Kirby"), although the need for this should be rare.

• [REQ] Of the form "Search [menu] for [textbox] (Go)"
• [REQ] Search zones (the stuff in [menu]):
  • Comics
    • This covers both series and issues. By default, it returns series.
    • [OPT] Trailing numbers (including comma-separated lists and ranges) will be parsed as issue numbers, causing a sets of issues to be returned instead of series.
      • Numbers at the end of a series title can be quoted to avoid this behavior.
      • See mycomicshop.com (among others) for an example.
      • A starting issue before the first issue or an ending issue after the last issue will be treated as the first and last issue, respectively.
      • [NON] Non-numeric issues cannot be found by this method. Advanced search or the search box functionality of a future release will address this.
  • Publishers
    • As with the current OI, this should return both top-level publishers and imprints, and just make it clear in the results which is which.
  • Credits
    • Searches Script, Pencils, Inks, Colors, Letters, Editing and returns sequences.
    • Each field is also present as a separate, specific search zone.
    • [OPT] Instead of individual search zones, have one zone for Credits (or even one zone for sequence-based searches) and specify fine-grained restrictions in the search text, like Google does with its qualifiers (i.e., on Google site:comics.org will restrict the search for all other terms to pages on comics.org)
      • This could allow more fields to be searched without bloating the UI.
  • Stories (Sequences?)
    • Searches sequence title and feature. Returns sequences.
    • [OPT] Split this and provide separate search zones for titles and feature.
  • Characters
    • Searches feature and character appearances. Returns sequences.
    • Feature is searched only because so many sequences fail to list the feature character in the character appearances, or fail to list characters at all.
  • Reprints
    • Searches reprint notes. Returns sequences.

• [OPT] Implement keyword searches, and use as the default in place of phrase searches.
  • By default, look for any keyword, not all.
  • Implement stop words ("the", "a", "an", "and", etc., need international list)
  • Allow grouping into phrases (words and phrases are known as "terms")
  • Allow required terms
  • Allow excluded terms
  • Allow searching for things that start with a term (with other terms searched anywhere in the text)
  • Allow searching for an exact match with a single term

Narrowing Results

• [REQ] Code structure must support adding functionality of this sort.
• [OPT] Narrow with another search (same search box functionality), ANDed to existing search. Will require more details if this is to go into this release.
• [OPT] Broaden search by ORing new terms to the existing search. Will likewise require more details if pursued.

Advanced Search

• [OPT] Support searching every field to some degree.
• [NON] Complex searches along the lines of Bugzilla's Boolean charts should be deferred to a later release.

Internationalization

• [REQ] All text encoded in UTF-8.
• [REQ] Database migrated to UTF-8 encoding.
• [REQ] Layouts friendly to LTR and RTL languages / bi-directional display
  • Bi-directional display will be needed if, for instance, the overall page is localized to an RTL language, but the user is viewing a record with data in an LTR language. Or vice versa.
  • [REQ] CSS structured to make switching between LTR and RTL layouts easy.
• [REQ] Layouts robust to character / word / phrase size changes.
  • Parts of the existing "new" layout rely on a specific width of text to avoid overflowing an area of the page. These assumptions should be removed.
• [REQ] All label text should be set up for localization.
• [OPT] All error messages, help text and front page verbiage should be set up for localization.

Localization

• TBD: Need list of locales to be supported in this release, and translators for them.
  • German (Jochen Garcke)

Accessibility

• [REQ] Must support screen readers (alt text, labels, etc.)
• [REQ] Must render decently on an 80-column text browser.
• See also mobile browser support.

Branding

• [REQ] The GCD logo shall be displayed in the upper left-hand corner of every page (TBD: upper right-hand when localized to RTL languages)?

Advertising

• [REQ] Space must be available at the bottom of most pages for Google Ads, as seen on the current site.
• [REQ] Prominent space must be available on the home page for the CBLDF ad.
• TBD: Google ads at the bottom of the front page?

User Profiles and Preferences

• [REQ] Architectural support for user profiles and preferences (per user and per session).
• [OPT] The ability to set preferences.
  • Search result order?
  • Display options? (filter out certain types? filter out synopses?)

Workflow

Unlike in the current system, all records (publishers / imprints, series and issues) must be reserved for editing and go through the approvals process. This applies equally to new skeleton records, unindexed issues, or existing issues to be edited. Implementing this will allow indexers to do more work without sacrificing quality control.

Throughout this section, "users" refers to editors, indexers, and unregistered potential indexers as a whole. "Approver" refers to any user who can approve changes (typically editors). See the section on User Roles for the details of exactly who can do what within the system.

Reservations

Reserving a record should be easy, but all such reservations should expire within reasonable amounts of time. Indexers or editors who tie up records should be prevented from reserving more until their existing backlog is completed or released. This process should be flexible enough to handle the varying amounts of time indexers have available to contribute.

Individual Reservations

There will not be a separate reservations UI. Records (issues, series, publishers) can be reserved in several ways:

• [REQ] Users can click the "Edit" button on the record's page.
  • [REQ] Logged-in users get a confirmation dialog with the expiration date of the reservation, so that they understand they are reserving the record, and that it will be locked until they release it or until the reservation expires.
  • [REQ] Anonymous users will be prompted to log in or register before seeing the confirmation box and being allowed to reserve / edit the record.
• [REQ] Users can select records from a search results table and clicking a "Reserve" button. This will only be visible for logged-in users.
• [REQ] Reservations initially expire after one month (but see extension functionality below).

Reservation and Record Creation

• [REQ] Newly created records are automatically reserved by the record creator, as even skeltons should go through the approval process.
  • [OPT] A "create and submit" shortcut may be provided.

On-Going Reservations

• [REQ] Series pages and results table entries will also allow the creation of an on-going reservation.
  • [REQ] If a series has an on-going reservation, only the reservation holder may create add new issues.
  • [REQ] Only new issues are automatically reserved by the on-going reservation. Existing issues are still open to the normal edit / reserve process.
  • [REQ] On-going reservations expire every six months, but may be renewed during the last month of each period.
  • [REQ] The expected frequency of a title over the next six months must be specified in the reservation (monthly, weekly, or simply an expected number of issues). This has to do with expirations and renewals, as described in the next section.

Expiration, Renewal and Extension of Reservations

• Reservations with no changes are immediately released upon expiration.
• Reservations with changes are not expired, but the user will not be allowed to reserve additional issues until all overdue reservations are submitted or released.
• Reservations may be extended at any time, up to the maximum reservation length of 3 months (from the original reservation date, not from the time of extension).
• [OPT] Email notification one week before the reservation expires.
• [OPT] Editors may extend reservations other than their own up to 6 months, to accommodate particularly complex projects like large anthologies.
• On-going series reservations may be renewed during the final month of each period.
  • If the indexer is more than one month behind, the reservation may not be renewed (i.e. more than one unsubmitted issue for a monthly (or any more rarely published series), more than 2 issues for a bi-monthly, more than 4 issues for a weekly, etc.)
  • [OPT] Email notification at the beginning of the final month of a registration period.

Reservations and Display

• [REQ] Reserved records (individual or on-going) will show the reservation holder's name and the reservation's expiration date in place of the Edit or Reserve button.
• [OPT] A form for contacting the reservation holder without disclosing their email address may be provided.

Editing

Once a record has been reserved, it may be edited repeatedly until it is ready for submission for approval, or is abandoned and released for others to edit. In this section "Editor" always means the user role of Editor, not simply a generic user who is editing the record. Generic users are referred to simply as "users".

• Edit screens will be based on the display screens
• Edits will not be reflected in the display until after approval
  • [OPT] Editors may view a record as it is being edited, and when logged in they will see a button that allows this.
• The following functions will be available to the user while editing:
  • Submit for Approval
  • Save Changes (without submitting)
  • Discard all changes and release reservation
  • Extend reservation (up to the maximum duration, as described above)

Publisher Input Fields

The following fields have a simple text box for value input (starred fields are required before submission):

• Name*
• Year Began*
• Year Ended (linked to a checkbox marked "Active", and can only be filled out if "Active" is unchecked)
• URL (validated for syntax when saved)

The following fields are free-form text:

• Notes

The following fields must be validated against the database (either by using a select box of some sort, or by validating the contents when the field is saved):

• Country*
• Parent publisher

The following flags do not directly correspond to existing fields or database columns, but will be mapped into either the notes field or a migration column (see the Migrations section) behind the scenes. They are all off by default:

• Imprint (available only if a parent publisher is set)
• Indicia publisher
• Multiple indicia publishers
  • Either this or Indicia publisher may be set (or neither), but not both.
• Assigned name (If this is set, then none of the other flags may be set).

In this UI (as will be made clear by the help text) the imprint flag indicates that this publisher is not only an imprint in our database (which just means that it has a parent publisher), but actually meets our (TBD) definition of an imprint, as opposed to just being a convenient place to shove indicia publishers or other corporate names that don't quite fit. Assigned name exists to flag publishers that don't correspond directly to any real-world corporate entity. The following table illustrates the use of flags (at least somewhat, as this is a contentious area- not all examples currently exist in the database):

Name	Has Parent?	Assigned Name?	Imprint?	Indicia Publisher?	Multiple Indicia Publishers?	Notes
Marvel	No	Yes	No	No	No	As far as I know, "Marvel", just one word, is not used in any indicia.
Timely	Yes	Yes	No	Yes	No	This is not the legal name of any company, but is the group name given to the earliest phase of what's referred to overal as Marvel.
Timely Comics, Inc.	Yes	No	No	Yes	No	Appeared in excatly this form the indicia of numerous issues.
Timely Publications: 1-20; Complete Photo Story: 21-68; Marjean Magazine: 69-75	Yes	No	No	No	Yes	Set of indicia publishers used for Captain America Comics. Flagging this separately lets us find these more easily when migrating later.
Vertigo	Yes	No	Yes	No	No	Vertigo is definitely an imprint. As for indicia publisher, casual survey of Vertigo issues and trades shows that they seem to use a DC corporate name for that.
Oni Press, Inc.	No	No	No	Yes	No	This is a real corporate name (so not an assigned name, even if it has child imprints).

Series Input Fields

Starred fields are required to be filled out before submission for approval.

The following fields will have a simple text box for value entry:

• Name (series title)*

TBD: What to do about publication dates (and implicitly, year began/ended). When all issues are noted, this can be inferred, but if not all of the issue records are there, this is useful to specify separatel.

The following fields must link to other database objects, either by validation upon submission or by using some sort of select box:

• Country
• Language
• Publisher (should be indicia publisher- group publisher can be inferred from that)
• Imprint (for the "real" use of imprint, i.e. Vertigo, not for indicia publishers)
  • TBD: Handling publisher/imprint fields while that concept is still a mess.
• Tracking
  • Separate entry for tracking from vs to
  • Must link to a series record
  • May link to an issue in the target series (i.e. the theory that Young Allies continues as All-Winners #21)
  • May link from an issue in this series (i.e. linking from All-Winners #19 to whatever picked up with #20)
  • This is also useful in weird co-numbering situations, as happened with two Fox series (on of which refers to the co-numbered series by name in the indicia)

The Format database field will be represented with the following UI elements. Format values set through this form will be recognized by the UI (see the Display section for details):

• Height (inches for U.S. books (?and older UK books, and others?), cm otherwise)
• Width (inches for U.S. books (?and older UK books, and others?), cm otheriwse)
• Height and Width will be in measurements as follows:
  • Inches to 1/16" for U.S. books (and UK? before a certain time? Any others?
  • Metric for all others, and by default if there is a question.
• Interior color
• Cover color
• Color fields will accept the following values:
  • Full Color
  • Black and White
  • TBD: other values such as grayscale or duotone?
  • TBD: [OPT] Allow free text to handle unusual situations?
• Binding (which accepts the following values):
  • Saddle-Stitched / Stapled (through fold- how to describe this?)
  • Stapled (across fold- how to describe this?)
  • [need to dig up the other terms and their definitions- squarebound? perfect?]
• Cover stock
• Interior paper stock
  • [Need terms and descriptions. How to deal with hardcovers, prestige, etc?]
• Flags/checkboxes for mini-series, max-series, on-going [what else? hardcover? tpb?]

The following field will be represented by a free-form text area:

• Notes

The following fields that are now visible will not be explicitly visible or editable:

• First and Last Issue
• Issues indexed

Issue and Sequence Input Fields

An issue and its sequences may be edited all on one page.

TBD: While properly a design rather than requirements issue, meeting the full requirements of editing everything on one page in a usuable manner will probably only be practical with liberal use of AJAX. The strategy for editing when JavaScript support is not present must be determined, unless we decide that while display must work without JavaScript, editing may require it. Options for edit support without JavaScript include:

• Reloading the whole page whenever any information needs to be written back to the database or the session (for instance, adding a new sequence, or adding a credit or character to a sequence since those will no longer be free-form fields). This could be prohibitively expensive for a large index.
• Splitting the edit into multiple pages, at least one page per sequence, possibly with sub-pages for things like adding credits or characters. While less appealing in many ways, if the performance of the site is improved, this may be more usable than one mega-page. It may also be easier to code.

The following issue-level fields will use a simple text box for value entry:

• Issue number
  • A linked checkbox may be checked to indicate that there is no number, which will produce the appropriate text (currently '[nn]') automatically.
  • 'nn' and '[nn]' will not be allowed- the error message will suggest checking the box.
  • [OPT] UI for indicating made-up numbers, or cover vs. indica numbers. This will probably get punted to the More Fun release.
• Volume number (linked to a checkbox that must be unchecked to indicate that a volume number is present)
• Sort code. Plain number, decimals allowed. Drawn from the cover sort code for existing records.

Publication Date:

• Modified date form
• Extra months (pairwise, seasonal, "Holiday", no month)
• "Early", "Mid", and "Late" modifiers for bi-monthlies or odd schedules
• Day (for weeklies)
• Checkbox for irregular dates- when checked a plain text box may be used to enter an unusual date.
• Dates entered or confirmed through this UI will be marked as "migrated".

Price:

• UI to enter one price at a time ("Add Price")
• Text box for decimal number
• Drop down for ISO codes
• Checkbox and separate text box for irregular prices
• Options will exist to specify "free" and "no price"
• [OPT] checkbox to flag newsstand prices

Credits:

• For adding editors, publishers, other issue-wide credits.
• Still read from and stored in sequence 0's Editing field, but marked as "migrated" in the migration column.
• See the sequence credits section for a detailed description of how this field works and enforces formatting rules.

Page Count:

• Simple text field.
• Read from and stored in sequence 0's page count field.
• No need to mark as "migrated" since there are no formatting changes.

Error Reporting

• [REQ] All pages must have a link to Error Tracker (Bugzilla)
• [REQ] All pages should have a link to the technical Bugzilla (TBD: Should we instead add a technical component to the Error Tracker? That would make end-user bug reporting much simpler. We could keep the technical Bugzilla internally.)
• [REQ] Links should always populate the URL field with the URL of the page holding the link, except for the the front page, which should leave the field blank. Any other administrative pages should also leave the field blank.
• [REQ] Links from pages representing publishers, series or issues should put those entities in the Summary field.
• [REQ] Links from cover galleries should auto-select the covers component instead of the default data errors component (users can still change it).

Migration Tools and Fields

• [REQ] SQL script to drop unused colums and migrate user data in to Django tables.
• [REQ] Python script to migrate passwords from plain text to encrypted.
• [REQ] Reprint parsing will be off by default *unless* it can be enabled in such a way that Action Comics #1 loads within the performance requirements.
  • Users logged in with Editor or Administrator privileges will be able to turn on reprint parsing on a page-by-page basis.
  • [OPT] Users with Editor or Administrator privileges may be able to turn on the faster version of reprint parsing globally through a preference, which will be set to off by default.
  • [OPT] Python script to flag reprint notes that the parser can recognize in a migration field (see below). This can be delivered as a standalone tool outside of the release, though.
• [REQ] Each of the publisher, series, issue and sequence tables will be given an additional column in which to store migration data. See the section on Editing for the details of how this will be used. General use cases include:
  • Flagging whether a record has been examined for some migration issue yet, and if so whether it has been corrected or if it needs some additional information that is not yet known.
  • Storing information that the new UI can capture and that has space in the new schema, but that does not yet have space in the current schema.
• [REQ] Editors must be able to search based on this information to track migration status and find records that require attention.

Browser Support

Desktop Browsers

Desktop browsers are the initial primary audience.

• [REQ] IE >= 6.0 (Windows)
• [REQ] Firefox >= 2.0 (Windows, Mac OS, Linux)
• [REQ] Safari >= 2.0 (Windows, Mac OS)
• [OPT] Opera (Windows, Mac OS, Linux)
• [OPT] Konquerer (Linux)

Mobile Browsers

Mobile browsers must be kept in mind, but full / optimal support may be deferred to follow-on release.

• Opera ?
• [TBD: What is on iPhone, Blackberry, Treo?]

Performance

• Comparable to the current site when it seems snappy? TBD. Need actual numbers for both simple and complex pages.
• Caching
  • Use memcached? Hosting implications?
  • Need caching strategy (cache the whole site vs more granular strategy)

Test Infrastructure

• [REQ] Locate or write a Selenium test driver for Django (it seems highly likely that one exists somewhere already).
• [REQ] Classes to model the UI for Selenium tests.
  • Factor xpath or javascript access out of test code as much as possible for behavioral tests, while making the xpath clear for layout tests.
• [OPT] Locate or write a Nose test driver (Nose is an enhancement over the built-in unittest package, and ReviewBoard includes its own Nose driver which may be of use to us).
• [REQ] Build fixtures with data sets that load quickly enough to run the necessary tests in some reasonable amount of time. Obviously, this requirement needs some details.

Feature Specifications

Detailed feature specifications will be recorded in Bugzilla under the New Fun milestone so that they may be assigned to developers and tracked.