Release New Fun

GCD Web Site Release: New Fun

This is the main page for the first release of the new GCD Web Site, code-named New Fun. This page will holds general release-specific information plus links to documents on specific aspects of the release.

[back to the Web Site Project Page]

Release Goals

• To implement the minimum set of functionality required to deploy to production:
  • Display capabilities no more limited than what we have now.
  • Modification capabilities similar to what we have now, but with as much as possible subject to an approval workflow.
  • Full edit access restricted by user permissions for anything that is not yet under an approval workflow.
• To support the existing data and schema, with only the following minimal modifications:
  • Drop unused columns.
  • Migrate user management to Django's authorization framework.
  • Additional tables/columns needed to implement the full workflow.
• To lay the groundwork for a new look-and-feel (including navigation and search capabilities) of the site which will support our growth into the new schema.
• To thoroughly support internationalization.
  • UTF-8 support throughout the code and display.
  • Localization for all countries/languages currently involved in the GCD which have native speakers available to provide text.
• To ship with a test suite that will support rapid but reliable future growth.

Release Timeline

Note that this list implies a rather strict separation between phases (such as feature specification and coding, for instance). It is likely that these will actually overlap in various ways, and this section will be updated to reflect that.

• Detailed Requirements: 2008-11-02
• Design Specification and initial round of Bugs in Bugzilla: 2008-11-30
• Test Plans: TBD
• Test Data: TBD
• Code Complete (includes automated tests): TBD
• Deployment: TBD

Detailed Requirements

Throughout this section, [REQ] denotes a hard requirement, without which the release may not ship. [OPT] denotes an optional requirement, which may be bumped to a later release but is included because it is thought to not add much time over the hard requirements, or because it is particularly desirable. [OPT] requirements will mostly either become [REQ]s or move to a later release as planning solidifies. [NON] designates a non-requirement- a feature that is explicitly deferred to a later release, but is mentioned here for clarity.

Django Middleware

The following Django Middleware functionality needs to be integrated:

• Authentication
• Sessions
  • Stale sessions are not automatically garbage-collected, so we need to implement that, or find an available implementation.

Look and Feel

Colors

• Color scheme will be adjustable through default and user-contributed style sheets.
  • Support for user-contributed style sheets is not part of this release, but internal code should allow it.
• Default colors:
  • Must work well with common forms of color-blindness.
  • Background: TBD
  • [other color roles TBD]

Navigation

The GCD will use a fairly standard layout for navigation, with the global navigation across the top and loal navigation down the left side (TBD: possibly flipping to the right side in RTL locales). The global navigation will include widgets for switching between sections, running basic searches and logging into the site. Local navigation will be specific to a given section or page type, for instance listing the sequences on an issue page. Supplemental navigation will include site maps and guides, and be in a separate area accessible through the Global Navigation.

Search

The focus for search in New Fun will be the search box- the primary interface seen on the front page and as part of the global navigation of the site. This is because it is the search interface that is immediately available to the user, and the one the vast majority of users will use. Advanced search generally appeals to a much smaller set of users, especially if the search box is well designed. It will be present only in a basic form, or possibly even deferred to More Fun. It is important to take advantage of this major change in the UI to get the user base accustomed to a new way of using the search box. Changing its behavior drastically *after* the main UI change is likely to be much more confusing.

The Search Box

The ideal interface for the main search form on a web site is a single text box that by default searches the entire site, considering each term separately unless they are grouped by quotes. However, this works best when each sort of thing the user might want to search is represented by its own database object. This is true of some things in our site (publishers, series, issues, sequences), but not yet true of others (creators, characters, features).

Because of this and some other cases where specific searches are needed (i.e. reprints), we will continue with a zoned search system instead of a single box. However, we will move incrementally toward the single box model by reducing the number of zones in the menu and in exchange allowing more specific requests to be made through search terms. This follows the model broadly popularized by Google, whose search box behaves very simply, but allows a number of advanced features if one pokes around enough. Our search results will include hints to people as to how to better format their search, in addition to the traditional help page.

The operators proposed are not quite identical to Google, mostly because we are searching a structured data set rather than the entire web. The principle behind using seemingly cryptic modifiers such as "+", "-" and "^" is that they are quick to type, and do not clutter up the search interface for those folks who don't need them. Similarly, using qualifiers like pencils:Kirby is similar to how Google applies restrictions (i.e. site:comics.org). Use of quotation marks allows those symbols and words to appear in search terms (i.e. "pencils:Kirby"), although the need for this should be rare.

• [REQ] Of the form "Search [menu] for [textbox] (Go)"
• [REQ] Search zones (the stuff in [menu]):
  • Comics
    • This covers both series and issues. By default, it returns series.
    • [OPT] Trailing numbers (including comma-separated lists and ranges) will be parsed as issue numbers, causing a sets of issues to be returned instead of series.
      • Numbers at the end of a series title can be quoted to avoid this behavior.
      • See mycomicshop.com (among others) for an example.
      • A starting issue before the first issue or an ending issue after the last issue will be treated as the first and last issue, respectively.
      • [NON] Non-numeric issues cannot be found by this method. Advanced search or the search box functionality of a future release will address this.
  • Publishers
    • As with the current OI, this should return both top-level publishers and imprints, and just make it clear in the results which is which.
    • One possiblility is to return two tables, one of publishers, one of imprints.
    • Another possibility is to return one table with a type field, as done in the current OI.
  • Credits
    • Searches Script, Pencils, Inks, Colors, Letters, Editing and returns sequences.
    • Each field is also present as a separate, specific search zone.
    • [OPT] Instead of individual search zones, implement prefixes like
      • pencils:"Lou Fine"
      • script:"Grant Morrison"
      • Combined with the optional term parsing outlined below, this is much more flexible, allowing searches for things like "Kirby on pencils but not Lee on script":
        • pencils:"Jack Kirby" -script:"Stan Lee"
  • Stories (Sequences?)
    • Searches sequence title and feature. Returns sequences.
    • [OPT] Split this and provide separate search zones for titles and feature.
  • Characters
    • Searches feature and character appearances. Returns sequences.
    • Feature is searched only because so many sequences fail to list the feature character in the character appearances, or fail to list characters at all.
  • Reprints
    • Searches reprint notes. Returns sequences.

• [OPT] Parse terms as follows:
  • Split into words, search for any of the words.
  • Double quotes may be used to group phrases.
  • A + immediately before a term (word or phrase) requires it.
  • A - immediately before a term (word or phrase) excludes it.
  • [OPT] A ^ immediately before a term (word or phrase) indicates "starts with". Only one term may be starred.
  • [OPT] A ^^ immediately before a single term (word or phrase) indicates an exact match. Only one term may be present.
  • TBD: Escape sequences?
• TBD: Continue to have explicit "by name" vs "by year" sorting, or assume reasonable defaults depending on the search zone and allow the user to change the sort in the result set?

Narrowing Results

• [REQ] Code structure must support adding functionality of this sort.
• [OPT] Narrow with another search (same search box functionality), ANDed to existing search. Will require more details if this is to go into this release.
• [OPT] Broaden search by ORing new terms to the existing search. Will likewise require more details if pursued.

Advanced Search

• [OPT] Support searching every field to some degree.
• [NON] Complex searches along the lines of Bugzilla's Boolean charts should be deferred to a later release.

Internationalization

• All text encoded in UTF-8.
• Database migrated to UTF-8 encoding.
• Layouts friendly to LTR and RTL languages / bi-directional display
  • Bi-directional display will be needed if, for instance, the overall page is localized to an RTL language, but the user is viewing a record with data in an LTR language. Or vice versa.
  • CSS relating to any non-default justification / alignment must reside in a separate document that may be replaced in order to switch layout. TBD: Is this the correct way to handle this? Find examples.
• Layouts robust to character / word / phrase size changes.
  • Parts of the existing "new" layout rely on a specific width of text to avoid overflowing an area of the page. These assumptions should be removed.
• All label text should be set up for localization.
• TBD: Front page strategy
• TBD: Strategy for possible future article pages- not an issue for this release, though

Localization

• TBD: Need list of locales to be supported in this release, and translators for them.
  • German (Jochen Garcke)

Accessibility

• [REQ] Must support screen readers (alt text, labels, etc.)
• [REQ] Must render decently on an 80-column text browser.
• See also mobile browser support.

Branding

• [REQ] The GCD logo shall be displayed in the upper left-hand corner of every page (TBD: upper right-hand when localized to RTL languages)?

Advertising

• [REQ] Space must be available at the bottom of most pages for Google Ads, as seen on the current site.
• [REQ] Prominent space must be available on the home page for the CBLDF ad.
• TBD: Google ads at the bottom of the front page?

User Profiles and Preferences

• [REQ] Architectural support for user profiles and preferences (per user and per session).
• [OPT] The ability to set preferences.
  • Search result order?
  • Display options? (filter out certain types? filter out synopses?)

Workflow

Unlike in the current system, all records (publishers / imprints, series and issues) must be reserved for editing and go through the approvals process. This applies equally to new skeleton records, unindexed issues, or existing issues to be edited. Implementing this will allow indexers to do more work without sacrificing quality control.

Throughout this section, "users" refers to editors, indexers, and unregistered potential indexers as a whole. "Approver" refers to any user who can approve changes (typically editors). See the section on User Roles for the details of exactly who can do what within the system.

Reservations

Reserving a record should be easy, but all such reservations should expire within reasonable amounts of time. Indexers or editors who tie up records should be prevented from reserving more until their existing backlog is completed or released. This process should be flexible enough to handle the varying amounts of time indexers have available to contribute.

Individual Reservations

There will not be a separate reservations UI. Records (issues, series, publishers) can be reserved in several ways:

• Users can click the "Edit" button on the record's page.
  • Logged-in users get a confirmation dialog with the expiration date of the reservation, so that they understand they are reserving the record, and that it will be locked until they release it or until the reservation expires.
  • Anonymous users will be prompted to log in or register before seeing the confirmation box and being allowed to reserve / edit the record.
• Users can select records from a search results table and clicking a "Reserve" button. This will only be visible for logged-in users.
• Reservations initially expire after one month (but see extension functionality below).

Reservation and Record Creation

• Newly created records are automatically reserved by the record creator, as even skeltons should go through the approval process.
  • [OPT] A "create and submit" shortcut may be provided.

On-Going Reservations

• Series pages and results table entries will also allow the creation of an on-going reservation.
  • If a series has an on-going reservation, only the reservation holder may create add new issues.
  • Only new issues are automatically reserved by the on-going reservation. Existing issues are still open to the normal edit / reserve process.
  • On-going reservations expire every six months, but may be renewed during the last month of each period.
  • The expected frequency of a title over the next six months must be specified in the reservation (monthly, weekly, or simply an expected number of issues). This has to do with expirations and renewals, as described in the next section.

Expiration, Renewal and Extension of Reservations

• Reservations with no changes are immediately released upon expiration.
• Reservations with changes are not expired, but the user will not be allowed to reserve additional issues until all overdue reservations are submitted or released.
• Reservations may be extended at any time, up to the maximum reservation length of 3 months (from the original reservation date, not from the time of extension).
• [OPT] Email notification one week before the reservation expires.
• [OPT] Editors may extend reservations other than their own up to 6 months, to accommodate particularly complex projects like large anthologies.
• On-going series reservations may be renewed during the final month of each period.
  • If the indexer is more than one month behind, the reservation may not be renewed (i.e. more than one unsubmitted issue for a monthly (or any more rarely published series), more than 2 issues for a bi-monthly, more than 4 issues for a weekly, etc.)
  • [OPT] Email notification at the beginning of the final month of a registration period.

Reservations and Display

• Reserved records (individual or on-going) will show the reservation holder's name and the reservation's expiration date in place of the Edit or Reserve button.
• [OPT] A form for contacting the reservation holder without disclosing their email address may be provided.

Editing

Once a record has been reserved, it may be edited repeatedly until it is ready for submission for approval, or is abandoned and released for others to edit. In this section "Editor" always means the user role of Editor, not simply a generic user who is editing the record. Generic users are referred to simply as "users".

• Edit screens will be based on the display screens
• Edits will not be reflected in the display until after approval
  • [OPT] Editors may view a record as it is being edited, and when logged in they will see a button that allows this.
• The following functions will be available to the user while editing:
  • Submit for Approval
  • Save Changes (without submitting)
  • Discard all changes and release reservation
  • Extend reservation (up to the maximum duration, as described above)

Publisher Input Fields

The following fields have a simple text box for value input (starred fields are required before submission):

• Name*
• Year Began*
• Year Ended (linked to a checkbox marked "Active", and can only be filled out if "Active" is unchecked)
• URL (validated for syntax when saved)

The following fields are free-form text:

• Notes

The following fields must be validated against the database (either by using a select box of some sort, or by validating the contents when the field is saved):

• Country*
• Parent publisher

The following flags do not directly correspond to existing fields or database columns, but will be mapped into either the notes field or a migration column (see the Migrations section) behind the scenes. They are all off by default:

• Imprint (available only if a parent publisher is set)
• Indicia publisher
• Multiple indicia publishers
• Assigned name

In this UI (as will be made clear by the help text) the imprint flag indicates that this publisher is not only an imprint in our database (which just means that it has a parent publisher), but actually meets our (TBD) definition of an imprint, as opposed to just being a convenient place to shove indicia publishers or other corporate names that don't quite fit. Assigned name exists to flag publishers that don't correspond directly to any real-world corporate entity. The following table illustrates the use of flags (at least somewhat, as this is a contentious area- not all examples currently exist in the database):

Name	Has Parent?	Assigned Name?	Imprint?	Indicia Publisher?	Multiple Indicia Publishers?	Notes
Marvel	No	Yes	No	No	No	As far as I know, "Marvel", just one word, is not used in any indicia.
Timely	Yes	Yes	No	Yes	No	This is not the legal name of any company, but is the group name given to the earliest phase of what's referred to overal as Marvel.
Timely Comics, Inc.	Yes	No	No	Yes	No	Appeared in excatly this form the indicia of numerous issues.
Timely Publications: 1-20; Complete Photo Story: 21-68; Marjean Magazine: 69-75	Yes	No	No	No	Yes	Set of indicia publishers used for Captain America Comics. Flagging this separately lets us find these more easily when migrating later.
Vertigo	Yes	No	Yes	No	No	Vertigo is definitely an imprint. As for indicia publisher, casual survey of Vertigo issues and trades shows that they seem to use a DC corporate name for that.
Oni Press, Inc.	No	No	No	Yes	No	This is a real corporate name (so not an assigned name, even if it has child imprints).

Error Reporting

• [REQ] All pages must have a link to Error Tracker (Bugzilla)
• [REQ] All pages should have a link to the technical Bugzilla (TBD: Should we instead add a technical component to the Error Tracker? That would make end-user bug reporting much simpler. We could keep the technical Bugzilla internally.)
• [REQ] Links should always populate the URL field with the URL of the page holding the link, except for the the front page, which should leave the field blank. Any other administrative pages should also leave the field blank.
• [REQ] Links from pages representing publishers, series or issues should put those entities in the Summary field.
• [REQ] Links from cover galleries should auto-select the covers component instead of the default data errors component (users can still change it).

Migration Tools

• SQL script to drop unused colums and migrate user data in to Django tables.
• Python script to migrate passwords from plain text to encrypted.
• Reprint parsing will be off by default *unless* it can be enabled in such a way that Action Comics #1 loads within the performance requirements.
  • Users logged in with Editor or Administrator privileges will be able to turn on reprint parsing on a page-by-page basis.
  • Opportunistically: Users with Editor or Administrator privileges may be able to turn on the faster version of reprint parsing globally through a preference, which will be set to off by default.

Browser Support

Desktop Browsers

Desktop browsers are the initial primary audience.

• [REQ] IE >= 6.0 (Windows)
• [REQ] Firefox >= 2.0 (Windows, Mac OS, Linux)
• [REQ] Safari >= 2.0 (Windows, Mac OS)
• [OPT] Opera (Windows, Mac OS, Linux)
• [OPT] Konquerer (Linux)

Mobile Browsers

Mobile browsers must be kept in mind, but full / optimal support may be deferred to follow-on release.

• Opera ?
• [TBD: What is on iPhone, Blackberry, Treo?]

Performance

• Comparable to the current site when it seems snappy? TBD. Need actual numbers for both simple and complex pages.
• Caching
  • Use memcached? Hosting implications?
  • Need caching strategy (cache the whole site vs more granular strategy)

Test Infrastructure

• [REQ] Locate or write a Selenium test driver for Django (it seems highly likely that one exists somewhere already).
• [REQ] Classes to model the UI for Selenium tests.
  • Factor xpath or javascript access out of test code as much as possible for behavioral tests, while making the xpath clear for layout tests.
• [OPT] Locate or write a Nose test driver (Nose is an enhancement over the built-in unittest package, and ReviewBoard includes its own Nose driver which may be of use to us).
• [REQ] Build fixtures with data sets that load quickly enough to run the necessary tests in some reasonable amount of time. Obviously, this requirement needs some details.

Feature Specifications

Detailed feature specifications will be recorded in Bugzilla under the New Fun milestone so that they may be assigned to developers and tracked.