Theme Review Scoring

Before describing how the scoring system works, I want to take a moment to clarify a few things. First, I don’t think this scoring system is perfect. Creating a theme review scoring system is a delicate balancing act between the overload of information that could be reviewed and graded and the limited amount of time I have available to perform theme reviews. Second, the theme review guidelines I follow are a subset of the guidelines used by the WordPress Theme Review Team when reviewing themes submitted to the WordPress.org repository. Third, a low review score does not necessarily indicate a poor quality theme. There are many great developers who choose to deviate from the guidelines set by the WordPress Theme Review Team.

The theme review score is made up of three distinct sections:

  1. Theme Check Requirements
  2. Theme Check Warnings
  3. Manual Review

The first two sections are based entirely on the output of the Theme-Check plugin. This plugin was built to aid the theme review process on WordPress.org and checks each theme against a large number of the theme review guidelines. This plugin is what makes these reviews possible. The third section is the manual review, which involves personally (as well as programmatically) reviewing some of guidelines that are not covered by the Theme-Check plugin.

Scoring Philosophy

Each theme starts with a perfect ten in each section. One point is deducted for each guideline the theme fails to follow. However, it is impossible to score below zero, even though a theme could potentially generate way more than ten REQUIRED or WARNING messages. The final review score is simply the average of the individual section scores.

Theme Check Requirements

The Theme-Check plugin outputs three different types of messages. The first is the REQUIRED message. This message highlights any theme review requirements not met by the theme. Examples include:

  • REQUIRED: Description: is missing from your style.css header.
  • REQUIRED: Themes should use add_theme_page() for adding admin pages.
  • REQUIRED: .bypostauthor css class is needed in your theme css.
  • REQUIRED: No content width has been defined.
  • REQUIRED: Could not find wp_link_pages.
  • REQUIRED: Could not find post_class.
  • REQUIRED: Could not find language_attributes.
  • REQUIRED: Could not find comment_form.

For each “REQUIRED” message generated, one point is subtracted from the theme check requirements score.

Theme Check Warnings

The second type of message is the WARNING message. (The third type of message is the RECOMMENDED message, which I don’t use in the theme reviews). The WARNING message covers issues such as:

  • WARNING: Found base64_encode. base64_encode() is not allowed.
  • WARNING: Found shell_exec. PHP system calls are often disabled by server admins and should not be in themes.
  • WARNING: Found ini_set. Themes should not change server PHP settings.
  • WARNING: fopen/fread/fwrite/readfile/curl_exec/etc. were found. File operations should use the WP_Filesystem methods instead of direct PHP filesystem calls.
  • WARNING: thumbs.db Windows thumbnail store found.

For each “WARNING” message generated, one point is subtracted from the theme check warnings score. However, I attempt to exclude warning messages related to the use of the “tag” section of the style.css, since those requirements only have to do with the categorization that WordPress.org implements in their directory. An example of a “tag” message that I try exclude would be this:

  • WARNING: The flexible-width and fixed-width tags changed to fluid-layout and fixed-layout tags in WordPress 3.8. Additionally, the responsive-layout tag was added. Please change to using one of the new tags.

All warnings not related to the WordPress.org tagging system I leave in place.

Manual Review

This is the most time consuming aspect of the theme review scoring, so I’ve tried to keep it to things that I can check efficiently. Here are the checks I perform:

  • HTML Validation using the W3C validation service.
  • Check that sticky posts are displayed first.
  • Check that items using the WordPress built in alignment classes are aligned properly.
  • Check that large images don’t overflow the main content area.
  • Check that floats at the end of the main content area are cleared properly.
  • Check for features that should be separated into plugins, based on the Plugin Territory section of the theme review guidelines. I specifically check for SEO options, content sharing buttons/links, default favicons, shortcodes, custom post types, and custom taxonomies.
  • Check for other issues.

The “Check for other issues” section at the end of the listĀ above is where I enter other issues I noticed when reviewing the theme. This could cover a number of different things, such as:

  • Debug messages that show up in the theme when WP_Debug is set to true.
  • Placing ads on the admin home page.
  • Changing the look of the WordPress admin without warning.
  • Installation problems

Really, it’s a place to mention any issue I run into that I think a theme buyer would like to know before purchasing the theme. Each issue I add to the “other issues” section deducts one point from the manual review score. Even though there are officially only eight checks in the manual review section, if there are multiple “other issues” it is possible to get a score of zero for this section. “Other issues” is the most subjective part of the entire theme review process, so I try to use it sparingly.

Discussion/Change

While I am open to discussing the review process, it becomes more difficult to make changes as more themes get added to the site. Nevertheless, as a result of feedback I have received, I have already made a few changes to the scoring system between when the site went live and when I sat down to write this post. If you feel that any of these review guidelines are unfair, or that a theme was unfairly scored, please feel free to use the contact form to send me a message.