Reftest Overview

Two files: Test file + Reference file

Test file uses the feature you're testing

Reference file is an exact visual match without using the feature you're testing

Self-describing (works both manually and with automation)

Cross-browser & cross-platform

Directory structure

CSSWG Test Repository

contributors

ttwf

your_csswg_id

incoming
submitted

Sample Test Case

Basic test for the transform property described in the CSS3 Transforms spec using the translate() function

Spec Links:

http://www.w3.org/TR/css3-transforms/#transform-property
http://www.w3.org/TR/css3-transforms/#two-d-transform-functions

Sample files

Required Metadata: Title

Test file

					

<!DOCTYPE html>
<html>
    <head>
    <title>CSS Transforms Test: transform property with translate function</title>
        . . .

Reference file

                    

<!DOCTYPE html>
<html>
    <head>
    <title>CSS Transforms Reference file</title>
        . . .

Required Metadata: Author

Test file

                    

<!DOCTYPE html>
<html>
<head>
    . . .			
    <link rel="author" title="Your Name" href="mailto:youremail@address.com">
    . . .

Reference file

                    

<!DOCTYPE html>
<html>
<head>
    . . .
    <link rel="author" title="Your Name" href="mailto:youremail@address.com">
    . . .

Required Metadata: Spec Links

Test file only

                    

<!DOCTYPE html>
<html>
<head>
  . . .
  <link rel="help" href="http://www.w3.org/TR/css3-transforms/#transform-property">
  <link rel="help" href="http://www.w3.org/TR/css3-transforms/#two-d-transform-functions">          
  . . .

Required Metadata: Reference File Path

Test file only

                    

<!DOCTYPE html>
<html>
<head>
    . . .   
    <link rel="match" href="reference/ttwf-reftest-tutorial-ref.html">
    . . .

Required Metadata: Assertion

Test file only

                        

<!DOCTYPE html>
<html>
<head>
. . .   
<meta name="assert" 
  content="This transform moves the element by 100 pixels in both the X and Y directions.">
. . .

The Test

Test element with the transform applied

    				    

                <style type="text/css">
                    .greenSquare {
                        position: absolute;
                        background: green;
                        top: 0;
                        left: 0;
                        width: 100px;
                        height: 100px;
                        transform: translate(100px,100px);
                    }
                    . . .
                </style>

Reference element

    				    

                    <style type="text/css">
                        .greenSquare {
                            position: absolute;
                            background: green;
                            top: 100px;
                            left: 100px;
                            width: 100px;
                            height: 100px;
                           
                        }
                    </style>

Visible failure

Element is only visible when the test fails

        				    

                        <style type="text/css">
                            . . .
                            .redSquare {
                                position: absolute;
                                background: red;
                                top: 101px;
                                left: 101px;
                                width: 98px;
                                height: 98px;
                            }
                        </style>

Failing test example

Self Description

Simple statement of how the page renders when the test passes

Test file & Reference file

    				 

    <body>
        <p>The test passes if there is a green square and no red.</p>
        . . .
    </body>

Add the elements

Test file

    				 

                <body>
                    . . .
                    <div class="redSquare"></div>
                    <div class="greenSquare"></div>
                </body>

Reference file

    				 

                <body>
                    . . .
                    <div class="greenSquare"></div>
                </body>

Vendor prefixes

You may need to add a vendor prefix to the CSS property you're testing

If the prefix is necessary, your test will appear to fail without it

However, do not include vendor prefixes when pushing tests to the W3C

                      
                        
                .greenSquare {
                    position: absolute;
                    background: green;
                    top: 0;
                    left: 0;
                    width: 100px;
                    height: 100px;
                    -webkit-transform: translate(100px,100px);
                    -moz-transform: translate(100px,100px);
                    -ms-transform: translate(100px,100px);
                    -o-transform: translate(100px,100px);
                }

We recommend a script to use during test development:

http://leaverou.github.com/prefixfree

The prefixfree.jsapi.js extension can be used for unprefixed JS APIs tests:

https://github.com/LeaVerou/prefixfree/tree/gh-pages/plugins

                      
  
        . . .
        
        <-- Remember to remove before pushing to W3C test suite -->
            
        <script src="prefixfree.js"></script>
        <script src="plugins/prefixfree.jsapi.js"></script>
        
        . . .

Reftest Q & A

JavaScript Test Overview

JavaScript tests programmatically verify functionality
JavaScript tests have some advantages over Reftests
- Robust
- Flexible
- Performant
- Powerful
Reftests have some advantages over JavaScript tests
- Verify complete rendering
- Easy to understand
- Works for clients that don't support JavaScript
- Avoid introducing additional technologies (JS)

JavaScript Test Overview (cont'd)

JavaScript can be used in conjunction with Reftests
- Test can have both a reference file and a JavaScript section
- Reftest portion supports clients without script engines
Self-describing Reftests are the preferred test format
JavaScript tests should be used in the following scenarios:
- Testing a JS API or behavior (ex: CSSOM spec)
- Adding automation to Reftests
W3C provides a framework (testharness.js) to simplify and standardize JS test creation

testharness

testharness.js and testharnessreport.js

    			            
    <script src="/resources/testharness.js"></script>

    <script src="/resources/testharnessreport.js"></script>

JavaScript API for making common test assertions
Should be used by all JavaScript tests
Path must be to /resources directory at root level
Element with id="log" should exist in test file
- Test results table will be added to this element, if it exists

The test function

            			        
            test(test_function, name, properties)

Used for synchronous tests
- async_test() is also provided by testharness for asychronous testing
Documented and implemented in testharness.js
Used to define an individual test case in a file
- Common for a file to have multiple calls to test()
testharness will display PASS/FAIL for each call to test()

test() Argument 1: test_function

            			        
            test(test_function, name, properties)

test_function must be an object, not a function call
test_function will typically be one of the assert methods supplied by testharness
- Can also be user defined if no assert method provides the needed functionality
Result from the assertion will determine PASS/FAIL

Usage example:

                                        
            test(function() {assert_true(true)}, name, properties)

test() Argument 2: Name

                                
            test(test_function, name, properties)

name is a string that identifies the test
name should be short, unique, and must not change
name will be displayed in the test results table

Usage example:

                                        
        test(function() {assert_true(true)}, "test_name", properties)

test() Argument 3: Properties

                                
            test(test_function, name, properties)

properties is an object that overrides defaults
This argument is optional
Recognized properties include timeout and metadata

Usage example:

                                        
 test(function() { assert_true(true) }, "test_name", 

 {timeout:1000, 

 help: 'http://www.w3.org/TR/spec#section', 

 author: ['Bill Gates <bgates@msn.com>', 'Steve Jobs http://apple.com/sjobs'],

 assert: 'This test something.', 'This also tests something else.'})

Assert methods

testharness.js provides multiple assert methods
- Full list is provided in the comment section of testharness.js
- Typically supplied as the test_function for test()
All assert methods contain an optional description argument that is only output if the assertion fails
- Used to provide additional debug information
All asserts must be located in a call to test()
- asserts outside won't be detected correctly by the harness and may cause a file to stop testing

Assert Example

                                
        assert_equals(actual, expected, description)

actual: The actual value from the functionality being tested
expected: The expected value
description (optional): Output only if the assertion fails

Usage example:

                                        
 assert_equals(getActualData("myElement"), 100, "Expected value for myElement is 100")

Sample Test Case

API test for the transform property described in the CSS3 Transforms spec using the translate() function

Spec Links:

http://www.w3.org/TR/css3-transforms/#transform-property
http://www.w3.org/TR/css3-transforms/#two-d-transform-functions

Metadata

        			            
<head>
  <title>CSS Transforms API Test: transform translate</title>
  <link rel="author" title="Your Name" href="mailto:youremail@address.com">
  <link rel="help" href="http://www.w3.org/TR/css3-transforms/#transform-property">
  <link rel="help" 
    href="http://www.w3.org/TR/css3-transforms/#two-d-transform-functions">
  <meta name="flags" content="dom">
  <meta name="assert" content="The transform should be translate(100px,100px)">	
...    
</head>

JavaScript tests have similar requirements as Reftests
dom token in flags metadata specifies that JavaScript is required

        			            
<head>
...    
    <script src="/resources/testharness.js"></script>
    <script src="/resources/testharnessreport.js"></script>    
</head>

testharness.js and testharnessreport.js imported
Path to both scripts points at /resources directory at root level
- This is the required path before submitting to CSSWG Test Repository

Body

			            
<body>
    <div id="myDiv"></div>
    <div id="log"></div>    

    <script>
        // Set the transform 
        document.getElementById("myDiv").style.transform = "translate(100px,100px)";

        // Verify that the transform was set as expected
        test(function() {assert_equals(
            document.getElementById("myDiv").style.getPropertyCSSValue("transform").cssText, //Actual
            "translate(100px, 100px)",  //Expected
            "transform should be translate(100px,100px)")}, //Description
            "Transform_Translate_100px_100px"); //name
    </script>
</body>

myDiv div used as test element - translate applied in JS
log div used as container for test results
test() uses assert_equals as test_function
assert_equals actual argument supplied by API call

Test Results

PASS

Test Results

FAIL

Test the Web Forward

Reftest Overview

Directory structure

Sample Test Case

Sample files

Required Metadata: Title

Required Metadata: Author

Required Metadata: Spec Links

Required Metadata: Reference File Path

Required Metadata: Assertion

The Test

Visible failure

Self Description

Add the elements

Vendor prefixes

Reftest Q & A

JavaScript Test Overview

JavaScript Test Overview (cont'd)

testharness

The test function

test() Argument 1: test_function

test() Argument 2: Name

test() Argument 3: Properties

Assert methods

Assert Example

Sample Test Case

Metadata

Body

Test Results

Test Results

Questions & Answers