Test the Web Forward

Reftest Overview

  • Two files: Test file + Reference file
  • Test file uses the feature you're testing
  • Reference file is an exact visual match without using the feature you're testing
  • Self-describing (works both manually and with automation)
  • Cross-browser & cross-platform

Directory structure


  • CSSWG Test Repository
    • contributors
      • ttwf
        • your_csswg_id
          • incoming
          • submitted

Sample Test Case



Basic test for the transform property described in the CSS3 Transforms spec using the translate() function



Spec Links:

http://www.w3.org/TR/css3-transforms/#transform-property
http://www.w3.org/TR/css3-transforms/#two-d-transform-functions


Sample files


Required Metadata: Title


Test file

					

<!DOCTYPE html>
<html>
    <head>
    <title>CSS Transforms Test: transform property with translate function</title>
        . . .

Reference file

                    

<!DOCTYPE html>
<html>
    <head>
    <title>CSS Transforms Reference file</title>
        . . .

Required Metadata: Author


Test file

                    

<!DOCTYPE html>
<html>
<head>
    . . .			
    <link rel="author" title="Your Name" href="mailto:youremail@address.com">
    . . .

Reference file

                    

<!DOCTYPE html>
<html>
<head>
    . . .
    <link rel="author" title="Your Name" href="mailto:youremail@address.com">
    . . .

Required Metadata: Spec Links

Test file only

                    

<!DOCTYPE html>
<html>
<head>
  . . .
  <link rel="help" href="http://www.w3.org/TR/css3-transforms/#transform-property">
  <link rel="help" href="http://www.w3.org/TR/css3-transforms/#two-d-transform-functions">          
  . . .

Required Metadata: Reference File Path

Test file only

                    

<!DOCTYPE html>
<html>
<head>
    . . .   
    <link rel="match" href="reference/ttwf-reftest-tutorial-ref.html">
    . . .

Required Metadata: Assertion

Test file only

                        

<!DOCTYPE html>
<html>
<head>
. . .   
<meta name="assert" 
  content="This transform moves the element by 100 pixels in both the X and Y directions.">
. . .

The Test

Test element with the transform applied

    				    

                <style type="text/css">
                    .greenSquare {
                        position: absolute;
                        background: green;
                        top: 0;
                        left: 0;
                        width: 100px;
                        height: 100px;
                        transform: translate(100px,100px);
                    }
                    . . .
                </style>

Reference element

    				    

                    <style type="text/css">
                        .greenSquare {
                            position: absolute;
                            background: green;
                            top: 100px;
                            left: 100px;
                            width: 100px;
                            height: 100px;
                           
                        }
                    </style>

Visible failure

Element is only visible when the test fails

        				    

                        <style type="text/css">
                            . . .
                            .redSquare {
                                position: absolute;
                                background: red;
                                top: 101px;
                                left: 101px;
                                width: 98px;
                                height: 98px;
                            }
                        </style>

Failing test example


Self Description

Simple statement of how the page renders when the test passes


Test file & Reference file 

    				 

    <body>
        <p>The test passes if there is a green square and no red.</p>
        . . .
    </body>

Add the elements

Test file

    				 

                <body>
                    . . .
                    <div class="redSquare"></div>
                    <div class="greenSquare"></div>
                </body>

Reference file

    				 

                <body>
                    . . .
                    <div class="greenSquare"></div>
                </body>

Vendor prefixes


  • You may need to add a vendor prefix to the CSS property you're testing
  • If the prefix is necessary, your test will appear to fail without it
  • However, do not include vendor prefixes when pushing tests to the W3C

                      
                        
                .greenSquare {
                    position: absolute;
                    background: green;
                    top: 0;
                    left: 0;
                    width: 100px;
                    height: 100px;
                    -webkit-transform: translate(100px,100px);
                    -moz-transform: translate(100px,100px);
                    -ms-transform: translate(100px,100px);
                    -o-transform: translate(100px,100px);
                }

We recommend a script to use during test development:

http://leaverou.github.com/prefixfree

The prefixfree.jsapi.js extension can be used for unprefixed JS APIs tests:

https://github.com/LeaVerou/prefixfree/tree/gh-pages/plugins 

                      
  
        . . .
        
        <-- Remember to remove before pushing to W3C test suite -->
            
        <script src="prefixfree.js"></script>
        <script src="plugins/prefixfree.jsapi.js"></script>
        
        . . .

Reftest Q & A

JavaScript Test Overview


  • JavaScript tests programmatically verify functionality
  • JavaScript tests have some advantages over Reftests
    • Robust
    • Flexible
    • Performant
    • Powerful
  • Reftests have some advantages over JavaScript tests
    • Verify complete rendering
    • Easy to understand
    • Works for clients that don't support JavaScript
    • Avoid introducing additional technologies (JS)

JavaScript Test Overview (cont'd)


  • JavaScript can be used in conjunction with Reftests
    • Test can have both a reference file and a JavaScript section
    • Reftest portion supports clients without script engines
  • Self-describing Reftests are the preferred test format
  • JavaScript tests should be used in the following scenarios:
    • Testing a JS API or behavior (ex: CSSOM spec)
    • Adding automation to Reftests
  • W3C provides a framework (testharness.js) to simplify and standardize JS test creation

testharness

testharness.js and testharnessreport.js

    			            
    <script src="/resources/testharness.js"></script>

    <script src="/resources/testharnessreport.js"></script>
  • JavaScript API for making common test assertions
  • Should be used by all JavaScript tests
  • Path must be to /resources directory at root level
  • Element with id="log" should exist in test file
    • Test results table will be added to this element, if it exists

The test function


            			        
            test(test_function, name, properties)
  • Used for synchronous tests
    • async_test() is also provided by testharness for asychronous testing
  • Documented and implemented in testharness.js
  • Used to define an individual test case in a file
    • Common for a file to have multiple calls to test()
  • testharness will display PASS/FAIL for each call to test()

test() Argument 1: test_function

            			        
            test(test_function, name, properties)
  • test_function must be an object, not a function call
  • test_function will typically be one of the assert methods supplied by testharness
    • Can also be user defined if no assert method provides the needed functionality
  • Result from the assertion will determine PASS/FAIL
  • Usage example: 
                                            
            test(function() {assert_true(true)}, name, properties)

test() Argument 2: Name

                                
            test(test_function, name, properties)
  • name is a string that identifies the test
  • name should be short, unique, and must not change
  • name will be displayed in the test results table
  • Usage example: 
                                            
        test(function() {assert_true(true)}, "test_name", properties)

test() Argument 3: Properties

                                
            test(test_function, name, properties)
  • properties is an object that overrides defaults
  • This argument is optional
  • Recognized properties include timeout and metadata
  • Usage example: 
                                            
 test(function() { assert_true(true) }, "test_name", 

     {timeout:1000, 

     help: 'http://www.w3.org/TR/spec#section', 

     author: ['Bill Gates <bgates@msn.com>', 'Steve Jobs http://apple.com/sjobs'],

     assert: 'This test something.', 'This also tests something else.'})

Assert methods

  • testharness.js provides multiple assert methods
    • Full list is provided in the comment section of testharness.js
    • Typically supplied as the test_function for test()
  • All assert methods contain an optional description argument that is only output if the assertion fails
    • Used to provide additional debug information
  • All asserts must be located in a call to test()
    • asserts outside won't be detected correctly by the harness and may cause a file to stop testing

Assert Example

                                
        assert_equals(actual, expected, description)
  • actual: The actual value from the functionality being tested
  • expected: The expected value
  • description (optional): Output only if the assertion fails
  • Usage example: 
                                            
 assert_equals(getActualData("myElement"), 100, "Expected value for myElement is 100")

Sample Test Case


API test for the transform property described in the CSS3 Transforms spec using the translate() function



Spec Links:

http://www.w3.org/TR/css3-transforms/#transform-property
http://www.w3.org/TR/css3-transforms/#two-d-transform-functions

Metadata

        			            
<head>
  <title>CSS Transforms API Test: transform translate</title>
  <link rel="author" title="Your Name" href="mailto:youremail@address.com">
  <link rel="help" href="http://www.w3.org/TR/css3-transforms/#transform-property">
  <link rel="help" 
    href="http://www.w3.org/TR/css3-transforms/#two-d-transform-functions">
  <meta name="flags" content="dom">
  <meta name="assert" content="The transform should be translate(100px,100px)">	
...    
</head>
  • JavaScript tests have similar requirements as Reftests
  • dom token in flags metadata specifies that JavaScript is required
        			            
<head>
...    
    <script src="/resources/testharness.js"></script>
    <script src="/resources/testharnessreport.js"></script>    
</head>
  • testharness.js and testharnessreport.js imported
  • Path to both scripts points at /resources directory at root level
    • This is the required path before submitting to CSSWG Test Repository

Body

			            
<body>
    <div id="myDiv"></div>
    <div id="log"></div>    

    <script>
        // Set the transform 
        document.getElementById("myDiv").style.transform = "translate(100px,100px)";

        // Verify that the transform was set as expected
        test(function() {assert_equals(
            document.getElementById("myDiv").style.getPropertyCSSValue("transform").cssText, //Actual
            "translate(100px, 100px)",  //Expected
            "transform should be translate(100px,100px)")}, //Description
            "Transform_Translate_100px_100px"); //name
    </script>
</body>
  • myDiv div used as test element - translate applied in JS
  • log div used as container for test results
  • test() uses assert_equals as test_function
  • assert_equals actual argument supplied by API call

Test Results

PASS

Test Results

FAIL

Questions & Answers