W3C Test Writing Tutorials
Setup
Complete all steps before proceeding
Press down for additional steps
Reftest Overview
- Two files: Test file + Reference file
- Test file uses the feature you're testing
- Reference file is an exact visual match without using the feature you're testing
- Self-describing (works both manually and with automation)
- Cross-browser & cross-platform
Step 1 - OSX
Press down-arrow for Windows instructions
Update your local CSSWG test repository
Complete this step unless you just cloned the repository during setup
From a terminal window, type the following:
$ cd [path to repo]/test
$ hg pull -u
$ ls -lR contributors/ttwf/samples
Step 1 - Windows
Press up-arrow for OSX instructions
Update your local CSSWG test repository
Complete this step unless you just cloned the repository during setup
From a command prompt, type the following:
> cd [path to repo]\test
> hg pull -u
> dir contributors\ttwf\samples
Step 2 - OSX
Press down-arrow for Windows instructions
Create a directory for your new test
$ cd [path to repo]/test/contributors/ttwf/
$ mkdir -p yourName/incoming
$ cd yourName/incoming
Step 2 - Windows
Press up-arrow for OSX instructions
Create a directory for your new test
> cd [path to repo]\test\contributors\ttwf
> mkdir yourName\incoming
> cd yourName\incoming
Step 3 - OSX
Press down-arrow for Windows instructions
Copy the starter files to your incoming directory
From your incoming directory created in previous step
$ cp -R ../../samples/ttwf-reftest-tutorial-starter/* .
Step 3 - Windows
Press up-arrow for OSX instructions
Copy the starter files to your incoming directory
From your incoming directory created in previous step
> xcopy /e ..\..\samples\ttwf-reftest-tutorial-starter .
Step 4
Open the ttwf-reftest-tutorial-starter-001.html file in a editor
Step 5.1
Add the test metadata
Title & Author
<!DOCTYPE html>
<html>
<head>
<title>CSS Transforms Test: transform property with translate function</title>
<link rel="author" title="Your Name" href="mailto:youremail@address.com">
. . .
Step 5.2
Add the test metadata
Path to reference file
<!DOCTYPE html>
<html>
<head>
. . .
<link rel="match" href="reference/ttwf-reftest-tutorial-starter-ref.html">
. . .
Step 5.3
Add the test metadata
Spec links
<!DOCTYPE html>
<html>
<head>
. . .
<link rel="help" href="http://www.w3.org/TR/css3-transforms/#transform-property">
<link rel="help" href="http://www.w3.org/TR/css3-transforms/#two-d-transform-functions">
. . .
Step 6
Add the transform property to the test element
<style type="text/css">
.greenSquare {
position: absolute;
top: 0;
left: 0;
width: 100px;
height: 100px;
transform: translate(100px,100px);
}
. . .
</style>
Step 7
Create an element that is visible only upon failure
<style type="text/css">
. . .
.redSquare {
position: absolute;
width: 98px;
height: 98px;
top: 101px;
left: 101px;
}
</style>
Step 8.1
Add green fill color to denote passing condition
.greenSquare {
position: absolute;
top: 0;
left: 0;
width: 100px;
height: 100px;
transform: translate(100px,100px);
background: green;
}
Step 8.2
Add the red fill color to denote failing condition
.redSquare {
position: absolute;
width: 98px;
height: 98px;
top: 101px;
left: 101px;
background: red;
}
</style>
Step 9
Add a statement to the test page to make this a self-describing test
<body>
<p>The test passes if there is a green square and no red.</p>
. . .
</body>
Step 10
Open the reference/ttwf-reftest-tutorial-starter-ref.html file in an editor
Step 11
Add the reference metadata
Title & Author
<!DOCTYPE html>
<html>
<head>
<title>CSS Transforms Reference file</title>
<link rel="author" title="Your Name" href="mailto:youremail@address.com">
. . .
Step 12
Position and fill the reference element so it matches the test file exactly in a passing condition
<style type="text/css">
.greenSquare {
position: absolute;
width: 100px;
height: 100px;
top: 100px;
left: 100px;
background: green;
}
</style>
Step 13
Add the self-describing statement here, too
<body>
<p>The test passes if there is a green square and no red.</p>
. . .
</body>
Step 14
Examine your test and reference files in the browser and confirm they are identical
Failing test example
Vendor prefixes
- You may need to add a vendor prefix to the CSS property you're developing tests
- If the prefix is necessary, your test will appear to fail without it
- However, do not include vendor prefixes when pushing tests to the W3C
Press the down-arrow for a vendor-prefixed version of this tutorial
.greenSquare {
position: absolute;
top: 0;
left: 0;
width: 100px;
height: 100px;
-webkit-transform: translate(100px,100px);
-moz-transform: translate(100px,100px);
-ms-transform: translate(100px,100px);
-o-transform: translate(100px,100px);
background: green;
}
Press the down-arrow for an alternative to adding vendor prefixes manually
Step 15 - OSX
Press the down-arrow for Windows instructions
Add your new test & reference file to the repository
$ cd [path to repo]/test/contributors/ttwf/yourName/incoming
$ hg add ttwf-reftest-tutorial-001.html
$ hg add reference
$ hg commit -m "TTWF reftest tutorial"
$ hg push
Note: If you receive the following error:
abort: push creates new remote head ...
Your local repository is out of sync with the test repository
Do the following to resolve the issue:
Only complete these steps if you received the error message above
$ hg pull
$ hg merge
$ hg commit -m "Merge"
$ hg push
Step 15 - Windows
Press the up-arrow for OSX instructions
Add your new test & reference file to the repository
> cd [path to repo]\test\contributors\ttwf\yourName\incoming
> hg add ttwf-reftest-tutorial-001.html
> hg add reference
> hg commit -m "TTWF reftest tutorial"
> hg push
Note: If you receive the following error:
abort: push creates new remote head ...
Your local repository is out of sync with the test repository
Do the following to resolve the issue:
Only complete these steps if you received the error message above
> hg pull
> hg merge
> hg commit -m "Merge"
> hg push
Step 16
Ask the spec expert to review your test
Step 17 - OSX
Press down-arrow for Windows instructions
Incorporate feedback & move test to the submitted folder
$ cd [path to repo]/test/contributors/ttwf/yourName
$ hg pull -u
$ hg merge
$ hg commit -m "merging"
$ hg mv incoming/ttwf-reftest-tutorial-001.html submitted/ttwf-reftest-tutorial-001.html
$ hg mv incoming/reference/ttwf-reftest-tutorial-ref.html submitted/reference/ttwf-reftest-tutorial-ref.html
$ hg commit -m "moved the TTWF reftest tutorial to the submitted folder"
$ hg push
Step 17 - Windows
Press down-arrow for OSX instructions
Incorporate feedback & move test to the submitted folder
> cd [path to repo]\test\contributors\ttwf\yourName
> hg pull -u
> hg merge
> hg commit -m "merging"
> hg mv incoming\ttwf-reftest-tutorial-001.html submitted\ttwf-reftest-tutorial-001.html
> hg mv incoming\reference\ttwf-reftest-tutorial-ref.html submitted\reference\ttwf-reftest-tutorial-ref.html
> hg commit -m "moved the TTWF reftest tutorial to the submitted folder"
> hg push
W3C JavaScript Test Tutorial
JavaScript Test Overview
- JavaScript tests programmatically verify functionality
- JavaScript tests have some advantages over Reftests
- Robust
- Flexible
- Performant
- Powerful
- Reftests have some advantages over JavaScript tests
- Verify complete rendering
- Easy to understand
- Works for clients that don't support JavaScript
- Avoid introducing additional technologies (JS)
JavaScript Test Overview (cont'd)
- JavaScript can be used in conjunction with Reftests
- Test can have both a reference file and a JavaScript section
- Reftest portion supports suites without script engines
- JavaScript section adds automation
- Self-describing Reftests are the preferred test format
- JavaScript tests should be used in the following scenarios
- Testing a JavaScript API or behavior (ex: CSSOM spec)
- Adding automation to Reftests
- W3C provides a framework (testharness.js) to simplify and standardize JS test creation
testharness
testharness.js and testharnessreport.js
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
- JavaScript API for making common test assertions
- Documented in the comment section of testharness.js
- Should be used by all JavaScript tests
Press down for more
- Path must be to /resources directory at root level
- Locally, resources directory can be located anywhere
- Prior to pushing to CSSWG Test repository, the path must match example above
- Element with
id="log" should exist in test file
- Test results table will be added to this element, if it exists
The test function
test(test_function, name, properties)
- Used for synchronous tests
async_test() is also defined by testharness.js for asychronous testing
- Documented and implemented in testharness.js
- Used to define an individual test case in a file
- Common for a file to have multiple calls to
test()
- testharness.js will display PASS/FAIL for each call to
test()
Press down for more
test() Argument 1: test_function
test(test_function, name, properties)
test() Argument 2: Name
test(test_function, name, properties)
test() Argument 3: Properties
test(test_function, name, properties)
properties is an object that overrides defaultsproperties is an optional argument- Recognized properties include
timeout and metadata
- Usage example:
test(function() { assert_true(true) }, "test_name",
{timeout:1000,
help: 'http://www.w3.org/TR/spec#section',
author: ['Bill Gates <bgates@msn.com>', 'Steve Jobs http://apple.com/sjobs'],
assert: 'This should always be true.'})
For detail on how to define per-test metadata, see the CSSWG Script Test Wiki Page
Assert methods
- testharness.js provides a number of assert methods
- Full list is provided in the comment section of testharness.js
- Typically used in the
test_function for test()
- All assert methods contain an optional
description argument that is only output if the assertion fails
- Used to provide additional debug information
- All asserts must be located in a call to
test()
- asserts outside won't be detected correctly by the harness and may cause a file to stop testing
Press down for more
Assert Example
assert_equals(actual, expected, description)
Step 1
A JavaScript starter file was copied during Setup
Open the ttwf-js-tutorial-001.html file in an editor
<!DOCTYPE html>
<html>
<head>
<title>CSS Transforms API Test: transform translate</title>
<link rel="author" title="Your Name" href="mailto:youremail@address.com">
<link rel="help" href="http://www.w3.org/TR/css3-transforms/#transform-property">
<link rel="help" href="http://www.w3.org/TR/css3-transforms/#two-d-transform-functions">
<meta name="assert" content="The transform should be translate(100px,100px)">
</head>
<body>
</body>
</html>
Step 2
Update Metadata
- Update the
author metadata
- Add a line for the
flags metadata
flags document a test's prerequisites- Full list of
flags is available on the CSSWG Wiki Test Format page
- Example:
<title>CSS Transforms API Test: transform translate</title>
<link rel="author" title="Jim Tester" href="mailto:jimmyt@yahoo.com">
<link rel="help" href="http://www.w3.org/TR/css3-transforms/#transform-property">
<link rel="help" href="http://www.w3.org/TR/css3-transforms/#two-d-transform-functions">
<meta name="flags" content="dom">
<meta name="assert" content="The transform should be translate(100px,100px)">
Step 3
Import the W3C JavaScript Test Harness files
Add the following lines to the starter file above the style block
...
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
...
Note: During development, the path to the resources directory should match the directory's location on the local machine
The path to these scripts must point to the /resources directory before pushing to the CSSWG Test repository
Checkpoint
The head block of your starter file should appear as follows
With your actual name and email address for the author metadata
<!DOCTYPE html>
<html>
<head>
<title>CSS Transforms API Test: transform translate</title>
<link rel="author" title="Your Name" href="mailto:youremail@address.com">
<link rel="help" href="http://www.w3.org/TR/css3-transforms/#transform-property">
<link rel="help" href="http://www.w3.org/TR/css3-transforms/#two-d-transform-functions">
<meta name="flags" content="dom">
<meta name="assert" content="The transform should be translate(100px,100px)">
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
</head>
...
Step 4
Add a div to be used as the test element to the <body>
...
<body>
<div id="myDiv"></div>
<body>
...
The transform will be applied to this element
Step 5
Add a <div> with id="log" for test results
...
<body>
<div id="myDiv"></div>
<div id="log"></div>
<body>
...
testharness.js will add an HTML table that contains test results to this element
Step 6
Create a <script> block that will contain the JavaScript
...
<body>
<div id="myDiv"></div>
<div id="log"></div>
<script>
</script>
<body>
...
Checkpoint
Starter file should now appear as follows
<!DOCTYPE html>
<html>
<head>
<title>CSS Transforms API Test: transform translate</title>
<link rel="author" title="Your Name" href="mailto:youremail@address.com">
<link rel="help" href="http://www.w3.org/TR/css3-transforms/#transform-property">
<link rel="help" href="http://www.w3.org/TR/css3-transforms/#two-d-transform-functions">
<meta name="flags" content="dom">
<meta name="assert" content="The transform should be translate(100px,100px)">
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
</head>
<body>
<div id="myDiv"></div>
<div id="log"></div>
<script>
</script>
</body>
</html>
Step 7
Set the transform on the test element in JavaScript
...
<body>
<div id="myDiv"></div>
<div id="log"></div>
<script>
document.getElementById("myDiv").style.transform = "translate(100px,100px)";
</script>
</body>
</html>
Step 8
Add a call to test() to your script block
Use assert_equals inside test_function
...
<body>
<div id="myDiv"></div>
<div id="log"></div>
<script>
document.getElementById("myDiv").style.transform = "translate(100px,100px)";
test(function() {assert_equals(
document.getElementById("myDiv").style.getPropertyCSSValue("transform").cssText,
"translate(100px,100px)",
"transform should be translate(100px,100px)")},
"Transform_Translate_100px_100px");
</script>
</body>
</html>
Comments on test arguments added for clarity, can be omitted
Completed File
<!DOCTYPE html>
<html>
<head>
<title>CSS Transforms API Test: transform translate</title>
<link rel="author" title="Your Name" href="mailto:youremail@address.com">
<link rel="help" href="http://www.w3.org/TR/css3-transforms/#transform-property">
<link rel="help" href="http://www.w3.org/TR/css3-transforms/#two-d-transform-functions">
<meta name="flags" content="dom">
<meta name="assert" content="The transform should be translate(100px,100px)">
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
</head>
<body>
<div id="myDiv"></div>
<div id="log"></div>
<script>
document.getElementById("myDiv").style.transform = "translate(100px,100px)";
test(function() {assert_equals(
document.getElementById("myDiv").style.getPropertyCSSValue("transform").cssText,
"translate(100px,100px)", "transform should be translate(100px,100px)")},
"Transform_Translate_100px_100px");
</script>
</body>
</html>
Passing test example
Failing test example
Step 10 - OSX
Press the down-arrow for Windows instructions
Add your new test file to the repository
$ cd [path to repo]/test/contributors/ttwf/yourName/incoming
$ hg add ttwf-js-tutorial-001.html
$ hg commit -m "TTWF JavaScript tutorial"
$ hg push
If you receive the error:
abort: push creates new remote head ...
Your local repository is out of sync with the test repository. To resolve:
$ hg pull
$ hg merge
$ hg commit -m "Merge"
$ hg push
Step 10 - Windows
Press the up-arrow for OSX instructions
Add your new test file to the repository
> cd [path to repo]\test\contributors\ttwf\yourName\incoming
> hg add ttwf-js-tutorial-001.html
> hg commit -m "TTWF JavaScript tutorial"
> hg push
If you receive the error:
abort: push creates new remote head ...
Your local repository is out of sync with the test repository. To resolve:
> hg pull
> hg merge
> hg commit -m "Merge"
> hg push
Step 11
Ask the spec expert to review your test
Step 12 - OSX
Press down-arrow for Windows instructions
Incorporate feedback & move test to the submitted folder
$ cd [path to repo]/test/contributors/ttwf/yourName
$ hg mv incoming/ttwf-js-tutorial-001.html submitted/ttwf-js-tutorial-001.html
$ hg commit -m "moved the TTWF JavaScript tutorial to the submitted folder"
$ hg push
Step 12 - Windows
Press down-arrow for OSX instructions
Incorporate feedback & move test to the submitted folder
> cd [path to repo]\test\contributors\ttwf\yourName
> hg mv incoming\ttwf-js-tutorial-001.html submitted\ttwf-js-tutorial-001.html
> hg commit -m "moved the TTWF JavaScript tutorial to the submitted folder"
> hg push
Congratulations!
You have completed both a W3C Reftest and JavaScript Test
Mercurial Command Reference
- Basic Commands:
- hg pull (gets the latest changes from the CSSWG Test repo)
- To update after pull, use: hg pull -u (alleviates need to do hg update)
- hg update (applies the latest changes pulled to your local repo)
- hg status (displays list of locally changed files)
- hg add (stages a new or modified local file for commit)
- hg remove (stages the removal of a local file for commit)
- hg merge (merges local changes with updates pulled from CSSWG Test repo)
- hg commit (commits local changes to local repository)
- To include commit message, use: hg commit -m "Commit message here"
- hg push (pushes local changes to the CSSWG Test repository)
Press down for more
- User wants to update their local repo:
- hg pull -u (pulls and applies latest changes from CSSWG repo to local repo)
- User wants to push local changes:
- hg status (check which files are stages for commit)
- hg add fileName (stages file for commit, repeat for each file)
- hg status (confirm all files are stages for commit)
- hg commit -m "Commit message here" (Commit to local repo)
- hg push (pushes locally committed changes to CSSWG Test repo)
Press down for more
