Testing and analysing designs

Learn it by playing

Answer these questions to earn energy, then fish and explore. No account needed.

For teachers: ready-to-use lesson slides, revision notes, diagrams for Testing and analysing designs (NGSS Middle School Science, Engineering Design) — use them in your lesson, or run the topic as an interactive class activity your students play as a live game.

Lesson notes

Purpose of Testing Designs

Testing helps engineers **evaluate** how well a design meets criteria and constraints.
Data from tests reveal **strengths and weaknesses** of different solutions.
The goal is to identify the **best features** from multiple designs and combine them into an improved solution.
Testing provides evidence for **decision-making** in the engineering design process.

Key Variables in Experiments

**Independent variable**: the factor you change (e.g., material, shape).
**Dependent variable**: the outcome you measure (e.g., strength, speed).
**Control variables**: factors kept constant to ensure fair testing (e.g., temperature, size).
Changing only one independent variable at a time helps isolate its effect.

Designing a Fair Test

A fair test compares results under controlled conditions.
Use a **control group** or baseline for comparison (e.g., standard design).
**Random assignment** reduces bias by distributing unknown factors evenly.
**Repeated trials** improve reliability and reduce the impact of errors.
Document procedures carefully so others can **replicate** the experiment.

Collecting and Analyzing Data

Record **quantitative data** (numbers, measurements) and **qualitative data** (observations).
Use tables and graphs to organize data and spot patterns.
Calculate **averages** to summarize repeated trials.
Look for **trends** and **outliers** that might indicate errors or unexpected results.

Comparing Solutions

Compare test results against the **design criteria** (requirements) and **constraints** (limits).
Identify which design performs best on each criterion.
Combine the **best features** from different designs into an improved prototype.
Trade-offs may be necessary when no single design excels in all areas.

Iterative Improvement

Engineering design is **iterative**: test, analyze, modify, and retest.
Each cycle builds on previous results to refine the solution.
Failures provide valuable information for **redesign**.
The process continues until the design meets the criteria within constraints.

Validity, Reliability, and Replicability

**Validity**: the test measures what it intends to measure.
**Reliability**: consistent results across repeated tests.
**Replicability**: other teams can repeat the experiment and get similar results.
Clear documentation and controlled variables support all three.

A simple test circuit: changing the battery voltage (independent variable) changes bulb brightness (dependent variable).

Particle model of matter: solids, liquids, and gases. Testing designs often involves controlling temperature (a control variable).

Slides

Step through every slide for this topic — plus flashcards and revision notes — with a free account.

Practice questions

Free preview — 8 of 40 questions. Sign up to see them all.

1.What is the purpose of testing and analyzing designs in engineering?
Easy
- ATo identify the best features and combine them into an improved design
- BTo prove that the first design is always the best
- CTo make the design more expensive
- DTo avoid making any changes to the design
2.In an experiment, what is the variable that is changed by the experimenter called?
Easy
- AIndependent variable
- BDependent variable
- CControl variable
- DConstant variable
3.Which of the following is an example of a dependent variable in an experiment testing the strength of different bridge designs?
Easy
- AThe maximum load the bridge can hold
- BThe type of material used
- CThe length of the bridge
- DThe shape of the bridge
4.Why is it important to have a control group in an experiment?
Medium
- ATo provide a baseline for comparison
- BTo make the experiment more complicated
- CTo ensure all variables are changed
- DTo eliminate the need for replication
5.What does it mean if an experiment is 'replicable'?
Medium
- AOther scientists can repeat the experiment and get similar results
- BThe experiment can only be done once
- CThe results are always the same no matter what
- DThe experiment is very expensive
6.In a test of a new solar panel design, what is a control variable?
Medium
- AThe angle of the sun
- BThe amount of electricity generated
- CThe type of solar panel
- DThe temperature of the panel
7.A student tests three different paper airplane designs by throwing each one three times and measuring the distance flown. Which of the following is the best way to analyze the data?
Hard
- ACalculate the average distance for each design and compare the averages
- BUse only the longest flight for each design
- CIgnore the results that are very different from the others
- DAdd all distances together and divide by the number of designs
8.What is the main advantage of random assignment in an experiment?
Hard
- AIt helps reduce bias and confounding variables
- BIt guarantees the experiment will work
- CIt makes the experiment easier to conduct
- DIt ensures all participants are the same

Revision notes

## Testing and Analysing Designs — Revision Notes ### Key Concepts - **Design of Experiments (DOE)**: A procedure to explain how changes in one aspect of a system affect other aspects. It involves choosing **independent variables** (inputs you change), **dependent variables** (outputs you measure), and **control variables** (factors kept constant). - **Goal**: Analyse test data to identify the best features of different solutions and combine them into an improved design. ### Steps in Experimental Design 1. **Define the question** – What do you want to learn? 2. **Identify variables**: - **Independent variable**: The factor you change (e.g., material type). - **Dependent variable**: The outcome you measure (e.g., strength). - **Control variables**: Factors kept constant (e.g., size, shape). 3. **Plan the experiment**: Decide how many trials, randomize order, and use a control group if possible. 4. **Collect data**: Record measurements accurately. 5. **Analyse results**: Look for patterns, compare features, and decide which design elements work best. ### Important Principles - **Randomization**: Assigning treatments randomly to avoid bias. This helps ensure that differences are due to the variable you changed, not other factors. - **Replication**: Repeating trials to check consistency and reduce error. - **Comparison**: Compare your results to a baseline (control) or to other designs. - **Validity, Reliability, Replicability**: Design experiments so that results are trustworthy, consistent, and can be repeated by others. ### Improving Designs - Test multiple solutions and identify the **best features** from each. - Combine those features into a new, improved design. - Test the new design and repeat the cycle. ### Example - **Question**: Which bridge shape holds the most weight? - **Independent variable**: Shape (arch, beam, truss). - **Dependent variable**: Maximum weight held. - **Controls**: Same material, length, and testing method. - **Analysis**: Compare average weights; combine the strongest shape with other good features. ### Key Formulas/Definitions - No formulas required at this level. Focus on the process of testing, analysing, and improving designs. ### Tips for the Exam - Remember the three types of variables: independent, dependent, control. - Explain why randomization and replication are important. - Be able to describe how to use test data to improve a design.

Independent variable: The factor that is changed or manipulated in an experiment to observe its effect on the dependent variable.
Dependent variable: The factor that is measured or observed in an experiment; it is expected to change in response to the independent variable.
Control variable: A factor that is kept constant throughout an experiment to prevent it from affecting the results.
Randomization: The process of assigning treatments or subjects to groups by chance to reduce bias and confounding.
Replication: Repeating an experiment or trial multiple times to increase reliability and reduce the impact of random error.
Design of Experiments (DOE): A systematic method for planning experiments to determine how changes in independent variables affect dependent variables.

Flashcards

What is the main goal of testing and analyzing designs in engineering?

To identify the best features from different solutions and combine them into an improved design.

What does DOE stand for in engineering and science?

Design of Experiments.

Define independent variable in an experiment.

A variable that is changed or manipulated by the experimenter to observe its effect on the dependent variable.

Define dependent variable in an experiment.

The variable that is measured or observed; it is expected to change in response to the independent variable.

What is a control variable?

A variable that is kept constant during an experiment to prevent it from affecting the results.

Why is randomization important in experiments?

It helps reduce confounding by ensuring that differences between groups are due to the treatment, not other factors.

What is a quasi-experiment?

An experiment where natural conditions that influence variation are selected for observation, rather than directly introducing changes.

What are the three main concerns in experimental design?

Validity, reliability, and replicability.

What does 'replicability' mean in experiments?

The ability to repeat the experiment and obtain similar results, ensuring the findings are reliable.

What is a scientific control?

A baseline treatment or group that is not exposed to the experimental variable, used for comparison.

What is the purpose of comparing treatments in an experiment?

To determine which treatment is more effective by comparing results against a control or traditional treatment.

What is a 'design point' in DOE?

A unique combination of settings of the independent variables used in an experiment.

Who pioneered the methodology for designing experiments in agriculture?

Ronald Fisher.

What is the 'lady tasting tea' experiment?

A famous example by Ronald Fisher to test whether a lady could distinguish whether milk or tea was added first to a cup.

What is sequential analysis in experimental design?

A method where the design of each experiment may depend on the results of previous experiments, including the decision to stop.

What is a multi-armed bandit problem?

A type of sequential design where you choose among multiple options to maximize reward, balancing exploration and exploitation.

What does 'statistical power' mean in experiments?

The probability that the experiment will detect an effect if one exists.

What is 'sensitivity' in experimental design?

The ability of an experiment to detect small differences or changes in the dependent variable.

What is the difference between a true experiment and a quasi-experiment?

A true experiment uses random assignment; a quasi-experiment does not.

Why is documentation important in experiments?

To ensure the method is detailed enough for others to replicate the experiment.

What is a 'response variable'?

Another term for dependent variable; the outcome measured in an experiment.

What is a 'predictor variable'?

Another term for independent variable; the variable that is manipulated to predict changes.

What is the role of hypotheses in experimental design?

They guide decisions about which variables to change and control, based on expected sources of variance.

How can measurement error be reduced in experiments?

By carefully choosing and calibrating instruments, and using consistent procedures.

Unlock all 40 questions, flashcards & more

Create a free account to see every question, the slides, flashcards and revision notes for this topic.

Past papers

Past-paper practice for this topic is coming soon.

Coming soon