Category Archives: assessment

You Wouldn’t Believe The Dumb Answers I Get

The Setup

One time I made my students measure the heights of buildings around our school with an engineer’s transit. They had to use trigonometry.  I was a pretty great teacher so I made them take all of their measurements multiple times.  Man, they loved it.  It was a great activity because I give out points for each successful thing they do, which really motivates them. Everything is on a scale of five to ten (I have a progressive attitude about grades), so when a team gives an answer I give them some points, and at the end of the week we average them to get final grades. (side note: this is extra great because I can give students feedback pretty quickly)

Anyway, when they were done we did some serious analysis on the board. One team measured the gym, and they took four separate measurements. They found that the gym building was 40, 39, 42, and 39 feet tall. To really make the significance of that stick I made a spark graph.

I gave that team three 9/10s, one for each incorrect measurement, and one 10/10, for the correct measurement of 40 feet.

The other team measured the science building, and I guess they were slow or something because they only measured it twice – 19 and 21 feet. In the end it worked out because we could still make a spark graph. Two 9/10s.

I was starting to think that the spark graphs weren’t that useful, but luckily this was on a smart board so I could have one of the students come up and drag them around.

We figured out that the science building was about 20 feet tall and the gym was about 40 feet tall.

Here’s what I couldn’t believe, though.  I asked them what the average height was, and here’s what they did:

\frac {(19 + 21) + (40+39+42+39)} {6} = 33.33

Whoa guys! I know the scale is supposed to start at 5 but I’ve got to give a zero for that.  Very disappointing.  Are you sure you’re in the right class?

The Problem

The students couldn’t understand that the numbers they collected shouldn’t just be added up and averaged together.  I mean, you can add up numbers and average them, but you have to understand what you’re doing.  Averaging is an algorithm that really only works for equal, independent measurements of the same thing.  Obviously the average height of the buildings is 30 feet. We have to make sure that the numbers we’re sticking into our averaging algorithm are actually compatible.  Even though all six of our measurements were in feet… some are measurements of the science building and some are measurements of the gym.  If you average them you get the average measurement, not the average height!

The Punchline

Overall, I think the activity was a success.  After I added up the points everyone got during the day and averaged them together, everyone had over 90%!  Then I added up the points they got on homework and tests, and averaged that in too.  Finally, I added up all the points that everyone had earned in the whole week and averaged those together, and I had a final grade of B+ for the class overall – pretty good!

Afterwards, someone asked me what they needed to work on to improve.  I looked up their grade and saw they had an 85%, so I suggested they try to get more points the next time I asked a question. I love that self-motivation that points systems provide.

But I was the proudest when the director of maintenance heard that our class had been measuring the buildings.  He actually came into the class to ask some advice! He needed to get new ladders so that he could easily repair the roofs of the building, and almost my entire class could easily answer that he should get – you guessed it – the thirty-foot model!

Strategies for keeping active SBG cohesive

Does breaking the grade into 10 or 20 different topics help? or does it foster a reductionist attitude toward learning—that everything is discrete and independent of everything else?

Does allowing lots of reassessment help? Or does it focus kids on point chasing?

GSWP left these poignant questions about SBG for us, and today Sam Shah said that so far, the answers for his classes are bad.  To keep the punches coming, Alfie Kohn points us to “When Good Teaching Leads to Bad Results: The Disasters of ‘Well Taught’ Math Classes,” which concludes that “Despite gaining proficiency at certain kinds of procedures, the students gained at best a fragmented sense of the subject matter and understood few if any of the connections that tie together the procedures that they had studied.”

A scary and sad morning for people just starting to use active SBG!

Sam’s last point is that practice of active SBG needs to include some kind of protection against the choppiness that splitting your curriculum into discrete chunks brings.  If we assume that students will see their grades as the final result of our classes (a depressing but realistic assumption), what can we do with our grades to include learning and connections in them?

My Strategies

Here are two strategies I used to hold my courses together and fight tendencies of reductionism.

  1. I worked hard to reward students in ways that didn’t include grades, and heavily rewarded higher-level, holistic thinking.
    • Praise: “Great realization, John – that’s the real root (hah!) of the connection between factors and intercepts here.  This isn’t going to be on a quiz, but it’s one of the most beautiful parts of this stuff.”
    • Recognition: “And this is another case of what Sarah was describing before!”
    • Interest: “Whoa, how did you think of that?” followed by a 90-second conversation.  (Requires students to do something interesting, of course).

    This stuff felt cheesy at first, but I realized that if I only used it when it was really genuine, then, well, it would be really genuine.  Unfortunately, it tends to reward some students more than others, but we also give some kids As and some kids Cs, so I’ll leave that debate out of it.

  2. I reserved my highest grade for students who showed more complete understanding. At my school the highest grade was “honors,” but you could use “A” or “A+” or whatever.  If a student earned 100% on every skill in the whole course, he or she still wouldn’t earn the highest grade without completing a few projects that brought together a larger scale of knowledge.  This is, I admit, very un-SBG.  The effect was that students who just learned each concept individually and minimally could never get an A.  I made that very clear from the beginning, so that students wouldn’t be surprised, and I spaced the projects so that I introduced about one per month, keeping the emphasis on holistic learning throughout the year.I didn’t like requiring more work for the highest grade, but I did really like requiring a different magnitude of understanding.

SBG is no guard against point-chasing, and even active SBG has a lot of loopholes kids can exploit.  And, I mean, of course.  Teaching is freaking hard and teaching 80 kids at a time is even harder.  We’re set up to use grades as a reward by the system, and have to fight to keep them low in importance.  When grades are the reward, how can we really expect learning to be most important to all kids?

Strategy 2 above is the most practical – an administrator could just drop that into your class without caring if you’re a total grump with your kids.  But strategy 1 is really the most important, I think.  You know how I feel about personally connecting to our students already – I think it’s our only tool that actually does elevate learning over grades.  And with that our only tool, every point-chasing kid looks like a nail!

Active SBG

SBG is all about description and specificity, but “SBG” doesn’t describe what I’m talking about when I say “SBG.”  Here’s the problem:

“Standards-based” means “organized into topics,” but we’re doing more than that.  When we talk about letting students improve and show their improvement, we’re not just talking about organizing information.  When we talk about breaking kids of their addiction to points, we’re not talking about adding 20 columns to a gradebook.  When I talk about SBG, I’m talking about a philosophy of empowered students who have control of their education and their grades.

We implement this philosophy by organizing our feedback into helpful topics, making sure that our students can understand our feedback, and allowing students to react to that feedback.  You (not you, of course, but one) can implement SBG without any fundamental changes to your philosophy, and students in an SBGed course may still chase points, so “SBG” is not enough.  I think we need a new term.

“Active SBG” means:

  • In conversation with students, emphasizing the learning that grades represent, and trying to avoid holding grades as the final product of education.
  • Allowing students to react to their grades.  Grades are the beginning of a conversation, not the end.
  • Helping students to understand their grades by organizing them into topics (vanilla SBG).
  • Actively keeping students informed by assessing their skills often and giving them feedback as soon as possible.

After reading this list, can you see why “standards-based” wasn’t cutting it?  Joshg wonders “whether SBG really means anything without a slight philosophical shift,” and countless others blog about the “philosophy” of SBG.  “Active” is a great word to sum up the extensions that SBG needs to really shine – active student involvement, active feedback, reactive grades.

Most importantly, active SBG means that grades are used as one of the catalysts for learning in a class – that even though all that’s going on your report card is a single letter, the 100+ hours of imagination, concentration, and sweat are the real prize.

Letting Go of the Past

I’d like to give my students as many chances to learn as possible.  When they’re interested, I’d like to sit with them forever.  Unfortunately, there are some pretty significant logistical obstacles here.

Let's go over it again.

Almost everyone who disagreed with my automated debater for SBG+ (+remediation, +forgiveness of earlier scores, +timely & empowering reporting) disliked the idea of throwing out old assessment scores.  The most convincing criticism I’ve read is at GSWP (alternate link, scroll to bottom):

SBG aficionados believe in instantaneous noise-free measures of achievement.  If a student takes a long time before they “get it”, but then demonstrate mastery, that’s fine.  This results in the practice of replacing grades for a standard with the most recent one.  I think that is ok, as long as the standard keeps being assessed, but if you stop assessing a standard as soon as students have gotten a good enough score (which seems to be the usual way to handle it), then you have recorded their peak performance, not the best estimate of their current mastery.  Think about the fluctuations in stock prices:  the high for the year is rarely a good estimate of the current price, even if the prices have been generally going up.

The author at GSWP points out that averaging multiple assessment scores together works under the assumption that those different assessments were measuring the same thing, and that students’ skill levels are essentially unchanging.  I think this is why some of us have such a strong reaction against averaging.  We like to think that our students learn and improve so much during our class that the first assessments they take have almost no correlation to what they understand at the end of the class.

So.  Given that we have to choose a final score eventually, how do we do it?

I’ve written up a google doc with some ideas about different grade calculation methods. You can edit it by clicking here, or just read it (and what others have added) below.

*Update: argh, sorry, the “don’t require signin” button isn’t working. You’ll need a google account to edit & view. I guess Google is working on it.

I think my favorite (right now) is the decaying average, but I’ve never tried it on actual data. Please leave your thoughts in the doc (you can create a new table to include a different method altogether) or in the comments!

Two and a Half Ways to Make Your Next Test More Readable

Math questions are hard to read.  It’s easy to mix up numbers, mis-attribute modifiers, confuse powers with multipliers, etc.   Unless you’re testing for reading skill, it’s important to put work into making questions as readable as possible.  Here are two (and a half) ideas you can use on your next test.

One: Repeat the Question in an Answer Box

My test questions include space for work and a very specific location for answers: the Answer Box.

The answer box is the dark box you see in the lower-right corner of the question area.  I pre-print that box on every test question I ever give so that, regardless of the sloppiness of a student’s work, I can see what they thought the final answer should be.  I indicate the type of answer I expect right in the box.  My hope is that even a student with low English-reading skills will be guided by that indicator to provide at least the right type of answer.

Two: Provide a graphical representation with your text, where possible.

In this problem I wanted to test students’ ability to transform parabolas.  I accompany the text with a picture of what I mean.

Without the picture, “compressed by a factor of 2” is ambiguous.  When a student answers y=\frac 1 2 (x+2)^2 +1 I don’t want to have to guess if he doesn’t know what to do or if he just got right and left mixed up.  In the previous example, the MS clipart of a die reminds ELL students what a die is (not obvious from the word itself!).

Two and a Half: Easy Stuff

Actually, more like five tenths of a big idea:

  1. Use large fonts (obviously easier to read)
  2. Include grids on your graphs (the grids above are more obvious in print)
  3. Use the same format every time (students know what to expect even when they don’t command the language)
  4. Number and name questions with the standard they are testing (make double-sure ELL students know what you’re asking)
  5. Leave room between things (not only room for work.  Spaces help us separate ideas)

If you don’t make your questions crystal clear, you can’t be sure you’re testing what you think you’re testing.  Reading is hard, everyone messes it up occasionally, and students who have only been learning English for a couple years need more help than you think!

Of course, I’m just a math teacher, not a design expert or language coach.  What have you used to make your print materials more readable?

Team Tests

I administered my skills tests for the week today, but instead of making students complete the tests individually, they were encouraged to work within their team (of 3 or 4 students total).  Each student turned in a separate piece of paper.  I picked problem 1 from student A, problem 2 from student B, problem 3 from student C, and so forth, to determine the grade that all group members would receive.  Interesting.  Overall, I recommend you try it.


  1. A different format.  The kids were really in to it.  Afterwards, every kid I asked said something between “I liked it” and “It was way better.”
  2. Easy to administer and grade.
  3. I felt like I could put tougher problems on the test.  In fact, all of the problems I put on it were harder than usual, and the scores on this test were 10.6% higher than the scores on the rest of my tests.
  4. Students got practice communicating in a “high stakes” environment.
  5. It didn’t feel like I was spending the whole class on assessment.  There was also learning going on.  I heard things like, “this is the way I’ve started to think about this,” and “can we check this [answer]?”


  1. Tension was high, particularly in one group, when there were disagreements.  There was some snapping and flustered rustling of papers.
  2. Probably at least one student earned a grade higher than his understanding should indicate.

The pros are great, here.  Compared to the atmosphere during individual tests, today’s vibe was much more collegiate and… educational.  How can I mitigate the cons?  I could alternate individual and team tests.  I could talk with the students about strategies to work under pressure.  Would more time alleviate the tension?

Anyone else have experience with team tests?

Percentages don’t have the power to express a grade.

The traditional model for grades in a class lacks the flexibility required to reflect what I really think of a student.  When I used weighted categories (e.g. 50% exams, 30% homework, 20% class participation), I found that some of the students passing my class didn’t really seem to deserve it, and some of the students failing my class really should have been passing.  “Well, adjust your weights,” you say, and that’s a good idea: I made several improvements and was progressively more satisfied with my results.


One test of the 29 I’ll give this semester deals with simplifying exponential expressions.  If a student gets 100% on each test except the exponential simplification test, on which he gets a 0%, his average will be 96.5%, A+, Honors.  He doesn’t have to worry about exponential simplification at all, and he can just move on and never learn it.  I’m not suggesting that this hypothetical kid be made to retake Algebra 2, of course.  I’m suggesting that he be required to learn exponential simplification.

So, in my class, I require that every student earn at least a 3/5 on every single skill that we study.  Then, I require an overall average of 75% on top of that minimum requirement.  Students get some leeway, and they do not need to master every single subject (I understand that there are time constraints involved in my students lives, and that they may not really care about my class).  However, they can’t just do well on a lot of skills and decide they’re not even going to bother with one.  I am not willing to send a kid who can’t simplify exponential expressions at all to the precalc teacher.

The same philosophy can extend to homework.  If you think homework is vital, make it a requirement of passing.  If you don’t think it’s vital, don’t.  Averaging test scores with homework scores is harmful because it dilutes the meaning of your tests and the meaning of your homework.  Averaging mathematically destroys information!

At my school we only have three grades, Pass, No Pass, and Honors.  Each grade has certain clearly stated requirements that I give the students at the beginning of the semester.  I think that a teacher using letter grades could more clearly define what a C was and what an A was by stating the objectives vital for that award than he or she can by trying to come up with a formula to fit every student.  We shouldn’t be afraid to use some criteria that cannot be expressed with percents.

How to create a skills list

My last post focused on three major mistakes I made in my first semester of skill-focused, mastery-based assessment: separating skills into chunks that were impractically small, choosing some skills that were too simple (almost trivial), and neglecting to plan for the end of the semester.  This post will focus on my process creating the skills list for semester two (for Algebra 2).  I’d love to hear your opinions or your own process – leave a comment or a link below!

The place to start is your curriculum map, whether that’s a list of topics, a set of state standards, a final exam, some chapters of a textbook, or whatever.  Find or create the document that describes what you hope to teach this semester.  The list I used last semester I got directly out of my textbook.  I knew what chapters I was going to teach, and I just ripped concepts out of the table of contents.  This process got me a list that I was moderately happy with; click here to download it.  Your list will almost certainly need to be different, since I was planning for 36 class periods.

For semester two, I set out in much the same way.  I went through the chapters in my text book I was planning on studying, and every time something popped up that seemed like it would be a good candidate for a skill, I wrote it down.  This gave me the following list of 36 skills:

Evaluating functions
Analyzing the domain and range of functions
Modeling relationships with functions
Modeling arithmetic sequences with functions
Modeling geometric sequences with functions
Distinguishing between arithmetic and geometric sequences
Recognize exponential growth from situations, tables, graphs, or equations
Understand multiple representations of exponential functions
Represent exponential functions algebraically
Using basic laws of exponents to simplify expressions
Use exponential functions to solve problems involving growth or decay
Find equations of exponential functions through two given points
Identify graphs of quadratic, cubic, square root, absolute value, etc functions
Transform a graph by stretching, shifting, or flipping it
Write a general equation for a family of functions
Use the “completing the square” technique
Model physical situations with quadratic functions
Write equations in graphing form
Invert functions analytically and graphically
Form compositions of functions
Express the relationship between a function and its inverse
Understand logarithms and transform their graphs
Use properties of logarithms
Use logarithms to solve exponential equations
Count possibilities in situations that require a particular order
Count possibilities in situations in which order does not matter
Draw a tree diagram to represent and calculate probabilities
Draw an area diagram to represent and calculate probabilities
Calculate expected value
Using the fundamental principle of counting
Calculate conditional probabilities
Find the value of arithmetic series of arbitrary length
Find the value of geometric series of arbitrary length
Find the value of geometric series with infinite length
Writing a series with summation notation
Using mathematical induction

The next step in the process is to look at each skill from the brainstorm and ask,

  1. How will I test this skill?
  2. Is this skill big enough to be its own skill?
  3. Is this skill small enough to be a single skill?
  4. Does this skill have multiple levels, so that intro level tests will be significantly different from master level skills?
  5. If a student does not understand this skill at all, am I willing to flunk him? (My grades are set up that each student must get a minimum of 3/5 in every skill to pass the class.  If you’re using a simple average, you can ignore this question).

Consider the first skill, “Evaluating Functions.”

  1. How will I test this skill?  What leaps to mind is showing a kid f(x)=3x+6, and asking for f(2).  Maybe f(f(2)) – or is that composition?  They also need to be able to evaluate functions from graphs and tables.  Maybe the question should be a three-parter?
  2. Is this skill big enough to be its own skill?  Hmm… it’s pretty small, isn’t it?
  3. Is this skill small enough to be a single skill? Yes, I am confident that it is.
  4. Does this skill have different levels?  So, I could ask them to evaluate f(x)=3x+6, or f(x)=3x-2/x+(x+5)^-3, but those aren’t different levels of evaluating functions, those are different levels of order of operations or something.  I could ask them to find g(f(2)), but maybe that’s composition.
  5. Is this skill a requirement of passing the course?  Absolutely.  I am not letting anyone who can’t evaluate a function out of Algebra 2.

So this first skill has some complications.  I really like testing f(g(2)) because it requires students to understand the input/output aspect of functions where I feel like a simple g(2) might let them slip by without it.  Since this skill is so essential, I’m leaving it in, though it might be a little bit small.  It may end up conflicting with the composition skill, but there might be flexibility in that skill.  “Evaluating functions” seems like a solid requirement.

Let’s take another skill, “Modeling arithmetic sequences with functions.”

  1. How will I test this skill?  I’m looking for students to be able to come up with functions that describe arithmetic relationships, like, “write a function that outputs the number of gloves that x people will need,” or something.  I could show a table of inputs 1, 2, 3, 4, and outputs 8, 11, 14, 17, and have them write this function.
  2. Is this skill big enough to be on its own?  You know, I think it’s possible it could be combined with the skill before it, “modeling relationships with functions,” and the one after it, “modeling geometric sequences with functions.”  These three skills are so closely related, with the only difference being the arithmetic skills required.  I’m not trying to test those arithmetic skills – I hope the kids already have them – so I’m going to combine these three skills into just “modeling relationships with functions.”  I’ll answer the rest of these questions for the new skill.
  3. Is this skill small enough to be on its own?  Clearly the title can involve arbitrarily complex functions and relationships, but I think the kinds of simple relationships we’ll study in class can all be combined under one roof.  This skill may be a little bit too big.  If I was the organized man I wish I were, I’d note somewhere that I should revisit this skill after we touch on it to see how I felt.
  4. Are there different levels for this skill?  For intro level questions I can ask the students to model the relationship between Celsius and Fahrenheit, or some other linear relationship.  For master level questions I could ask a geometric volume question which includes an extra level of abstraction.
  5. Is this skill absolutely required for every student that passes the class?  Yes, I think so.  Who wants to teach precalc to students who can’t create their own functions?

Now, I don’t have time to write an essay about each of these questions for each of these skills, and neither do you.  In this post and in my brain I’m deciding to move quicklky here.  I only have so many hours to get this done, and it’s not going to be perfect.  I hope that you can make sacrifices like this – it’s taken me a long time to accept the impossibility of perfection in a finite time frame.  I spent about 30 minutes considering this list, eventually deciding to cut 7 skills and reword several.  Click here for my final checklist.

I hope this post has showed you how easy it can be to come up with a list of representative skills to assess.  It’s an unglamorous process, and the hardest part is coming up with the rough list, but once you do that you can have an effective list in less than an hour.  I don’t recommend using my skills lists wholesale.  I am in the process of trying out several different textbooks and my order is wonky.  My school is exempt from most standardized tests and if you have specific objectives you need to hit you’ll need to take them into consideration, obviously.  That said, here are my skills lists for Algebra II and Calculus this year:

Algebra II, Semester 1

Algebra II, Semester 2

Calculus, Semester 1

Calculus, Semester 2

Also, Dan Meyer has posted his suggestions for Algebra 1, Geometry, and Precalculus (imagine my chagrin when I saw he covered exactly everything but what I needed!).  So, if you’re considering getting started with this system, you at least have a launching point for your class.  If you have comments about my lists or process, I’d love to hear about them!  I’m especially interested what you think of the questions I use on each skill.  What else needs to be asked?

Mistakes I Made Creating a Skills Checklist

I made some rookie mistakes with my Algebra 2 skills checklist in semester one this year.  I have invented a word to describe each problem.


In my enthusiasm for separating skills, I gave determinants of 2×2 matrices and determinants of 3×3 matrices each their own spot on the list.  They should have been combined.  I also separated multiplication of polynomials from their division and even, astonishing in the euphoric clarity of hindsight, addition of matrices from their subtraction.  This overzealous separation led to inflated grades (students got a 5/5 on matrix addition, matrix subtraction, both kinds of matrix multiplication, AND twice for finding determinants) and tests that felt kind of… stupid.


A more subtle mistake: some skills on the checklist did not have discernible differences between intro-level problems and master-level problems.  For instance, an intro simplification test might look like “3x+2p-x+2x,” and a master might look like “3x+2p-x+2x-p+2p+r,” but for some students it felt silly to bother giving the intro level test first.  If a student can calculate “3+2+8+9,” do you really need to check to make sure they can calculate “3+2+8+9+1?”  In contrast, my favorite skills had some fundamental difference between the intro level and master level tests.  For example, the “dividing polynomials” intro test asked for a division that would have no remainder, and the master test involved remainders.  This was nice because a student could pass the class with a basic understanding of the concept, but would need a more advanced understanding before getting the 100%.

Bad Planning

I did not have an effective way to deliver master-level tests to the students who wanted them.  Early in the year I promised that students would be able to attempt a master-level test on any skill once per day, and early in the year this worked great.  Late in the year, when two thirds of my students wanted six tests a day, it was harder.  I need a better system.  Luckily, this won’t be a problem again until the end of this semester, so I’ll start thinking about it then.  😉

So, now the task is to create the lists for next semester (my school doesn’t start until January 10!), hoping to avoid these problems and keep the amazing benefits I got from the system last semester.  Stay tuned!

Involving students in assessment

The norm is for the teacher to write a test, lead a few lessons, describe what will be on the test, and then administer the test, right?  Students are not involved in making the assessment that will determine their grades.  My form of summative assessment changed drastically this year, but it still does not address this issue.  From what I hear from other teachers (you?), there’s a lot of room in all of our classrooms for more student involvement in assessment creation.

The tempered radical recently wrote about the benefits of explaining to students exactly what the point of each lesson is.  It’s the kind of thing that seems so obvious when you say it like that, but this is relatively new research pointing at this stuff.  In the November 2009 issue of Educational Leadership, “The Quest for Quality” references research from 2006 and 2009 to make the radical claim that “students learn best when they monitor and take responsibility for their own learning.”  It goes on to say “This means that teachers need to write learning targets in terms that students will understand.”  Sam Shah wrote about an experience talking directly with students about what it means to think and act like a mathematician that was so powerful for him that he considers it a genesis for himself as teacher.  And meanwhile, I think I’m totally rad for talking with students about what they want out of my class.

It’s incredible that this stuff is new, right?

Some teachers are on to this already.  I read about teachers developing rubrics with their classes, and others having students write questions from which the teacher will select his favorite three, etc.  These teachers are already reaping the benefits:

  • students feel (are) respected
  • students feel (have) ownership of the assessment, which gives them a new responsibility
  • students know a lot about the assessment before the lessons are all over, which seems, you know, better.

These benefits are obvious and supported by research.

So, if you use an assessment scheme based on written tests (like I do), what are the best ways to get some of these benefits?  I want to experiment with having kids write and critique their own questions for sure, since this seems easy to implement and, at its worst, is a form of review.  I already ask them to assess their progress towards their personal goals.  What else can I do?

I am hereby declaring a new goal for the first month of next semester: I will find a way to include each student in the act and process of his or her own assessment, at least a little.  I’m aiming high – I don’t mean that I will include “the class” in creating “the assessment.”  Whew.  There’s something to think about on the 14-hour drive home for Christmas!

Please leave comments if you have ideas.  I just set this kind of big goal, and to be honest, guys, I don’t know how I’m going to meet it yet.