Final Transcript

STATE OF FLORIDA: Race to the Top: Rule Development Workshop
February 14, 2013/4:00 p.m. EST

PRESENT IN THE MEETING ROOM

Kathy Hebda, Deputy Chancellor for Education Quality for the Department of Education
Eileen McDaniel, Bureau Chief for Educator Recruitment, Development, and Retention, Department of Education.

PRESENTATION

Coordinator	Good day, ladies and gentlemen. Welcome to the Rule Development Workshop conference call. My name is Anne and I will

be your coordinator for today’s call.  As a reminder, this conference is being recorded for replay purposes. At this time, all participants are in listen only mode. We will be facilitating a question session and a comment session during the workshop. I would now like to turn the presentation over to your host for today’s call, Kathy Hebda, Deputy Chancellor for Education Quality for the Department of Education. Please proceed.

K. Hebda	Thanks, Anne. Good afternoon, everyone. Welcome to the Rule Development Workshop where we’re collecting public input on Draft Rule Text for two State Board of Education Rules that are under development. The first is 6A-5.030 – District, Instructional Personnel, and School Administrator Evaluation Systems. The second is Rule 6A-5.0411 – Calculations of Student Learning Growth for Use in School Personnel Evaluations.

I have with me today, Eileen McDaniel from the Department of Education.

E. McDaniel	Good afternoon, everyone.

K. Hebda	Eileen is the Bureau Chief for Educator Recruitment, Development, and Retention.  I’m going to walk you through the process for today’s meeting. Then, we’ll talk about all the options for providing input as well.

We’ll have three parts for today’s meeting and for those of you that are in the room with us, welcome. We welcome the folks that are on the webinar and people that have just called in listening on the conference call line. If you are listening only on the conference call line, you can locate the PowerPoint that we used during this rule meeting at www.fldoe.org, by clicking on Educator Evaluations on the right hand side of the page.  For those of you in the room, I apologize for sitting down during the meeting.  It’s just that I’m running the webinar from the computer and that’s easier to do sitting down than it is standing up.

The first part of our meeting is an overview presentation of authorizing statutes and the content of the draft rules. We do this for two reasons: to provide people an opportunity to see what the law requires that we put into rule, and also so that we can provide you with an overview of what we’re intending to say in this draft text.  That’ll be important when you give us comments at the end about what it is we’ve said in these draft rules.

The second part, Clarifying Questions and Answers, is so any audience participant on the phone, via the web or in the room can ask clarifying questions. Questions about what’s in the law, questions about what’s in the rule text, questions about something you’ve heard during the presentation; anything that you need to clarify what you’ve heard or read before you make a comment. That part will be facilitated by the operator, Anne, if you’ve called in over the phone. We have a microphone here in the room for those wishing to speak. Even though we’re small room, for the sake of the recording of this webinar, we ask that you speak into the microphone. For those who are participating via webinar, I’ll be reading those questions aloud so that

everybody can hear them and then providing an answer if possible or, if necessary, taking questions back for a later response.

Then, when that part is concluded, we’ll have a part of public comments. At that point, we want the audience to feel free to provide the comments they wish on the rule aloud via the same method we talked about for Part Two, or in writing.  They can do so without being debated or responded to. Your comments will stand as you provide them.

So, those are the three parts. A little bit more about participation: If you wouldn’t mind when you’re asking a question or making a comment, if you’re doing so orally, please state your name and where you’re from, which can be your school district, your association, or any other information you want to provide about where you’re from. If you’re making a comment please follow the instructions with the operator if you’re on the phone. Via webinar, again, I’ll be reading your questions aloud.  If you’re here in person, if you would please fill out a speaker’s card to make a comment, that way we have

a good spelling of your name for the recording and the transcript.

Comments are going to be timed to allow who wish to speak the opportunity to do so, but only if we get toward the end of the meeting. We’ll ask Anne how many folks are in queue for comments and if it’s a lot of people in queue and we only have  a little bit of time, that would be the only reason we would time comments. Otherwise, we won’t put any time limits on anybody’s comments. Of course, we gladly take comments as written statements either today in writing, via the webinar, on the website provided here, again, fldoe.org and clicking on Educator Evaluations. You’ll go this URL you see here on Slide Three. There’s a button there where you can provide us your comments at any time.

There are two types of main input that we’re seeking today from everybody:  suggestions for changes or notes where you agree with something in the rule. We want to know what you think we should keep as well as what you’d like us to change. Changes can be comments that say “Keep this.”  “Delete a specific

passage.”  “Delete a section.” Add information that you don’t find in the rule that you think is important for us to add. Or, “A portion of this rule is not clear to me. Here’s a better way to word it, I think.”

You can provide us text on how you think we should work it to make things more clear. We have received some of those comments already from this morning’s workshop and we’re really, really pleased to have those. To the extent that you can provide a why or a context around what your suggested changes are, sometimes that’s helpful if that context or that reason why you’re suggesting something helps us find that similar kind of thing throughout the rule. We can also look for those things there.

Then, of course, we’d also like you to tell us whether or not the rule text actually accomplishes what the law tells us we have to put in the rule and what we’ve said in this presentation with our intent in the language. For example, we may have intended to say and we’ll say in our presentation a specific thing that has to be required in district evaluation systems that we’re looking for.

We intend to collect that evidence from districts in a certain way.  Then, by the time you go back and look at the rule text, you may say, “Well, that may be what you intended, but that’s not what you said, I don’t think.”  You can provide us that kind of commentary as well.  All of that is going to be helpful to us to improve the text of the rules.

So, what’s our timeline? We are now in February, so we’re having workshops. We have two today by webinar; our second one today.  Then, the week after next, we have those workshops around the state, five workshops around the state. In that same website that’s on the previous slide, on Slide Three, actually website at the bottom of the page, you can see all of those dates and locations and times for those workshops.

During the months of March and April when the workshops are over, we’ll be reviewing the public comments, monitoring legislative session for any bills that become signed into law that may affect these rules. Also, we will leave the public comment on the website open during that time.  So, even if we aren’t

having a face-to-face or a webinar workshop, you can still continue to make comments on line.

In May, once all of that is over with and we’ve collected all that public comment and we know what’s happened or not during legislative session, then we’ll revise the rule text from what you see today and we’ll provide it to the State Board for consideration. Our target date to present to the State Board is the month of June at their regularly scheduled June meeting. What we’d like to do then by having it looked at by the State Board in June is potentially then these rules would be effective for the ’13-’14 school year.  It wouldn’t effect evaluation systems that are in place now for ’12-’13, but they would affect ’13-’14 evaluations the way the text is currently drafted.

So, with that, we’ll begin the first rule. We’ll take these rules in order.  First rule, 6A-5.030 is a rule that governs review and approval of district evaluation systems. On Slide Seven, the first thing you see is what the law says we’re supposed to put into this rule.

We are supposed to establish uniform procedures for submission, review, and approval of district evaluation systems. Those are instructional personnel and school administrator systems. We’re supposed to explain the reporting requirements for annual evaluations of instructional personnel and administrators. We’re supposed to explain a process for monitoring school district implementation of evaluation systems in accordance with this section of statute.

I stopped on this slide for a few minutes because there are several sections of the rule the way it’s currently drafted for your consideration that pertain to these specific things.  For example, the first two sections of the rule sections (1) and (2) actually speak to how evaluation systems are submitted by school districts; and, what definitions are used throughout the rule. Sections (4) and (5) of the rule talk about how district evaluation systems can be rated for evaluation either fully approved, conditionally approved or denied, under what circumstances, how could a district system that’s been conditionally approved,

for example, if it becomes fully approved. Those are considered in sections (4) and (5).

Then, section (6) has to do with how districts can submit modifications to their evaluation systems because the law says that if the district substantially modifies their evaluation system, it needs to come back for review and approval.  So, we’ve tried to provide in the rule what does it mean to have substantially modified an evaluation system.

Section (7) of the rule actually relates to the monitoring system that the law says we need to establish. We’ve established the monitoring process based on how we have been reviewing district professional development systems for the past ten years. So, we do those on a five-year cycle meaning we go on- site to every district once every five years and have a full review of their professional development system. We are proposing  for your consideration in this rule that we would do the same kind of process for evaluation systems.

Annual reporting requirements are in section (8) of the rule.  The last section of the rule, section (9), provides guidelines for training evaluators. One of the things the law says is that districts have to have requirements in their system for how they’re going to train their own evaluators in their district.  But, those training processes have to be based on consideration of guidelines provided by the Department. So, we’ve added those guidelines for district consideration to the rule.

The remaining slides talk about the content of the district evaluation system as described in the law.  This all goes into section (3) of the rule. The first thing the law says is that for the purpose of district educator evaluation systems they’re designed to afford effective instruction of student learning. The results are supposed to be used in districts for developing school level improvement plans. And, the results are supposed to be used to identify professional development. Those things are important because there are some things in the rules that are particularly related to professional development and feedback systems that are included in evaluation systems.

Also, the law says evaluation systems have to differentiate among four levels of performance: highly effective, effective, needs improvement or developing, and unsatisfactory.  The law also says the State Board is supposed to establish student growth standards...performance level and the Commissioner should consult with experts and instructional personnel, school administrators, and other stakeholders in developing those criteria.  Since the criteria are going into rule, that’s what we’re doing now with these workshops.

The law also says the districts’ evaluation systems should be considered in three separate areas of criteria:  performance of students, instructional practice for instructional personnel evaluation or instructional leadership for school administrator evaluations, and professional and job responsibilities.  It also indicates that these evaluations should be based on sound educational principles and contemporary research in effective practices.

The law sets forth what instructional practice is supposed to include. You can see here on the slide it’s the Florida Educator

Accomplished Practices for everybody; all instructional personnel. Then for instructional personnel who are not classroom teachers, the district may also include specific job expectations that are related to student support for that person’s job. The expectation is that teachers can increase their expertise from year-to-year producing gains in student achievement with a powerful cumulative effect.  Remember, we go back to the first evaluation system that’s supposed to help improve instructional practice and student learning growth. There has to be a feedback system where teachers can work on their own practice.

...what the law says about instructional leadership, again based on the state’s principle leadership standards in this case, for instructional leaders. Then, the law prescribes three things that relate to how the school principal and school administrator develops his or her faculty and supports effective instruction in the school.

The next area of criteria in the evaluation system is performance of students.  In summary, the performance of

students in the law it says that at least 50% of the performance evaluation must be based upon data indicators that student learning growth assessed annually and measured by statewide assessments; or, for subject and grade levels not measured by statewide assessment by district assessment as provided in the section of law that governs student assessment programs.

In addition in the summary, the law says that the overall percentage for any individual can be reduced to 40% in the absence of three years of student data. The percentage for non-classroom instructional personnel can be flipped between statewide assessment results and other local measures of student outcome determined by the district. And, districts have options to combine state and local assessments for those teachers who teach local assessment subjects and courses, and can choose performance measures or achievement measures over learning growth measures for those courses when they are more appropriate.

The last section on performance of students that is quoted here on Slide 14 is the section because of the timeline of the law by

which evaluations are fully developed indicates that classroom teachers of courses where the district has not implemented an assessment under 1008.22 for any grade or subject that the teacher teaches would then outline other measures of student assessment that should or could be used, including as the next to the last sentence relates. A superintendent can assign instructional personnel team data to an individual instructional personnel individual for purposes of student learning growth.

Slide 15 shows you the list of everything else in the law that is to be included in the district evaluation system. Again, all of these things are captured in section (3) of the rule.  Observation instruments with indicators, two evaluations per year for newly hired personnel, evaluator training programs that we mentioned earlier, the method of informing personnel in the district about the evaluation system, multiple data sources that are to be used in the evaluation system, how the system data is used for professional development and school improvement as we saw earlier, parental input, teaching fields that need special procedures, the process the district would use for annual review of its evaluation system, and then additional options for districts

to include if they choose to such as peer review, assistance processes, input from additional personnel into the evaluation, evaluation amendments, and additional professional responsibilities beyond State Board rule.

The professional responsibility of the State Board rule considers those would be for the Code of Ethics and the Principles of Professional Conduct. Again, all of this is the information that section (3) explains how we collect evidence...to make sure that we can approve the district evaluation system.

Then, we have the second rule, Rule 6A-5.0411 – Calculations of Student Learning Growth for Use in School Personnel Evaluations. This is what the law says needs to be codified into rule regarding this section. This rule contains the formula, the actual mathematical calculations for measuring student learning growth approved by the Commissioner for any statewide assessment.  At this point in time, we only have one of those, which is the one approved for use of FCAT data. So, this rule only contains one approved learning growth formula.

The measurement of student learning growth and the associated implementation procedures, which means how do we actually provide that information to school districts within the calculations, all of that; how does the model work. Then, process by which teachers may review their class rosters. In addition, this rule needs to contain specific discrete standards for each performance level.  The performance levels, as you may remember, are highly effective, effective, needs improvement/developing, and unsatisfactory that provide consistency and meaning across school districts.

In addition to providing those standards, we have to establish a standard for student learning growth that if not met would result in the employee receiving an unsatisfactory performance evaluation rating. We have to establish student learning growth standards that must be met in order for an employee receive or be considered highly effective or effective by the district. So, we’ll take those things one at a time.

The law talks about two types of student learning growth measures – two categories of student learning growth measures. Those that are associated with statewide assessments that have to be approved by the Commissioner. Once they are, then everybody uses them together; all districts use them; then, learning growth formulas for local assessments.

The law says that districts need to adopt equally appropriate learning growth formulas for local student assessments. The Department’s responsibility in that process is only to provide example models and technical assistance. But, the decision on what those growth formulas are is the district’s decision.

Because the calculation for student learning growth has to be in the rule, then we’ve added a few slides that remind everybody or present to everybody if you haven’t seen these before – some of you have probably seen them a hundred times – but, these are the slides that explain the FCAT student growth model that was approved by the Commissioner because that actual calculation is in the rule.

So, a little background: It was determined by the Student Growth Implementation Committee and recommended to the Commissioner that the model selected for measuring student growth for FCAT data would be a Value-Added Model. This gives you a little reminder of who is in the Student Growth Implementation Committee. On the website, you can actually see the names and the districts and all of the teachers, administrators, higher ed faculty, business and parent representatives that are represented on the Student Growth Implementation Committee.

The model they selected was a covariant adjustment model. That’s described in the rule, which means that we have a model that begins by establishing expected growth or performance levels for each individual student that’s assigned to a teacher. That growth expectation is based on historical data meaning how the student has performed in the same assessment in the past; then, looking at the characteristics that are included in the model as recommended by the Committee and approved by the Commissioner, how students that share those similar

characteristics actually perform. What that does because all those things influence student learning growth, then it permits the model to determine the amount of learning growth contributed by the teacher.

This is the graphic example on Slide 23 of what that might look like for an individual student with a prior year performance, an actual performance in the current year, and then there’s a predicted performance based on the things we know about the student: their prior performance history and all the characteristics included in the model. The difference between the predicted performance and the actual performance, because the predicted performance represents everything already accounted for that could influence the student’s learning, the difference is what’s attributed to the teacher. That’s what the teacher adds to student learning growth for each individual student.  So, all the students that are assigned to the teacher, those differences are rolled up together and that produces the teacher’s value-added score for all the students assigned to him or her.

These are all the factors that are included in the model that also have been determined to contribute to student learning and they’re described in the rule. We have student characteristics, which you see here:  up to two prior years of achievement scores, which is in all the literature the strongest predictor of student learning growth and it’s the one factor required in law to be included in the model; the number of subject relevant courses, which means there are some students that are enrolled in multiple courses that could be associated with reading, language arts, or mathematics; the student’s disability status, which is actually based on the individual disability that the student has; English Language Learner status; attendance, which is the number of days attended throughout the entire year; gifted status; mobility, which is the number of transitions into and out of school during the year whether it’s to a different school or even back and forth in the same school; and the difference from modal age and grade as an indicator of retention or promotion or any other reason that could mean that the student is significantly different in age than the other kids in the class.

All of those characteristics each year are provided in the technical paper that shows how the model is calculated and is very transparent about that.  You can see the coefficient and how much difference it makes.  For example, that’s what the coefficient is how much difference does it make that the student had a specific disability, or that the student attended only so many days a year, or the student was gifted, or the student had a certain number of transitions. Those things are calculated to provide and feed into that predicted score.

The classroom characteristics that are included are class size, homogeneity of students’ entering test scores. That was brought up by the Committee because the Committee...looked at data. So, they included this in the model. It does make a difference in some cases whether or not the teacher has similarly performing students in his or her class, whether at a high level or a low level, or whether the teacher has a mix of student performance levels in her class.

Then, school characteristics: It was clear looking at the data to the Committee when they made the recommendation to the

Commissioner that there were differences, significant differences, in the performance of some schools overall; some very high performing or high growth schools, and some low performing or low growth schools. In those cases, the Committee was convinced that there were factors in that school that could also influence student learning beyond what the teacher could control, and beyond even what could be captured in some of the student characteristics that are already provided. So, the Committee determined that we should also consider the performance of the school in the model.

This is just a reminder that the Value-Added Model and the student learning growth portion of the evaluation system is just one part of a multi-faceted evaluation system and one contributing factor for the ultimate result.  In the last bullet here, it talks about an ongoing process as represented in the rule in the fact that we provided for an annual review of results from all statewide adopted models.

The next part of the rule talks about what the results are and what the data points are that you receive when you do this

calculation, the specific learning growth calculations. So, we receive results from the model formula produces a value-added score, which we’ve been talking about that is the average amount, essentially, of the learning growth of all the students assigned to the teacher.  Then, what does that score mean?

A score of zero in this case actually means that those students on average performed just like other students around the state in that grade and subject that shared those same characteristics. So, zero means students performed on average as expected. If the score is above zero, a positive score would indicate the student outperformed their expectations on average.  Or, a score below zero, which would be a negative score, would indicate the student did not perform as well as expected on average.

In addition to the value-added score, the model also yields some other pieces of information. We also note the number and percent of students assigned to each teacher or school if it’s the principal that met or exceeded their statistical performance expectations. So, though these data don’t provide

information on how far students may have improved, how far they exceeded the expectations, or how far it was that they missed the expectations, it does provide information on the quantity of students who met their expectations and can be a good piece of information for a teacher to know or a school principal to know to see how many students in the class or how many students in the school are being reached to meet their goals.

The other thing that we get from the model, including the estimate of the teacher’s impact on student learning, is the fact that that score contains some variability.  This is the statistical process. Some of you may be familiar with things you’ve seen on television such as opinion polls and presidential popularity polls where you see the result is of all the persons polled in these five states who are registered voters, the president’s popularity, for example, is 50% plus or minus three points.  What that means is if the polling company has polled a similar group of people, not the same people, but a similar group of people meaning the same five states, also registered voters, all those things, then they are sure that the result would have been

the same within three points on either side of that 50%; so, somewhere between 47% and 53%.

The way this relates to the standard error and the confidence interval that are constructed using that standard error, that plus or minus three, or plus or minus whatever the standard error is for that score and its teacher value-added score, means that it helps the district construct a confidence interval around the score to say if this teacher had been assigned another group of students that were similar in nature in a different school year or even in the same school year, then we have this much certainty inside that confidence interval that the result would have been the same.  It helps the school district use this one component of the evaluation system to add accuracy to the classification of performance for that teacher. Those are the pieces of the rule that pertain to just what’s in the calculation, how does it work, what’s the mathematical formula itself, and all the data that are yielded from the Value-Added Model for FCAT. Remember, that’s the one model we have adopted so far into rule.

The next section actually goes back to – I’m going to page back a few slides just to remind everyone – the next section of the rule actually talks about the performance standards. We said back on Slide 18 that in addition to putting the calculation itself in the rule and how the calculation works that we also have to establish performance standards in this rule. Based on the input we got last June, the first time we did workshops around this rule, we’ve established some performance standards in this draft rule for your consideration. Part of your input that we want from you over the next month or so is what you think about these performance standards themselves. Are the standards too low, too high, just right; or/and, whether we are applying the standards at the right time to the right individuals within the evaluation system.

Given that, remember on the slide, the law talks about two things. It talks about establishing performance levels for the sake of consistency in the evaluation systems from district to district. Then, it talks about establishing performance standards for use in determining an overall unsatisfactory evaluation or a threshold for the individual that would need to meet to be

considered by the district as effective or highly effective.  So, we have two reasons for doing the standards: consistence, number one, and then establishing the use of the standards under certain circumstances for impacting the summative rating.

Let’s talk about what the rule says. What we’ve tried to do in the rule, and again, we want your opinion on whether or not we’ve succeeded in doing this in addition to whether or not you think this is the right thing to do in the rule. The first thing is we were trying to establish the performance standards based on ’11-’12 data.  One of the things that many of you may know about value-added models or covariant adjusted models or any kind of learning growth models is that they’re normed just like you’re used to percentile rankings and things like that that are on normed scales.

In this case, every year when the Value-Added Model is run, you get a distribution of scores. That’s why zero in the middle is typical growth.  Half the folks are on one side of zero and half the folks are on the other side of zero.  But, what was

concerning to us based on feedback and experience that we’ve had over the years with performance pay and other things was we wanted to see and make sure in the evaluation system that every year we weren’t just producing a new rank; that, in fact, we had criterion reference standards that everybody could eventually exceed each year.

So, we know that even though we get a distribution each year what we’ve attempted to do in the rule is anchor and provide benchmarks based on the average growth by grade and subject during the ’11-’12 school year.  It would become the benchmark so that every year when the model is calculated, that average learning growth by grade and subject in the first year would be the bench mark that everybody could try to exceed in future years.

First thing is we would establish these performance standards based on the ’11-’12 average growth by grade and subject. We also note here that the aggregate VAM score, which means that for a teacher who, for example, teaches three classes of 6th grade reading and three classes of 7th grade reading, the data

that we provide the districts every year would give a Value- Added Model score, a result, for 6th grade reading for that teacher and one for 7th grade reading for that teacher. Then, we would provide an aggregate score that would take all of
those and put them together.

The aggregate score is converted to a proportion of that average year’s growth so that it has some kind of context and some kind of meaning. If your value-added score is, for example, positive; it’s above zero; it’s 0.2 that means your students on average grew 20%, 0.2, 20% greater than typical for students in that grade and subject across the state.  So, the score is converted into a proportion of the year’s growth.

Districts, as I mentioned, would then – the intent is the district would apply those standards each year using the confidence intervals that are constructed from the standard error to ensure that the score that’s received would be a typical score for that teacher given another set of students. Then, we would be using these beginning in the ’13-’14 school year.  So, even if the State Board adopts this rule in June, which would still be

considered the ’12-’13 school year, these requirements would not go into effect for performance evaluations this year, but would go into effect, the way the text is constructed now, for next year.

The other thing I should mention on this page before I go to the next page is that these standards would be used to apply just the performance of students’ criteria beginning next year. Remember, the evaluation has three parts: performance of students, instructional practice, and professional and job responsibilities. Some districts merge professional responsibilities with instructional practice. But, those three categories are part of the evaluation system. These standards are applied only for performance of students’ criteria, not the entire evaluation, beginning next year.

Here’s a visual example of what we were talking about earlier with the confidence interval.  The dot in the middle of the line would represent the actual Value-Added Model score that a teacher earned. The score itself, which would represent what actually happened that year with that teacher and those

students, using one standard error that’s provided with that score and constructing a confidence interval of one standard error on either side of the score just like you saw in the polling earlier, would mean that a district would be 68% confident that given a different group of students, that teacher’s score would lie somewhere within that interval.  Using two standard errors provides 95% confidence that given a different group of students that teacher’s score would have been the same or within that confidence interval.

Given what we’ve put in the rule, how would that work?  For example, the dark black line in the middle of the page would be zero.  That would be typical performance for students in that grade and subject for that year.  Using the picture of the visual example we used just a minute ago, if you look at the standard for highly effective, what you see is a score. You see with even using the 95% confidence interval, given a different group of students, you can be very sure that that teacher’s performance would have been above typical.

Likewise, using a 95% confidence interval around the score you can be sure that the standard of unsatisfactory performance would have been achieved, the way these standards are written for your consideration, because the entire confidence interval level is below zero.  The two in the middle actually represent two definitions we’ve included in the rule for effective performance.

The first one you see, or the higher level one you see, is where the value-added score is actually above zero. It’s above typical performance. Even though applying a confidence interval, it could be possible, if given a different group of students, that that teacher may have performed slightly differently or less than state average.  Because that score itself was above zero, then in that performance of student section of the evaluation, should be considered effective performance.

The one lower than that you see where the value-added score is actually below zero, the score...below zero, on average the students in that class when you put the numbers together perform below typical for the rest of the state. When I apply the

confidence interval, I am 95% confident that it could have been better with a different group of students. The performance could have actually been above typical.  Since we apply those confidence intervals, because that’s really good, solid statistical practice and we’re trying to use this information not just to see what happened that year but to actually classify performance as a portion of an evaluation, then applying the confidence interval means that we could also classify that performance as effective performance for that part of the evaluation.

That’s how we’ve defined the performance standards themselves using two standard errors of 95% confidence interval.  Those were highly effective and unsatisfactory and then having two options listed two definitions for effective performance, we consider the score itself and consider a confidence interval when the score is just below zero.  You notice that we have not defined needs improvement or developing. The law requires that we do performance levels for unsatisfactory, effective, and highly effective. So, the remainder would be needs improvement or developing if you’re in your first three years of employment.

Again, our proposal in the rule is that these performance standards would go into effect next school year, but only for how you classify performance on the performance of students’ parts of the evaluation. So, a district would use this classification for teachers in courses associated with FCAT. Every district would use it.

Then, it would be combined with the local district’s evaluation system for instructional practice, professional and job responsibilities, whatever else is included in that evaluation system, and the district determines the summative rating.  The only thing that remains consistent among districts is that use of assessment data and the growth model would have the same performance classification in all districts. The rest of it remains a local decision.

That takes care of setting performance standards in the rule and it takes care of setting them to provide greater consistency among evaluations throughout districts. The next part, though,

we have to take care of is applying those performance standards, and when we do apply those performance standards to actually impact the summative ratings. That goes back to the part of the law where it says you have to establish a student growth standard that if not met would result in an unsatisfactory overall evaluation regardless of whatever else may have been determined in the other parts of the evaluation, and then set a performance standard that has to be met in order for a district to, once everything else is combined, consider the individual to have effective performance and the same thing for highly effective.

So, you can think of the unsatisfactory part as an automatic unsatisfactory if the student growth is below a certain level. You can think of the effective and the highly effective application of the standard as a floor.  You have to reach at least this threshold to be considered effective or at least this threshold to be considered highly effective.  In doing so, what we considered were conditions around which those would actually go into effect and be used for that purpose.

We gathered those conditions from the law itself.  One of the things you may know about the law is, as I mentioned earlier, whereas it sets the standard for the percentage of evaluation based on student performance is at least 50%, the law does say that if you have less than three years of student performance data for a teacher, you can reduce that to 40%. So, in our mind, that was a condition of diminishing the emphasis on the student performance data in the evaluation because you only had one year of data or maybe two.

We used that same condition on when these standards would actually apply to impact the final summative rating. These standards would not impact a teacher’s summative rating or become the unsatisfactory rating or set the threshold for effective and highly effective unless that teacher had the three years’ worth of data. Second, those three years may be based on what the teacher is actually teaching.

We know that in the first years, people are using the option for student performance data in those grades and subjects where the district isn’t comfortable using their local assessments yet,

such as team results, school results, statewide assessment data for the assigned students rather than the data based on the content of the courses being taught. Since that’s an exception, since that’s a temporary measure that expires July 1, 2015, we used that as a condition as well.

So, if a teacher is being evaluated on those temporary measures, then these performance level standards would not impact the summative rating.  They would not result in an automatic unsatisfactory.  They would not be required to use as the threshold for effective and highly effective. Again, we would still expect districts to use the performance standards just for the performance of students criteria for those teachers that were teaching courses associated with FCAT data because that’s what this model is designed for; but, not for other teachers under these conditions or not even for those teachers if they didn’t have three years’ worth of data.

Most of the rest of the slides are just there for information purposes. I’m not going to go through each one, but for individuals who may be attending workshops or trying to

provide input on the rules, who weren’t part of district evaluation team design or design teams and all those kinds of things and don’t know what’s going on the last two years to develop evaluation systems, these slides are here for that purpose.

One other thing I would mention before we go to Part Two of this section is that you may remember at the beginning of the presentation the law said that districts had to adopt an equally appropriate growth formula and equally appropriate standards. What we have tried to do in the rule is provide options for districts they could use to set equally appropriate growth measures and standards for their local assessments, and even an option for a district who didn’t want to use one of those, a way to use some other measure or some other set of standards for determining student learning growth for a set of teachers and just by providing an instructional rationale for that method they would choose to use.

Again, trying to get at what the law is intending with these performance standards to provide consistency across the state, districts choose their local growth models for their grades and

subjects that are not assessed by the state. They would choose the performance level standards. But, the rule tries to get at methods by which they could so to improve consistency and a mechanism by which districts could suggest another method to be used.

So, reminding where we are at this point. We’ve finished Part One with a lengthy explanation of everything in both rules and what’s in the law.  Now, we’re going to go to Part Two, which is Clarifying Questions. Again, in this section in this part of the rule workshop, this is your opportunity just to ask questions about anything that doesn’t make sense to you, you didn’t understand, or you’d like to hear repeated for clarifying purposes. These won’t necessarily be your final comments. This is just an opportunity to ask questions. You’ll still have an opportunity to provide comments.

Anne, I’m going to ask you if you’re there to go ahead and compile the queue.

Coordinator	Our first question is from the line of...Connelly.

P. Connelly	Hi.  My name is actually Patrick Connelly.  I’m from Land O’ Lakes High School, United School Employees of Pasco. My question about Slide 32, the...referring to the non-classroom instructional personnel not being subject to the rules you established in terms of unsatisfactory and highly effective/effective.  Does that mean that a non-classroom instructional personnel will never be in danger of getting an unsatisfactory student growth score to be added to their summative evaluation or that they could never get a highly effective or effective student growth score to be added to their summative evaluation? That’s my question.

K. Hebda	Thank you very much for your question. The answer to that is no. That’s not what that means. What it’s intended to mean in the language of the rule is remember we’ve done these performance standards now for two reasons. Number one just to apply to performance of student section for courses related to/associated with FCAT. Then, we also provided districts methods by which they could choose to, or suggest another,

that they would set comparable growth standards for other grades and subjects. That would be the same thing for non- classroom instructional personnel. Using standards or comparable standards for just performance of students would apply to everyone.

What this language is saying that the way we’re putting this out here for your consideration is that the use of the standards if the student growth standard is so low that it would automatically be an unsatisfactory evaluation that that would not apply to non- classroom instructional personnel. Right now under the law they are required to use in evaluations of non-classroom instructional personnel some state assessment data for those individuals.  But, they don’t have direct classroom responsibilities, so considering that part of their evaluation is based on state assessment data but the other part is not, then we took that as a criterion that in this instance that would not override their entire evaluation.  For the performance of students, we would expect the district to set comparable standards for what those student growth measures are for non- classroom personnel. But, those non-classroom personnel

would not be subject to using performance standards to make an automatic unsatisfactory evaluation.

P. Connelly	Thank you.

Coordinator	Our next question comes from the line of Sam Whitten. Please proceed.

D. Steele	Yes, my question is could you give us the reference in the law, of the section where it talks about the need to require the performance standards particularly for highly effective and effective?

K. Hebda	Hi, I’m guessing that’s David Steele, not Sam Whitten.

D. Steele	Right.

K. Hebda	It’s actually 1012.34 (8) under rule making.  Get back to that slide, too. Way up here somewhere.  Right, it’s (8).

D. Steele	Thank you.

K. Hebda	You’re welcome.

Coordinator	Our next question comes from the line of Vickie Cartwright.

Please Proceed.

V. Cartwright	Yes, ma’am. We have a couple of questions when we’re looking at this. When we are looking at the teachers who are involved with this, a couple of things.  I’m on Slide 32.  On Slide 32 on your first and second bullet we have questions on.

The first bullet where you have three years of students’ performance data, if there is only two years’ worth of student performance data, will they get a VAM score that is used in their evaluation or not.

K. Hebda	Yes, they will.  ...provide the VAM score, the Value-Added Model scores for each individual year and we’ll still provide you the aggregate score that combines across years and across grades and subjects. The only difference is here; the only thing we’re talking about on Slide 32 is when those performance standards would impact a summative rating directly.

V. Cartwright	Okay.  The second bullet on teachers must be evaluated based on content areas they are teaching, not evaluated based on the measures as set forth in statute.  I’m looping back to the language in the other rule making.  It sounded as though there was language in there referring to our traditional professional development plans and setting those goals and teachers meeting those goals if there is not already a test created for that subject. Am I understanding that correctly?

K. Hebda	Right.  I’ve gone back to Slide 14 to put up the language directly from the law that is in that paragraph (7) (e) that talks about for classroom teachers of courses for which the district has not implemented appropriate assessments under the student assessment statute, or for which they don’t have an equally

appropriate growth formula. Then, student learning must be measured by growth and learning of the classroom teacher’s students on statewide assessments or learning targets or there’s a sentence next to the end where it says the superintendent can assign instructional team data to an individual for purposes of evaluation.

So, if that’s the case for the teacher, or like you’ve done in some districts. They’ve used what the learning targets are also used in the individual professional development plan that’s set out at the beginning of the year, which is a great way to do that. It just means that that’s a temporary measure in this case.  It’s only available for those teachers until July 1, 2015 because the assessments that are used now in those courses are not the ones the district wants to use for evaluation purposes. Until the district makes that decision for those courses, if the teacher’s evaluated on one of the options under this paragraph, then we don’t think it’s a good idea to apply those performance standards where you could be automatically unsatisfactory.

V. Cartwright	I have one last question for you talking about the confidence intervals.  I’m going to state and make sure I’m understanding this correctly.  In order for a teacher to be in the category of highly effective or unsatisfactory, their confidence level has to be completely in the region of highly effective or unsatisfactory.

K. Hebda	That’s right. The way we’ve constructed the standards right now, the first thing to remember, though, is these performance standards would not be make a teacher automatically highly effective or unsatisfactory beginning next year.  They would only be used to classify performance in that portion of the evaluation based on student performance.  You would still do everything you normally with instructional practice and you would determine what the summative rating is of highly effective.

But, at the same time when these do take effect for when all the conditions are met, and they would be applied to be the threshold for highly effective or the automatic unsatisfactory, that’s what these would be.  For your consideration, the 95% confidence interval means the entire range of scores would

have to exceed typical growth for highly effective, or the entire range of teacher’s scores based on student performance would have to be below typical performance for unsatisfactory.

V. Cartwright	So, hypothetically, a score of 0.5, and let’s say hypothetically the critical range there was 0.4, with a standard error of 0.2 – so their range is 0.3 to 0.7, they would be effective because their range was not completely in the highly effective even though  the raw score was in the highly effective.

K. Hebda	Actually, in that case, if I understand what you’re saying, applying the 95% confidence interval based on two standard errors for that particular number, if that confidence interval, the entire range of scores, the confidence interval was above zero, which it sounds like in your example it was, that would be highly effective performance.

V. Cartwright	Thank you.

Coordinator	Our next question comes from the line of Patrick Connelly.

Please proceed.

P. Connelly	Hi, this is Pat Connelly again.  It’s the same slide.  If I’m understanding what you said correctly, the auto trigger only applies to unsatisfactory in terms of if your student growth 95% confidence was fully below zero, you automatically get unsatisfactory summative regardless of what your classroom practices....  Is that a correct?  Once the year group meets and all these other criteria are in effect, is that a correct statement?

K. Hebda	Yes, that’s correct. The law requires that we set those standards to have that function and that’s what we’ve done.

P. Connelly	Understood. Does that apply equally to highly effective?  In other words, in the scenario that was given where my student growth scores are within the two standard errors well above zero, does that automatically supersede any ratings given by classroom observation?

K. Hebda	It does not. The way the law provides that is a threshold for effective and a threshold for high effectively.  So, you would have to have that performance to be considered by the district as highly effective.

P. Connelly	So, overall?

K. Hebda	Overall, correct.

P. Connelly	For example, if my classroom observation score was highly effective, and that’s supposed to be 50% of my evaluation, but my student growth score was such that the bottom of the bar touched zero or potentially gets below zero, I could not be considered highly effective?

K. Hebda	That’s right.  Once all the conditions kicked in, that would be correct.

P. Connelly	Okay.  Then, I guess my next question is this: We’re looking at all kinds of different subjects and eventually the situation where every subject has its own measurement of student growth.  How has or how will your role take into account the fact that some subjects such as, say, FCAT for the lower grades has a long standing history of data with a huge amount of student
data to make valid comparisons with say a new subject such as perhaps maybe art where there’s at the high school level or maybe even AP physics or Physics I Honors where there is not nearly the number of students, not nearly the data, not nearly the same amount of historical data that one could call upon when you only had a couple of years of end of course exams for these people.  How are you going to determine exactly what effect attendance has, because I would posit that the attendance and all these other contributing covariant factors the number in the formula should be different for various subjects. Is my question clear?

K. Hebda	Yes, I think your question is clear.  I have a partial answer.

Well, I have the answer based on the text we have in the rule, which is our intent – and I went back to Slide 32 – is that, for

example, let’s take the art teacher. Our teachers give their students assessments now of all kinds, including projects and everything else.  But, if the district’s not comfortable using those for evaluation purposes, then that art teacher could be on a temporary measure right now.

So, let’s say that the art teacher selects assessments they want to use for art and it’s the ’14-’15 school year. So, ’14-’15 would be the first year that that art teacher, actually the clock would start ticking for that art teacher, to then accumulate three years of performance data in the subject area for which he or she is teaching.  So, it would be ’14-’15, ’15-’16, and ’16-’17. The ’16- ’17 school year would be the first year that that art teacher that these conditions could apply to the art teacher.

P. Connelly	I understand. But, in terms of the comparative reliability of the VAM formula for the different subjects, you understand my concern that a subject with a great deal of historical data from which you can mine all kinds of effects. The confidence one would have in the value of the covariant factor would be substantially higher than for one where you have a much

smaller pool from which to draw.  You still might have three years, but you might have one percent or 2% of the students and therefore you’re not going to have, for example, enough students with three movements to determine what the effect of that should be on a Physics I student.

K. Hebda	That’s a point well taken.  I think that a couple of things I would say that in the FCAT model if you go to what we’ve  incorporated into the rule and you can find on the web now, the coefficient at the back of the technical report that lists all the different disabilities and the mobility and everything else and how much they actually affect or don’t affect a student’s predicted scores, sometimes it’s one developmental scale score point, sometimes it’s two.  The biggest impact is always that prior score.

At the same time, I think one of the things that’s going to be very important, and we’ve got this built into the work we’re doing under Race To The Top with the districts that are working on those assessments for areas like art and music and physical education. But, it’s going to take some time to make sure that

those are used properly in evaluation systems.  Every growth model is going to have a limited set of capabilities.

We are very proud of the model that the Student Growth Implementation Committee selected. It works beautifully for a vast majority of teachers who teach reading, language arts, mathematics courses that are associated with FCAT. But, it has its limits, too. We need to determine what those are for these other grades and subjects that maybe you can’t do it in the growth model itself, but if we can’t we need to make sure that the evaluation system process deals with those issues somehow.

P. Connelly	I understand the mathematics and the statistical analysis used to develop these things.  Actually, I do believe that with proper research it could maybe be effective.  But, my concern is there are many, many, many, many, many subjects for which the best predictor of performance is not available because there is no direct precursor.  For example, what would be the precursor for World History? What would be the precursor for Art? What would be the precursor for Physics?

Until such time as you can identify a valid precursor score that everyone has, not everyone who takes Physics I has had Chemistry.  So, what do you do about the kid who didn’t have Chemistry as a precursor score? There are issues of, I believe, equity in how some people will be held to a standard that is extremely solid, whereas some people are being held to a standard that’s extremely nebulous.

K. Hebda	What I would appreciate is when we get to comments or as you go back and think about that more, are there things that need to be in the rule that would address your question and concerns that we should add.

P. Connelly	Thank you.

Coordinator	Our next question comes from the line of Vickie Cartwright.

Please proceed.

V. Cartwright	One additional question. When we’re looking at this if we’re doing the definition of effective being within the 95% confidence interval, being within plus or minus two standard deviations, not including two.  How would you come up with a needs improvement or developing?

K. Hebda	Well, in this case since we’ve defined effective in two ways, the score is above zero, but when you apply the confidence interval, part of the confidence interval could be below zero, that’s effective.  The score itself is below zero, and the confidence interval, though, stretches above zero; either of those in the rule could be considered effective.

It seems to us that the remainder would be left in needs improvement and developing.  I think we only use the 68% confidence interval for effective, rather than the 95% confidence interval.  That leaves the room between effective and unsatisfactory. We think.  Now, if we haven’t, make that comment and give us a suggestion

V. Cartwright	Thank you.

Coordinator	We have no further questions from the phones.

K. Hebda	Great. Let’s start with questions from the room, then.

M	Yes, I have some questions...

K. Hebda	Hold on one second. We have to give you the microphone so we get a good recording.  Thanks for your patience.

M	Paul...from Levy County Schools.  I have a couple of questions.

I may have missed this, but I had heard that there was a statement in here that district developed tests must be equally appropriate to the FCAT test. Now...my question.  My question is equally appropriate to what?

K. Hebda	Thank you for your question. The law requires that you adopt an equally appropriate growth formula.

M	Not an equally appropriate assessment?

K. Hebda	Right.  You determine what the assessment is under the assessment statute and that tells you it’s got to be based on mastering content of the course and the state course description and all that stuff in the assessment.

M	How will it be judged whether it’s equally appropriate? The growth standard?

K. Hebda	What we tried to put into rule was two suggestions how you could do that, and then also give a third option for where if you don’t like those two suggestions, you’ve got a better instructional rationale for what you want to do, you present that to us.

M	Okay.  Also, I’m looking at the rule here. Number seven, it talks about three years of data for evaluations...’11-’12.  Is the three years of data; is that going to be ‘12’, ’13, ’14 or ’14, ’15, ’16?

K. Hebda	Which rule are you in, Paul? I’m sorry?  Oh-three-oh or 0411?

M	Oh-three-oh.

K. Hebda	Oh-three-oh. Thanks.

M	Number seven. So, if you say you need three years of data, are you going to go retroactively from ’14 to ’13 to ’12, or start with ’13-’14 data? Because you state the implementation of the rules is ’13-’14.

K. Hebda	If I can make sure I know what you’re talking about. I’m sorry.

I’m in Section (7). Seven is monitoring, implementation monitoring.

M	No...says Confirmation...including...

K. Hebda	Oh, I see number seven.  I’m sorry. Where are we?

M	The question is when we’re talking about three years of data and we’re talking about implementation of the rules start next year. Does the three years start next year? Or are you going retroactively three years backwards?

K. Hebda	Thank you. I see where you are now.  For folks that are not in the room, we’re in 6A-5.030, and we are in section (3) that has the content of the evaluation system. It’s in lower paragraph (a.), which is the evidence that we would need to approve the evaluation based on performance of students’ requirements. He’s looking at number 7 under (a.)

That says:  One of the pieces of evidence we would be required to have in the district so we could approve the evaluation system would be “Confirmation of including current year student performance data in each individual’s summative rating for that school year. For evaluation systems implemented in the 2011- 12 or 2012-13 school years, the years of student growth data that will be included in the evaluation must be specified in

addition to the current year data. Beginning in the 2013-14 school year, for all personnel, at least three years of student performance data must be used in the evaluation when available”, which means that ’11-’12 was considered year one. so, we gave you the option to use the old data, the historical data if you wanted to, but almost nobody did.

M	Thanks. So you’re using ’11-’12 data, ’12-’13, and ’13-’14.

K. Hebda	’12-’13, and ’13-’14. That’s correct.

M	When you go back to ’11-’12 data, are you using the districts cut score or zero?

K. Hebda	That’s a good question. We had this on this morning as well.

When you evaluate a person every single year, you get a brand new evaluation every year. The evaluation law says, when you can, you should use at least three years of student data, that teacher’s student data in that one year annual evaluation. So, you have one year of instructional practice data that year for

’13-’14, for example. I’ve got one year of instructional practice data I’m looking at.

But, if that teacher, let’s say, has been teaching 6th grade reading for three years, and that would have the student growth rate model in place for FCAT courses since then, the she may have three years of student performance data.  But, that’s all rolled up into one aggregate Value-Added Model score for that ’13-’14 year.  You don’t worry about whether she was highly effective or effective overall those different years.  So, you’re not summing up and averaging overall summative ratings, you’re just collecting all that student data from three years and making a new value-added score.

M	So, therefore, those three years that...comes out as unsatisfactory.

K. Hebda	I guess that’s a possibility.

M	...possibility.  So, therefore...

K. Hebda	...cut scores were.

M	Yes, but you’re saying that the new rule is that if they’re unsatisfactory for the growth data, then an unsatisfactory for evaluation no matter what.

K. Hebda	If they meet those criteria.  That’s what’s called for.

M	That’s if they meet the criteria.  So, therefore, theoretically, you’ve got a teacher who’s been rated effective for two straight years, and now you’re going to say no, that interpretation of the district is not correct, even though we were told you’ve got to do it for two straight years.  Now the teacher, who’s been told they’re effective or even highly effective, is now unsatisfactory.

K. Hebda	Right.  That’s exactly the way this text is presented to you for consideration. That is how it would work, because if that teacher had all the conditions met, three years’ worth of data in the subject she was teaching for all three years, and you had

determined the summative rating each year in those prior years and those were her evaluations...year –

M	...because of her FCAT scores, her value-added data was satisfactory.

K. Hebda	Either way, you determine the summative rating overall.  Now that year, if these go into effect next year, that following school year it could happen.  If, in fact, I mean if you rated her effective, I don’t know how likely this is, but if it’s all three years of data resulted in all of her students on average with a 95% confidence interval were all below zero, then it could happen.

M	...the teacher’s...because we didn’t use zero as a test score.

So, therefore the teachers are going to feel that they’re evaluations have been retroactively changed as well then.

K. Hebda	The past years’ evaluations stand. That’s important to note, because it matters what you get every year for contract purposes. So, it does not change their previous years’

evaluations. If you gave them effective, then they were effective that year, period. That’s done; in the books.

M	It’s the third year’s data is based on three years...

K. Hebda	Every year you get evaluated again.

M	So, it does not affect the first two years’ evaluations?

K. Hebda	That’s right.  It does not.  You are correct. It does not affect the first two years of evaluation.

M	...

K. Hebda	You’re right.

(58 seconds of silence on audio)

K. Hebda            We are getting lots and lots more data points than you have with one year data, and that’s converted to a proportion of a year’s growth.

F	How will we take into consideration the confidence intervals with that measure so that we know how to appropriately classify them?

K. Hebda	The aggregate VAM score has its own standard error.  So, you’ll be provided, just like you were this year.  You were provided with a standard error for each score. That aggregate VAM score has its own standard error with all the data points put together.

F	Okay.  Then, another just a technical question question.  As I’m reading through this with effective, I know that we’re looking at going two times the standard error for above zero, but when we go below zero, we’re only doing one time the standard error.

K. Hebda	That’s right. That was part (b) of our definition of effective.  Our second option for being effective was your score is actually below zero that year, but if I apply one standard error confidence interval, 68% confidence interval and it stretches above zero, then you can consider. You’re pretty close to zero that means. So, you could consider that performance effective.

F	Okay.  So, for application purposes, when we get down to the end product here, if we are using the aggregate scores, you can technically use the aggregate scores even if they are only teaching one grade level, one subject because they have multiple years of data –

K. Hebda	Correct.

F	– there’s really only one scale that we are going to be working with.  Is that an accurate understanding?

K. Hebda	Right.  These would be the performance level standards for the use of the aggregate score.  Is that what you’re asking?

F	Right.  So, there would only be one scale.

K. Hebda	That’s correct.

F	Thank you.

Coordinator	We have no further questions from the phones.

K.	Hebda	Thanks. I would like to add just to make sure: Remember these are the performance level standards for use with FCAT data.  So, if you have teachers that teach some FCAT associated courses and another part of the day they’re teaching something else, then you’re still going to combine all of that to determine with your local performance measures and local performance standards what that threshold would be.

I’m going to check on the web and see if we have any questions from the web. We didn’t earlier. There are still no questions

from anybody else in the room?  One more question in the room.

L.	Hearn	Hi, I’m Lynn Hearn with the law firm of Meyer, Brooks, Demma

& Blohm. My question really relates to the Instructional...Rule. In sub-section (b.), it lays out what would be a completed exceptional practice criterion. In paragraph...second sentence, “a district must provide documentation of how indicators and/or domains are prioritized to present evidence that the evaluation system is designed to support effective instruction and student learning growth.”

Am I correct that the Department will be reviewing the evaluation system to determine whether it’s believed the district system meets the...?

K.	Hebda	That’s correct.

L.	Hearn	How will the Department determine that?

K.	Hebda	The requirement here is that they prioritize.  It doesn’t say how they have to prioritize or what method they use to determine that.  We had this question this morning, too. There are a couple of things that talk of being organized, all the indicators organized into domains and prioritizing indicators.  Remember, this system has to provide effective feedback. If you look at evaluation systems the district provides a laundry list of things to do; very difficult to know how well you’re doing on those things or which ones are more important than others.

So, the recommendation – because this is based on contemporary research/instructional practice – is that the district organize the evaluation system, organize its indicators in the instructional practice section into domains, which you can kind of think of as guiding principles or sections, and then prioritize those things that the district believes or based on the district’s research base will most directly improve student learning. They can do that a number of ways. We don’t say how they do it. But, again, what we’re trying to do is use that contemporary research base for the meaning of the purpose of evaluation

systems to make sure the district constructs the system that’s beneficial to the teacher.

L.	Hearn	Thank you. May I ask a second question...rule?

K.	Hebda	Of course.

L.	Hearn	What affects or determines these standard errors?

K.	Hebda	The standard error of measurement is primarily related in this case by the time you collect all this data to the number of students that are included in the teacher’s score.  A smaller number of students is going to result in a larger standard error. There is in every assessment that anybody ever gives from FCAT to the test that the teacher provides on her own for a mid- term or a final exam, there is measurement error.  A test can only assess so much for each student:  so much content, so much time.

So, there is a little bit of measurement error that this actually can statistically calculate based on what happens in the test. But, primarily in this case it’s based on the number of students. For example, if I’m a teacher, and we’ll calculate a value-added score for as many teachers as we can based on the data the district gives us of the assignment of teachers and students. But, if we return a value-added score to a teacher and the standard error, for example, if the score is 0.1, but the standard error is 0.5, then that standard error is enormous. That’s a very, very large error, almost to the point where you can’t tell, given a different group of students what the teacher’s performance might have been.

So, a district doesn’t actually have to use that score if the standard error is so large that they can’t find any confidence in that score. They would have to have in their evaluation systems some other mechanism to determine student performance or some way to deal with the fact that they can’t use that score, just like you would do in observations.  If for some reason someone collected observation data they were collecting on a piece of technology and for some reason the

technology broke down and you lost all the observation data, what provision in the evaluation would you have for dealing with that situation where you had no data for part of the evaluation.

Those kinds of things are going to happen. So, a district needs to be able to have provisions to deal with those things.

L.	Hearn     I didn’t actually see that in the rule. Is that set out in the rules somewhere that a certain standard error level that the district doesn’t have to use –

K. Hebda	No. We’ve not prescribed a standard error that requires a district to use or not use a score. We’ve left that up to the district. There are other considerations, too. A district may have...just with standard error whether the score was appropriate. There could be lots of things that could factor in that, just like there are lots of things that factor in to the usefulness of observation data that could be calculated. So, we’ve left that to the school district in this text.  It may be in the comments portion of the evaluation of this workshop or any

other time that you may have suggestions of what we should add if you think something should be added about that.

Seeing no further questions, let’s go to comments. Eileen, I’ll turn it over to you for comments.

E. McDaniel	Operator Anne, would you take a queue of who’s on the line now and we’ll open up for comments at this point?

Coordinator	Our first comment comes from the line of Patrick Connelly.

P. Connelly	This is actually more in the line of a question, I think. I just want to clarify the...of standard errors based on the number of people.  How is that going to impact, say for example, the following scenario?

A 3rd grade math teacher who has, say 20 students because they’re violating class size amendment, and a 7th grade math teacher who had 100 students. I believe that the standard error

for the 7th grade teacher would be significantly smaller.  So, if they both actually had the same score as a VAM score, maybe a 0.3 because they’re both pretty good, the standard error for the person with the fewer students might be so large as to automatically bar them from getting highly effective, whereas it might make it better with having lots of kids. That’s a question I’m not quite sure there’s an answer to it. Thank you.

E. McDaniel	Thank you.

Coordinator	We have no further comments from the phones.

E. McDaniel	Anyone in the room have a comment to make, please?

K. Hebda	I don’t see any comments from the web.

Coordinator	Pardon the interruption. We do have a comment from the phones.

E. McDaniel	Thank you.

Coordinator	It’s from the line of Vickie Cartwright.  Please proceed.

V. Cartwright	I have a technical question. If we provide comments today, does that prevent us from providing supplemental comments in the future?

E. McDaniel     Absolutely not. The web will be open from now until time for the State Board meeting, so we certainly encourage any comments that you might have.

K. Hebda	I would also suggest that you could come to additional workshops, too, if you wanted to; either way.

V. Cartwright	Yes, ma’am, thank you.  Just a couple of comments that I would recommend. In the rules, you’re putting the language of FCAT, when we had transitioned into FCAT 2.0. I know it’s very technical; also, in the rule where you’re defining about the

automatic classification for unsatisfactory. It appears as though there were two things that were written in there, but you have a copied one sentence identical to the one following that.

E. McDaniel	Thank you, Vickie.

Coordinator	We have one more from the line of Vickie Cartwright, again.

Please go ahead.

V. Cartwright	If we could add in language that is the specific to needs improvement/developing rather than leaving it as an assumption.

E. McDaniel	Thank you, Vickie.

Coordinator	Now, there are no further comments from the phones.

K. Hebda	Are you sure about that, Anne?

Coordinator	I think so, yes.

K. Hebda	Okay.  Go ahead and close up, Eileen.

E. McDaniel	Any further questions from here? All right; well, thank you everyone for joining us today in today’s webinar and for those who were able to join us in person today at the Department. We welcome comments through the web and we will have five additional face-to-face workshops the week after next as noted in the advertisements. Thank you for joining us and this concludes today’s workshop. Stop recording please.

K. Hebda	Thanks, Anne.

Coordinator	You’re welcome. Ladies and gentlemen, we thank you for your participation in today’s conference. This concludes the presentation and you may now disconnect. Have a good day.