Final Transcript

STATE OF FLORIDA: Race to the Top: Rule Development Workshop
February 14, 2013/9:00 a.m. EST

PRESENT IN THE MEETING ROOM

Kathy Hebda, Deputy Chancellor for Education Quality for the Department of Education
Eileen McDaniel, Chief for Educator Recruitment, Development, and Retention, Department of Education.

PRESENTATION

Coordinator	Good day, ladies and gentlemen. Welcome to the Rule Development Workshop conference call. My name is Anne and I will

be your coordinator for today’s call.  At this time, all participants are in listen only mode. We will be facilitating a question and answer or comment sessions throughout the presentation.  I would now like to turn the presentation over to your host for today’s call, Kathy Hebda, Deputy Chancellor for Education Quality for the Department of Education. Please proceed.

K. Hebda	Thanks, Anne, very much. Good morning everyone and welcome to the Rule Development Workshop, our public input on draft rule text for two of our rules that are up for development. The first one is Rule 6A-5.030 – District Instruction Personnel and School Administrator Evaluation System; and, Rule 6A-5.0411 – Calculations of Student Learning Growth for Use in School Personnel Evaluation. With me today is Eileen McDaniel.

Eileen McDaniel	Good morning, everyone. This is Eileen McDaniel. I’ll be your Chief for Educator Recruitment Development Retention.

K. Hebda	Eileen and I will be hosting you this morning for our workshop.

So that everyone is aware, we do have folks joining us via webinar and we have folks that are joining us here in person, and via conference call.  If you would, this morning I think everybody who is here present in the room; I’m going to go through a few instructions for all of our different groups of people. So, if you will bear with me to make sure that we get everybody’s comments and questions on the record, that’s very, very important to us.

We’re going to conduct our workshop in three sections. The first part today will be an overview presentation of the authorizing statutes that underline these rules and the content of these draft rules. At this point in time, in the first part what we’ll do is try to provide you the basic underlying of short...what does the law say we have to make rules on. Then, also, given that information what have we tried to present in the rule text itself; what’s our intent.

After we’ve done that, we’re going to have two parts that will be interactive. The first one is Part Two where everybody who’s

participating, whether on the web or in person or via the conference call, can ask clarifying questions and answers. Sometime prior to making a comment about a rule, someone has a question. They want to make sure they understand properly whether it’s in the law or in the rule text itself or something said during the presentation.

So, Part Two is an opportunity for people just to ask questions. Whether you’re asking questions or you’re making public comments, it’ll be important for us to get your comments and your questions on the record. All of that will help us improve this draft rule text so that we can make it better and publish it again for additional comment and review by the public.

After the question and answer session is ended, Part Two, we’ll go to Part Three, which are the actual public comments on the rules. During this time, audience participants who wish to make a comment, you aren’t required to, but if you wish to do so, you can make comments about the rules. At that time, there will not be responses or questions or debate. We want everybody’s

comments to be able to stand on the record. We want everyone to feel comfortable to make their comments.

At that time, we will ask people to make those comments.  If you’re on the web, you can type those comments in because the webinar is being recorded. Those will be recorded in the record as soon as you type them in. Or, if you’re here in person or you’re just listening on the phone, then we’ll ask you to make your comments orally.  If you’re here in person, we have an index card where you can write your name and your affiliation down so that we have the spelling correct for the record.  If you’re calling in on the conference line, the operator will facilitate your comments.  She’ll do the same thing for Part Two when you’re asking a question and then she’ll open the line again for Part Three for comments.

A little bit more about input and participation: As I mentioned, when you’re asking a question or making a comment, please state your name and your affiliation, which can be just your school district, your association, or anything else you want us to know about where you’re from. If you’re making a comment,

please follow the instruction with the operator, as I mentioned, on the conference call. If it’s the webinar, you can type in your questions, and if you’re in person, please fill out a speaker’s card.

When we get to the comment section, depending on how the number of people that are in the queue to make comments and depending on how much meeting time we have left, we may limit those comments to a certain amount of time that allows everyone the opportunity to speak who wishes to do so.  But, we’ll judge that at the time we get to Part Three.

If you don’t want to make a comment out loud, if you prefer to submit it in writing, we’ll take it in writing at any time.  You can mail them to us; you can leave them here if you’re in the room. Again, we’ll record whatever you type in on the webinar; and there’s a website here at on this page on Slide Three at the bottom where you can make comments online at any time.

One last thing about input and participation: As I mentioned, we’ve put this...out for public comment. This is draft rule text. We’re not ready to send this to the State Board of Education yet for their consideration. We’re still in the development stage of these rules. So, it’s very, very important to us at this time that we have suggestions from you if you’re listening or if you’re going to participate on the web or if you’re going to type us comments online, suggestions for changes or places where you want us to make sure we keep something in the rule that you think is important.

This can be things like “Please keep this text.” “Please delete this section.” “Please add something about – or I like where you’re going except that I think that you need to change it to clarify it to say – such and such.” So, there are lots of things you can do to help us to improve this rule text.

We would ask, also for the improvement of the rule, that you be as specific as possible. If you have a reason why you think your comments or your changes are important, that gives us a lot of context as well so we can check for those things...parts of

the rule and we understand the situation that you’re expressing when you ask us to make a change.

Lastly, we are also very interested on whether the rule text that we have drafted actually accomplishes what the law says we should do, and/or we indicate in our presentation that we are intending to do. If our presentation says we’re intending to make sure that we collect a specific kind of evidence from a district and you look at the rule text and you think it doesn’t really say that, then we want to know that it’s not clear to you, or if you think it’s not worded properly to get that full effect. So, those kinds of things are very important to us.

And, after all of that, all the participation requests and places, things to do, if you’re going to participate let’s go ahead into Part One, which is just our presentation on what’s in the law and what’s in these two rules as an overview.

The first thing we should know is the rule adoption timeline. I mentioned we’re still in rule development. So, the month of

February we’re having workshops. We have two today that are via webinar and conference call. Then, we have not next week, but the following week, the week of the 25th of February, we have several that are around the state. That same website on the previous page shows you where all the dates and locations of those workshops are if you want to participate in one of those in person.

During the months of March and April, we’re going to review the public comments that we’ve collected in the February workshops. We’re going to monitor legislative session for bill signed into law that may affect these rules. Also, during the months of March and April, we’ll leave the online comment section open. So, you can always go back to the website during the months of March and April and make additional comments even if we aren’t holding workshops.

But, in May what we anticipate is that we’ll revise the rule text based on all of that information and then publish a final version for consideration by the State Board.  Our target month for State Board adoption would be June. That way, people would

be on notice that the rules go into effect and then they could be applied beginning the next school year, the ’13-’14 school year.

The first rule we’re going to discuss of the two is Rule 6A-5.030

– District Instructional Personnel and School Administrator Evaluations Systems. As I mentioned, we’re going to talk about the rule authority and what the law says is the content for this rule.  In Sub-section 8 of Section1012.34, Florida Statute, the law prescribes the types of things the State Board is required to adopt rules on. Because we have two rules today, what I’ve done is I’ve divided the portions of Sub-section 8 that pertain to this rule from the ones that pertain to the second rule we’ll discuss.

For this rule, we have to establish the uniform procedures for submission, review, and approval of district evaluation systems; reporting requirements for the annual evaluation of instructional personnel and school administrators, and the process for monitoring school district implementation of evaluation systems in accordance with this section, meaning this section of law. The way that 6A-5.030 is constructed at this time is Sections (1)

and (2) of the rule begin to explain what are the uniform procedures for submission of a district evaluation system to the Department.  How is that outlined?

Section (4) and (5) deal with how those are reviewed and approved. What are the different approval statuses? Fully approved, conditionally approved, denied? What...the broad definitions of those, the conditions in which they’re granted and then how a system could move from a position of conditionally approved or denied into full approval status? We also discuss in those sections how districts can make modifications to their systems, as districts have been doing since they established them under the law.

What are the procedures for those things? The law talks about districts submitting to the Department when they’ve made substantial changes in evaluation systems. We’ve tried to provide a definition of what substantial means, so districts can know why do I have to submit it to the Department again for review and approval or when can I just make these changes on my own and re-post it?

We also, in Section (7) of the rule have gone into the monitoring process. How will we monitor a districts’ implementation? We thought it was very important that we set forth a monitoring process in the rule so the district would understand what this process would entail and what kinds of things it would be doing and looking for in the monitoring process.

We monitored our monitoring process for evaluation systems on the same process that we use to review and evaluate districts’ professional development system. We’ve had that process in place since about 2001.  So, we provided for a cycle of on-site visits where we would review each district’s system through monitoring once every five years.

The reporting section in Section (8) provides the basic information about how annual reporting is done. The law requires that the Department report evaluation results from all districts at the school and district level. So, this reporting section outlines very briefly how districts submit that data to us

so that we can then fulfill our obligations with the Department to provide that data and that information statewide annually.

The last section of the rule, Section (9) provides guidelines for training evaluators. Those are there because in Section 1012.34, it mentions the districts need to include in their systems training for evaluators; anybody that’s conducting evaluation in their system. It indicates that those training processes the districts adopt and implement have to be based on guidelines provided by the Department. They don’t have to be exactly the same as the guidelines provided by the Department, but the district can consider those guidelines. So, we felt it would be very important that we put those guidelines in the rules so the districts can find them all in one spot.

All of those sections (1), (2), and (4) through (9) of the rule are what you see on this slide.  How do I submit; what are you going to do when you get my evaluation system from the school district; how will I be approved; what does that mean, how can I modify; how are you going to report; how will you monitor? Those sections answer those questions.

The next part of the presentation deals with the content of the evaluation system as prescribed by the law and how the Department will collect evidence from the school district in this evaluation system to actually make a decision on whether the system is approved and meets the requirements that are in the law.  So, reviewing the content that needs to be included in the district evaluation system, the first thing is the purpose.

In some of these slides, if you’ve been following all these processes, then 2010 will look familiar to you. The purpose of evaluation systems designed to afford effective instruction in student learning growth results are linked to school level improvement plans and linked to professional development, so, the three key important points that need to be reviewed in the evaluation system.

Another important section is that the law says that we need to have evaluations that are based on four levels of performance and it describes what those are. So, part of the district’s

evaluation system needs to include how they will determine that someone has achieved highly effective performance, effective performance, performance rating that needs improvement for developing, and one for unsatisfactory.  The law also says that we’re going to establish standards for performance levels and the Commissioner should consult with experts in instructional personnel, school administrators, and stakeholders in developing the criteria for the performance levels.

That’s, in fact, what we’re doing now. We’re holding workshops. We’re providing opportunities for public input. We’re putting out draft rule text and gaining comments back from everybody on what those standards would look like.

The law also divides evaluations into three sections, three criteria:  performance of students, instructional practice; or for school leaders, administrators and instructional leadership; and professional and job responsibility.  It also indicates that the evaluation needs to be based on sound educational principles and contemporary research in effective practices.  So, those things need to be included in the district’s evaluation systems.

This is just a reminder on this slide, which is Slide 11, of what’s included in instructional practice as defined by the law.  For classroom teachers, the Educator Accomplished Practices and for other instructional personnel who are not classroom teachers, the Educator Accomplished Practices and in individual job expectations that the district may develop related to supporting students.

Instructional leadership: The law provides that these debates on our standards for instructional leaders, which were adopted under Section 1012.96, Florida Statute, which are Florida Principal Leadership Standards; and then lays out additional criteria that are specific to the administrator’s role in faculty development at the schools.

The performance of students section in the Statute: This is a reminder of what that says.  So, we were looking for this kind of content in the district’s evaluation system.  At least 50% of performance evaluation is based on data indicators of student

learning growth assessed annually and measured by statewide assessments; or, for subjects in grade levels not measured by statewide assessments, by district assessments...under Section1008.22, which is the student assessment section of the law.

It also provides that the percentage of the evaluation based on the performance of students can be reduced to 40% in the absence of three years of data for a teacher or principal.  It provides for non-classroom instructional personnel. The performance in student’s criteria can be split between statewide assessments and other local measures of student outcome that are specific to their jobs, and provide the district some options to choose to combine state and local assessments when the teacher teaches a local assessment subject, and can choose performance measures over measures of learning growth when they feel that performance measures are more appropriate.

Finally, the law provides a timeline by which evaluations should be fully developed, and for classroom teachers, of course, if the district has not selected appropriate assessments under

1008.22, then it provides options for districts to use other measures of student growth that are based on that classroom teacher’s students, or as a last minute provides, or...provides based on team data as determined by the superintendent.

Then, the law goes on to provide additional components of district evaluation systems and here they are in a list on Slide
15. Observation instruments with indicators that be performance indicators or ways to collect evidence on performance. At least two evaluations and observations per year for newly hired personnel. Evaluator training as we mentioned earlier; methods to inform personnel about the evaluation system; use of multiple data sources; use of system data for professional development and school improvement as we saw in the beginning of the slide; parental input; teaching fields that may need special procedures; an annual review process by the district of its evaluation system; and, there are some optional procedures that districts may or may not include. That can be peer reviews, assistance processes, input from additional personnel into the evaluation system, evaluation amendments, and additional professional responsibilities

beyond those in the State Board rule.  For our purposes, the professional responsibilities in the State Board rules are those that pertain to ethics and professional conduct.

The second rule we’re examining today and asking for input on is Rule 6A-5.0411 – Calculations of Student Learning Growth for Use in School Personnel Evaluations. We’re workshopping these two rules in conjunction with each other because even though the first rule holds all the criteria we would use to approve a district evaluation system, we also need to place in rule what statewide calculations in student learning growth have been adopted. So, the rule authority in content for 6A-5.0411, again, comes from the same sections of law we’ve looked at before and 1234(7) and (8). These are the things that have to be included in this rule.

Each formula for measuring student learning growth that’s approved by the Commissioner and the formula and calculation itself, the measurement of student learning growth and associated learning implementation procedures.  So, how do  we actually implement that formula across the state? A process

by which teachers may would be either class rosters, specific discrete standards for each performance level to ensure clear and sufficient differentiation in the performance levels and to provide consistency in meaning across school districts.

Then, within those discrete standards, specifically establish a student learning growth standard that if not met will result in the employee receiving an unsatisfactory performance evaluation rating.  Then, also establishing student learning growth standards that must be met in order for an employee to receive a highly effective rating or effective rating as determined by the school district.

The department has some responsibilities for student learning growth and educator evaluation. As I mentioned, there are two types of learning growth formulas. So, formulas for statewide assessment, which is the one that has to be adopted into rule once they’re approved by the Commissioner. You can see by the bullets on Slide 19 that the formula that we have currently is the one that’s associated with FCAT. So, that’s the one that you’ll see in this rule at this time. As other learning growth

formulas are adopted for statewide assessments, we go back to the rules and add those formulas to the rules once they’re approved by the Commissioner.

Learning growth formulas for local assessments will be decided by the school district. The law says districts must adopt “equally appropriate” learning growth formulas for local student assessments. This rule takes, for example, after it sets forth the statewide assessment formula and as I walk you through the content of that rule, there is a section of the rule that briefly describes how districts could choose to set “equally appropriate” learning growth formulas for their assessments.

Because we have to put the formulas in the rule, I’ve added some slides here that remind us where the formula for FCAT came from, since that’s the one statewide assessment formula that has been approved by the Commissioner to date.  It was developed by the Student Growth Implementation Committee composed of 27 members from across the state.  You can still see at this website on Slide 21 who the members are, where they come from.

In fact, the Student Growth Implementation Committee will be meeting again later this month on the 27th to examine some new statewide formulas for use with our...assessment, such as Algebra I, Geometry, and Biology.  But, so far, the Student Growth Committee recommended the model for FCAT. It was ultimately approved as they recommended by the Commissioner in June of 2011.

The model they approved, which is the model in this draft text, is a covariant adjustment model, which means it begins with establishing expected growth for each individual student that’s assigned to a teacher based on that student’s historical data, typical growth by grade and subject in our state as assessed in this case by FCAT because this is the model that works with FCAT, and based on how students perform who share similar characteristics that are identified by the committee and are included in the model. Those characteristics fall in three categories: student characteristics, classroom characteristics, and school characteristics.

Slide 23 is an example of looking at how an individual student might perform and then taking the difference between how they actually and what they were predicted to perform on the assessment based on their prior score and all of those characteristics that we know about them that are included in the model. All of the differences between actual and predicted performance are what are put together and combined for all students assigned to a teacher to produce the value-added score.  All of the other characteristics that go into the predicted score are the other influences the model accounts for on student learning. The residual, or the difference, is what’s attributed to the teacher; that which the teachers to the student learning.

So, here are those characteristics that are in the model and these are the ones that are enumerated in the rule and defined in the rule.  Up to two prior years of achievement scores, the strongest predictor, and all research is the previous score on the same assessment. The committee decided if we had two prior achievement scores that that would help as well.  So, if there are two available, we use two prior scores.

The number of subject relevant courses in which a student is enrolled; so, the student could be multiple courses that relate to reading or mathematics; the student’s disability status as an indicator of the specific disability, not just a yes or no disabled or not; English Language Learner status; student attendance – annual attendance, yearly attendance; gifted status; mobility, which is the number of transitions in and out of school whether it’s the same school or multiple schools.  Districts report those to us across the state and those are considered. And, the difference from modal age and grade, which could be an indicator of retention, promotion, or anything else that causes the student in the class to be significantly different in age from the rest of his peers.

Classroom characteristics include class size, and hhomogeneity of students’ entering test scores in the class, which means the Committee discussed when they were developing this model would it matter when our contractors run data on whether it would matter if classes are made up of students who have similar entering test scores; for example, most students in that

class were a level 4 or 5 on the FCAT, or most students in that class scored level 1 on the FCAT, or I have a mix of students in my class, whether that would make a difference almost like a peer effect.  So, that was studied. It turns out it makes a difference sometimes, so that’s included in the model.

Then, finally the school characteristics, which is related to the overall performance of a school and how the Committee was concerned and convinced that when schools significantly perform different than other schools in the state whether they’re very high performing or very low performing, that there is something in that school that it can also have an influence on student learning beyond the influence of the teacher.

The next slide reminds us that the Value-Added Model is one part of a multi-faceted teacher evaluation system as you have already seen in the previous rule. The last bullet down there that talks about the Student Growth Committee, the Department, and our contractor continue to analyze data annually, we’ve written into the rule that we would have an

annual review process of all statewide student growth measures.

The next part of the rule, now that we’ve established what’s in the FCAT formula because that’s the one statewide formula we have to date, does the remaining requirements which are talking about what is the score; what’s the value-added score? What’s all the data that you receive and we actually calculate value-added scores for teachers and other personnel?

The first thing that we have in the rule is to explain that, as you will be familiar with;; but, to make sure it’s codified in rule, that when you run value-added scores, they’re based on typical learning growth in the state by grade and subject.  A score of zero means that students, on average, performed typically as other students that share their characteristics performed across the state.  A score above zero means that students performed, on average, better than expected. A negative score, which means a score below zero, would indicate that the students, on average, performed less well than expected.

Other data that are gleaned from Value-Added Modeling, in addition to value-added score, we also know the number and percent of students in a class or a school or a district that met  or exceeded their statistical performance expectations. If we go back to the chart that showed this difference on Slide 23, how many students exceeded, met or exceeded that particular score, that’s something we gleaned from running a Value- Added Model so we can know that. That data may be useful in looking at performance.

One of the things you don’t get from looking at the percent meeting expectations is how far a teacher may have brought student learning growth in her class. But, you don’t get to see the magnitude of the game or how much student performance has declined.  You only know whether or not, yes or no, if students met expectations. But, that can still be useful information when looking at improving instruction.

One of the other things that we do with Value-Added Modeling because this is a statistical process, is we employ really good solid statistical practices.  One of the things that we gleaned in a value-added results and Value-Added Modeling is a standard error.  As a statistical term, it describes variability in scores. Teaching and learning are a human process. We get different students every year or there are some teachers that may have students for a number of years if there’s a looping process going on. Things happen in different years.

So, just like when you see on TV, a presidential poll and it says we’ve polled a section of registered voters in five states and the president’s popularity is plus or minus three points around 50%, for example. That means that if they went back and polled a similar group of people from those states, from registered voters, it could have been a different group of people but they had those same characteristics – registered voters from those five states – that they are sure within three points that the results would be the same. So, the standard error of a variability lets us look at a teacher’s value-added score and provide some certainty around whether if the teacher had been

assigned a similar group of students, even though they were different, we’re sure within this many points of the confidence in her role, plus or minus, that the score would have been the same.

We look at that, the standard error, and construct confidence in a role, because what we’re using this information for is to try to determine teacher performance. This is for individual folks’ evaluation. So, it’s important that we be as sure as we can that we are looking at and making adjustment about the performance using all of our good statistical properties that are available to us and processes that are available to us.

Then, the rule goes into performance standards. We have to establish performance standards that would be used statewide for using in this case, statewide performance on the FCAT model that every district then would implement. But, they would classify performance in the same manner.

But, how do we go about that? What’s our intent? This is one of the sections of the rule where it’s very important that we explain what we try to do in the rule text and that you let us know when you read that rule text if we got there.  Did we really meet what we intended to do?

As you know from what I said earlier that zero means typical performance for a grade and subject, that means when you run a value-added result half of the people would be above that, half of the folks would be below that typical performance. So, it’s important for us that even though we want to use a model that takes into account all of these differences and influences on learning like covariant adjusted models do that we also have a system for evaluation that’s based on performance criteria, rather than a normed approach. Because if we just use the distribution every year, then even if everybody gets better, it could be that folks at the bottom of the distribution are still classified as poor performance, when in fact, they could have greatly improved since the year before.

So, what we tried to do is use the good information we get from the Value-Added Model, but establish standards based on ’11- ’12 data that would then be used as an anchor or a benchmark against which we could measure performance in the future so that every year all teachers have an opportunity to exceed the points for effective and highly effective as they were established in ’11-’12 based on typical student learning growth for that year.

The...how does this work and we’ve talked about confidence intervals and we talked about the standard error to construct a confidence interval using one standard error on either side of a confidence interval provides 60% assurance that the teacher’s performance would have been the same or within that range given another set of students that were not identical, but similar to the students she taught that year, and then using two standard intervals to construct a confidence interval means that I’m 95% sure that it would be somewhere in this range if she were given a different group of students. The VAM score really does represent what happened that year. But, what you’re trying to do is use the confidence interval when you’re classifying performance to say because students are not

randomly assigned to teachers to make sure that you’re getting an idea of how that teacher would typically perform.

So, given the way we’ve set the standards in the rule trying to use ’11-’12 as a benchmark, the dark line you see in the center within the typical performance by grade and by subject that would take 7th grade math separately from 7th grade reading, separately from 6th grade math, etc. Based on the typical learning growth ’11-’12 year going forward, a teacher who, if you look at the standard for highly effective, the teacher’s confidence level using a 95% confidence level completely cleared that typical growth.  If you look at the bottom of the screen, the standard for unsatisfactory would be that the teacher completely cleared, even with a 95% confidence number, we are 95% confident that her performance that year based on student learning growth at least, not the rest of the evaluation, which tests the student learning growth part, was in fact unsatisfactory because it was definitely below typical for all of her students.

Then, we actually defined effective in two ways how you would expect, which is the score itself was above zero, above zero or higher, above typical growth, and even with a confidence interval, even if the bottom part of the confidence interval goes below zero, that’s okay. The score itself was above zero.  So, as long as the score itself was above zero, that’s good. That would be effective performance.

But, then what if the score was just below zero? Would those teachers have to be determined for that section of the evaluation to need improvement or be developing? What we defined in this rule for your consideration is that if the score is below zero, but the confidence interval allowed that score to have possibly been, and if she had been assigned a different set of students, to have been above zero or at zero, that performance could still be classified as expected. So, using the confidence interval allows us to move around that typical performance in a standardized statistical way that we can bring more teachers into the effective realm considering their student performance.

Now, one thing we can ask the law, the law says we have to divide those standards for a highly effective and effective unsatisfactory. What about...? What about developing? Well, the remainder – anyone who didn’t meet one of these classifications then – would be either needs improvement or developing depending on whether they were in their first three years of employment or not.

Let me go back one slide just to make sure that I made something clear before I go to the next slide. One of the other things we tried to do with the rule, as I told you and I thought applied to you in the law, it says we have to set the standards, period, so that they’re consistent across the state. So, that’s the first thing we needed to do.

What I’m explaining to you now then is our proposal for what those standards would look like for your comments and your consideration. If those go into effect, then what that means is for the performance of students’ criterion alone everybody would use those standards when they’re using data from this model, for just that portion of the evaluation. All the rest would

be combined however the district’s approved system combined things and summative rating is based on a combination of all those things. So, that’s the first thing that we proposed these standards be used for to meet that bar of consistency across the state that the law was calling for.

The second thing the law calls for, though, was to set those standards such that there had to be a threshold to being considered by the district as effective or highly effective.  Then, the floor, I guess, which you would need if you were below this measure, then you would be unsatisfactory.  So, the whole evaluation would be considered unsatisfactory. We call that the impact on our summative rating. What conditions would have  to be met for that part of the law to be applied to a person’s evaluation, not just where we apply them consistently across the state in that one performance of student criterion.  But, when does it take effect – the portion of the law that says you’ll be automatically unsatisfactory or you had to meet this threshold to be effective or highly effective as determined by the district.

So, we went back to the law and looked at the conditions that are already placed into law on using performance of students’ data.  For example, I mentioned earlier that the law says if you have less than three years’ worth of data, you can reduce the percentage of the evaluation based on performance of students down to 40%. It made sense to us that the automatic results in the use of these performance standards to impact a summative rating would only be effective for someone who had at least three years of student data.  Otherwise, you’re just using it for the performance of the students’ criterion.

The second thing we saw that was important was actually in Paragraph (7) (e) of 1012.34. The paragraph that I mentioned for classroom teachers, of course, is not assessed by the state where the district has not yet selected their assessment teams in that course for evaluation purposes and those teachers are being evaluated perhaps on team measures or learning targets or things like that. If that was the case for a teacher, then these performance standards would not impact summative ratings. They would still only be used, comparable standards, for the performance of students’ part of the evaluation only. They

would not result in an automatic unsatisfactory or result in being the floor for highly effective and effective.

We also determined that for non-classroom instructional personnel who don’t have direct contact with students in an instructional capacity, but in otherwise supporting student learning in other different ways, that the performance impact, performance standards would not have a direct impact on their summative rating. They would still only be used in the performance of students’ criterion.

The rest of the slides here are just reminders for people that have not been involved in the process in all of our districts and since 2010 things that we have done and the districts have done to develop and implement their evaluation systems. So, I’m not going to go into some great detail because I want to get to the public comments. What I’ll do now is just remind everyone that we’re in a three-part agenda. That was part one. That was the overview of what’s in the rules and what’s in the law.

Now, we’re going to move to Part Two, which is to clarify questions and answers and remind everybody if you’re on the phone or you’re on the web, when you’re asking a question or making a comment, please state your name. If you’re in the room with me today, we have a microphone for you to come speak into, not because I can’t hear you, but because we’re recording this webinar and for the sake of a good report we.... So, Anne, will you please compile the queue for questions and answers...?

Coordinator	We have a question from the line of Ryan Hess. Please proceed.

R. Hess	Hi, this is Ryan with the Escambia County School District.  I just have a clarifying question on the part of the rule where if a teacher is unsatisfactory for the student measures portion, then he or she would be unsatisfactory overall.  My question is:  Is the intent of the rule that where a state formula is used like FCAT that it’s based on that; that that’s the trigger alone?  Or,

the question would be if we were using FCAT for those teachers and then we were also, in addition to that, using a local district assessment that averaged that in?  So, the effect of the local district assessment for this question would be that it actually raised the teacher’s student growth score.  Is that permissible or is it to be based just on FCAT where FCAT’s available?

K. Hebda      Thank you for your question. That’s a great question. I’m actually going to make a reference to the draft rule text to answer the question.

We did, as I mentioned earlier, the law says the district should adopt equally appropriate formulas, measurements of learning growth for their own assessments. In Section (4) on the performance standards, if you go to Paragraph (c) of Section (4), we actually list there district established performance level standards. Then, when we talk about in Paragraph (d) applying the performance level standard, we are explicit that it is all the student growth that is included in that teacher’s evaluation.

So, whereas we have set the requirements for using FCAT data with this model, you would have your equally appropriate standard for other grades and subjects. If the teacher, for example, taught two courses that were related to 6th grade
reading and four courses that were related to 6th grade social

studies and you were using your social studies assessment for that course, then you would combine that teacher’s evaluation with all of that data; two parts FCAT, four parts social studies. Or, some districts have done that based on the number of students in all the courses and divided it that way.  But, you would absolutely under this rule combine all of that to look at the performance standards. The FCAT would not trump everything.

R. Hess	So, that could be done if the teacher just taught reading as well; but, we also had a district measure for reading that we wanted to use.  Is that correct or not?

K. Hebda	That’s an interesting question.  I’m not positive when it is actually a set of state standards that’s assessed by the state assessment if another measure or how another measure could be used. The law talks about using multiple data points, so I don’t see that it’s restricted. One of the things that we would certainly want to look at in the review and approval process is how much you weighted your local assessment for something that was based on state standards such as reading state assessments and how much you weighted the FCAT.

R. Hess	Thank you.

K.	Hebda	I will add one quick thing that in Hillsborough County, for example, when you talk about a 5th grade teacher that teaches a self-contained class who may teach reading, mathematics, social studies, and science, or a 4th grade teacher that does that; Hillsborough has assessments for 4th grade social studies and 4th grade science. So, they combine FCAT results for reading and math with their local assessment on social studies and science to get the teacher’s complete value-added score.

Coordinator	Our next question comes from the line of Laurie Palido.  Please proceed.

L.	Palido	Hi.  It’s Laurie Palido from Hernando County. I have a question about the actual wording of the rule in the definition part. There’s a letter (h) and a letter (j).  On the letter (h) it says newly hired by the district means the first year that they are employed in the district full time. What about job share teachers who are not employed full time? Where do they fit into this?

K. Hebda	If I understand your question correctly, and just for everyone else who’s listening, Laurie is in Rule 6A-5.030(1) in definitions. You’re looking at –

L Palido	...(h).

K.	Hebda	Right.  Newly hired by the district means the first year in which the instructional personnel are employed by the district in a full

time instructional position. There are, I know, in districts a number of part-time positions and you can calculate your part time hours enough to reach – what is it 441 – to make a year of service.  I think districts have to determine how they’re evaluating part time teachers, whether they’re part time enough, close enough to full time to be under the evaluation system or not. The law doesn’t specifically address part time teachers who may teach one course or an adjunct coming in to teach  one or two courses a year.  So, I think at this point we have not addressed that in the rules. We’ve only covered those that are full time employees.

L.	Palido	So, there’s no possibility of addressing that in this rule?

K.	Hebda	I don’t want to say there’s no possibility of addressing it.  I would just have to go back and research what the law says about part time teachers because all of this law relates to the definition of personnel that are in 1012.05, I think.

L.	Palido	Right.  It’s more the newly hired by the district.  Now, if a person

– it says such person are newly hired for their first school year of employment of a district regardless of their prior work experience elsewhere.  If a teacher taught in our district, then retired and came back, they are not considered newly hired, correct?

K.	Hebda	That’s a good question.  I’m not sure that they might actually be considered newly hired by the district.

L.	Palido                Okay, so that’s where that wording is kind of difficult.  Even if they taught here, and they taught in our district 20 years ago and came back, we would like to consider them newly hired. However, this wording I’m not sure we’re allowed to do that.

K.	Hebda	Thank you. I appreciate that. I think our intent was to say even if they had prior work experience anywhere, but we did use the word elsewhere, which almost could be an exclusion of the current district. So, I appreciate that.

L.	Palido	Now, also while I have you one the phone, (j) the performance level.  This says there are four performance levels for teachers in their first three years of employment.  Now, first three years ever in a district or ever in the teaching position in your district. Let’s say a person was employed as an administrator and then becomes a teacher.  Are they, in their first three years at that point because they have been employed with us, maybe never as a teacher, but it’s their first three years of employment? We have the same thing with paras going to teachers, or teachers if they had a big break in service. When does that first three years start?  Is it employed as a teacher for the first time?

K.	Hebda	That’s a great question.  I think that’s one that we’ll need to look into to be sure about. Some of those situations are obvious.  I would think going from a para-professional to a teacher, it’s your first teaching job; that would be first employment as a teacher; your first three years doing that.

But, the situation you’re discussing where you were a teacher, were an administrator, now you’re back in the classroom.  I

don’t know the answer to that question. It doesn’t sound it’s your first three years as a teacher, but –

L.	Palido	But, the way the rule says, it’s your first three years of employment.

K.	Hebda	Right. Well, that wouldn’t be your first three years of employment. You’re right.

L.	Palido	Thank you.

Coordinator	Our next question comes from the line of Amy Bolte. Please proceed.

K. Shivley	Kathy, this is Kevin Shivley from Pasco County.  I just want to clarify that districts, according to this rule, no longer have the option of using the percentage of students meeting or exceeding the Value-Added Model target. We have to use that raw VAM number moving forward?

K. Hebda	That would be correct. Once these would go into effect, that would be the case to meet that requirement that we’re looking  at the data the same way across the state.  If you look in the section I was mentioning earlier about the district’s “equally appropriate” measures, how you can establish those, it would be a way for you in your own local assessments to look at whether students met or exceeded the expected scores on local assessments to make those equivalent to what you might see in the Value-Added Model.  But, for the sake of consistency  across the state, it would be that once these went into effect, which would be no earlier than next year, the way the rule is written currently then we would all use the same score.

K. Shivley	Thank you.

Coordinator	Our next question comes from the line of Diego DeRosa.

Please proceed.

D. DeRosa	Hi, Kathy. This is Diego from Broward.  Good morning.  Just to follow up on that last question because the way I saw the rule where you included that the percent meeting expectation could be included as a portion, and you’re saying for the local assessment.  So, I’m sensing that we’re running – I mean percent meeting expectation refers specifically to an expected score that comes from a value-added statistical progression model. That means that locally we would be running those kinds of models?

K. Hebda	No, you wouldn’t have to use a value-added model. What I meant by that is you could set standards, if you looked at the percent meeting expectations under the Value-Added Model for your FCAT folks that were teaching those courses, and you were trying to look for something comparable to use for social studies teachers on the local exam, you might be able to use that information to try to set standards in your social studies class just based on the passing score for the test.

We tried to leave that as broad as we could. We even put a caveat in there that said if you don’t even like one of these two

options for setting equally appropriate measures locally, you can do another one and just explain to us what your rationale is for doing that.

D. DeRosa	I understand. Let me ask another question while we’re here.

It’s a little bit different and I have mentioned it to a couple of...that I’ve been in touch with that...you as well. We use a...expectation in Broward and we ran into the ceiling effect that AIR, the company that did the analysis realized that that was an issue.  I didn’t know if it had been an issue for a district that used the VAM estimate and I talked to a couple that said they did.  I know you’ve said somewhere where they’d be going back to discuss possible adjustments. Is that one of the possible adjustments?

Because for us that was quite an issue where we had teachers of high performance students getting more needs improvement and that’s obviously an unintended consequence of this. I mean if AIR had noted so early back in February, March, April of 2011 or whenever it was that ceiling effect was an issue, why hasn’t that been incorporated or looked into further? Why is it a

whole year of VAM data went through and it happened and still not being talked about at that level?

K. Hebda	Thanks, Diego. I appreciate that comment.  It is something you and I talked about a couple of times before today.  The ceiling effect is going to matter more if you’re just using a percent of meeting expectations than it will if you were using the actual score because of the degree to which learning growth is measured with the score as opposed to the Yes/No, which is what the percent and the expectations looks at. Some of that is mitigated with using the score. But, it doesn’t happen to a lot of people, but it does happen.

So, it is something that we’re looking into. What we’re trying to discover and figure out that we want you all to help us with is the model itself does a lot to level the playing field for a vast majority of teachers that teach reading, language arts, mathematics courses. But, it can’t do everything.  So, when it can’t do something for a certain segment of teachers or certain populations, then what can either we substitute in its place or

what can the district do in its evaluation system to mitigate that issue and get to a better performance rating?

Those are the kind of questions that we want to discuss. I’m not positive that we can take care of it in the Model itself because we’ve actually asked our national review group, our VTAC to look at that, too, last year. They’ve not come up with any real solution for the Model. I think the solution’s going to lie in how a district deals with it in their evaluation systems.

But, you’re absolutely right.  It’s something we need to work on between now and the end of this year.

D. DeRosa	Thanks for that.

Coordinator	Our next question comes from the line of Gisella Field. Please proceed.

G. Field	Hi, Kathy. This is Gisella Field and I’m sitting here with my team. I want to go back to Diego’s question regarding the percent meeting expectations.  I know you’ve kind of indicated how you guys are interpreting the rule. We really still believe it’s a little different. The way it reads in (4)(c) Part (4) where it says for those teachers whose value-added score includes a large degree of variance as determined by the confidence interval, the use of this data element can provide additional evidence.

The reason I’m asking this is piggybacking on the AIR technical report that was published in January when it talks about the magnitude of the variants. There’s a statement in there that says, “The models are not able to distinguish between more and less effective reading teachers.”  It says, “It can distinguish for math, but it cannot for reading.”

So, we assumed that this part of the law was sort of giving us some opportunity to make some additional adjustments knowing that AIR has pretty much publicly put on their technical report that they can’t distinguish.

K. Hebda	Thanks. I’m going to ask you a clarifying question, if you don’t mind, because the draft that I’m looking at under (4)(c.) which begins with “District established performance level standards”. Is that where you are?

G. Field	Well, let me see.  Oh, you’re right.  I guess the overarching element is district.  Okay.  I guess the question isn’t. It particularly reports to those areas where there is a large variance, a degree of variance on the confidence interval and we already know we have issues with reading. We were thinking that that would be a way to mitigate some of the problems.

K. Hebda	It could be. That’s a great suggestion. We talked about the text further and you’re there with your team and working with others, let’s see how we can clarify that and either come up with solutions that we can broadly describe in the rule or things that we just, if they’re very, very detailed, perhaps we even want to just provide those through technical assistance to everybody.

But, you can let us navigate that as well; help us navigate that as well.

G. Field	All right.  Thank you.

Coordinator	Our next question comes from the line of Laurie Palido.  Please proceed.

S. Jackson	Hi.  This is Susan Jackson in Fernando County.  The language that says beginning in the 2013-‘14 school year for all personnel, at least three years of student performance data must be used in the evaluation when available.  If less than three years of data are available, all available years of data must be used.

In the instance of, for example, social studies teachers who in ’11-’12, and ’12-’13 were evaluated using FCAT data, and then in ’13-’14 have a district EOC, will those two years of off-content data count as part of their three years of data?  Or, can that first

’13-’14 year when they have an on-content EOC be counted as their first year of data, their baseline data?

K. Hebda	I appreciate that.  For purposes of this rule and for setting the performance standard that would actually go into effect to impact the summative rating, meaning the automatic unsatisfactory or something like that, then the way we conceive it and the way we think we’ve written it, because the exception to when to when those apply include when you’re being evaluated on those temporary measures, that that person would only be considered as having one year of data for the purpose of applying those performance standards.

S. Jackson	Thank you.

L. Palido	It’s also Laurie Palido here. I have a question about the instructional team that is mentioned in the rule.  I know a few districts that I know we in particular have some instructional personnel who have no student contact whatsoever.  They’re

housed at the district office. They’re doing other jobs; never have anything to do with students.

Where do they fit in as far as the student learning growth? They’re not really part of an instructional team unless we consider all the teachers in the district.

K.	Hebda	At this point, my first response would be that they would fall in a classification in the law of non-classroom instructionals.

L.	Palido	Right.  But, what students growth?  Is it just all district?

K.	Hebda	Well, that depends on what they’re job responsibilities are. I know that one of the things we talked about was districts throughout this process. They have sort of a sub-set of people that are still classified as instructional even though really their duties look much more like administrative duties than they do instructional....

L.	Palido	Correct.

K.	Hebda	You’re kind of caught locally – I put them in this hole, but they’re really in this other bucket over here.  So, I think in this particular case the rule does not address that because those positions  are pretty unique in each district. I think the district’s going to have to determine if that person really is instructional personnel or if they’re really administrative personnel.

L.	Palido	Right.  But, if they’re members of the union, we’re not going to be able to take them out of that instructional category.  But, we’re not going to put them in the classroom either. I know a lot of districts are having this issue. Can that be addressed in the rule at all?

K.	Hebda	We can work on that.  In fact, since you spoke up, do you want to put a team of folks together? Let’s see if we can get some good language around that to help define those situations for people. I don’t know if we can take care of every single caveat,

so we’ll need to be careful with that because they really are different in every district.  But, it is an issue.

L.	Palido	Okay, thank you.

Coordinator	We have no further questions from the phones.

K. Hebda	Great, thank you.  I do have one question from the web that I’m going to read out loud. It’s from Katherine Morrem.  She says, “When will the FDOE student growth formula options be available for other subjects not statewide assessed?”

Just to be clear, I did mention the Student Growth Committee is meeting on the 27th of this month to look at the other statewide assessments that are in the course assessments. I also mentioned on a previous slide that we have responsibility as a Department under the law of providing you options and example growth models for things you can use locally.  The SGIC, Student Growth Implementation Committee, has already considered one of those and I’m hoping we’ll finalize that at

their meeting and then be able to work with districts on how we can get that done.

They’ve looked at one for the SAT-10 assessment that some districts used in the early grades across the state. They’ve looked at a model they think works pretty well for SAT-10 learning growth.  The ones that will be example models that districts could choose to use such as that would for SAT-10, that those would pertain to more local end of course assessments that might be developed through the item bank that’s up and running next year. We plan on getting to those at the beginning of next year when we actually have something from the item bank we can use as examples. So, by the end of next year, you should have several example growth models that you can use – actually the middle of next year.

I just want to make sure that everybody understands. Those are optional. Just because they’re something that’s provided through technical assistance for a local assessment, you can choose to use something different.

Any questions from the audience that are present? Clarifying questions?

M	...

K. Hebda	Yes, if you would, Tony, if you don’t mind. Hand you the microphone...web reporting.

T. Demma	This is Tony Demma from Meyer & Brooks Law Firm in Tallahassee. I had a question that was dovetailing other questions about the data about the percentage of students performing above or below.  I wanted to get a clarification from you that teachers don’t ever get to see the individual student VAM score for those students that are on their rosters.  Is that correct?

I don’t know if Johnny did tremendously and he was my success story, or if Sally is really a problem. I don’t know if the students that are in my class that are ESE or have certain types

of learning disabilities or learning problems did well, did badly, because part of this has to be diagnostic to help.  It can’t just be punitive. That’s one question.

Second, I don’t really understand if you were saying that the data doesn’t show how much each student performed above or below.  It has to if they’re computing a VAM. So, what you’re saying is that’s not being shared with people. Is that correct?

K. Hebda	Thanks. I’ll go with it and answer the questions. Two questions, actually.  There are two pieces of data, never a...of the data.  But, the two we’re talking about right now are the score itself, which is based upon all of the different various levels of growth that those students assigned to that teacher made, calculated together. Then, the answer to the question for each one of those students even though the school work includes all the magnitude of learning growth or regression.

We also can answer for each student whether or not that student met or exceeded their expected growth, yes or no.

Both of those pieces are provided in the VAM score.  And, in fact, both of those pieces of information are provided to the districts. They have that information right now.

T. Demma	The yes or no answer?

K. Hebda	They have all of that. They have both the score; they have the pieces and components of the score meaning that the teacher has three classes of 6th grade reading and two classes of 7th grade reading. They have each one of those separately, and they have the aggregate VAM score, and they have which students were assigned to the teacher and whether that student met or exceeded their score and by how much.  So, you can see those things in the data.

Now, what may be difficult for some districts is having a mechanism to provide that to teachers right now.  One of the things the Department’s working on is a web-based interface for teachers to look at their own data. We want to pilot that this spring so that they can see their own students from that year,

that information exactly like you’re talking about. They would also be able to see, which the district couldn’t share with them even if they wanted to because they don’t have access to it, how they look compared to teachers across the entire state teaching the same subject and grade level.  So, it would give a teacher a way to look at her performance as the students did in her class, which as you said is wonderfully helpful and diagnostic when combined with all the other kinds of data a teacher might have, and also gives her an idea or him an idea of how other teachers who are teaching similar populations or similar grades and subjects might be doing. So, really we do want to share all of that information with folks.

T. Demma	So, can I follow up from here...I want to just sum up what I thought I heard you just say.  Districts have the information I was just asking about whether or not – and I’m not getting into the technical aspects whether or not they have difficulty providing it to teachers.  But, you’re telling me that it is made available to school districts, the type of more specific data on the part that says what percentage of students were above or below.  That part just has yes or no? That part is yes or no.

But, there is other data the districts have that would allow them to let a teacher know how each of their students actually did on their VAM scores. Is that correct?

K. Hebda	Yes, that’s correct. As I mentioned, I don’t know if everybody has the capacity to easily share that with teachers, but it is available.  It’s made available to school districts, that is.

T. Demma	But not prohibited...sharing...?

K. Hebda	But not prohibited. They are not...not prohibited from sharing.

No, they’re not prohibited from sharing.

Other questions from the field?

T. Demma	I have one more. This is Tony Demma, again, from Meyer & Brooks Law Firm in Tallahassee. This is about the instructional practice rule. I’ve forgotten the question.

K. Hebda	I have some here I can read while you’re thinking.

T. Demma	Yes, go ahead.

K. Hebda	I have a couple of these from the web. This is from Dawn...in Dade County.  “We were under the impression that SAT-10 was not aligned to common core.  Do you all know differently?  Is there a plan for aligning it to common core that you know of and when is it due to be aligned?”

Thanks, Dawn. We don’t control what’s on the SAT-10. However, we know that districts do use some. The districts do use SAT-10 in early grades. Because of that, it was requested by the Student Growth Committee to be a model that the Department considered looking at optional growth measure for. So, we’ve done that to accommodate those districts who use SAT-10. Then, it’s going to be an instructional decision whether or not a school district chooses to use that assessment or not.

Another question from the web.  “For clarification, if the teacher’s value-added score is based on FCAT reading and math data, we are still able to combine the average score with an additional local assessment so long as the calculations of that assessment utilize similar growth models and the overall equals 50%.”

I think that question is yes. Remember, the fundamental principle in the law is that you’re evaluated on what it is you’re responsible for teaching.  So, if I teach multiple subjects, then ideally, I would have the assessments that are used for the students in those courses, whatever is used to measure student mastery of the content of the course, whether it’s one assessment or multiple assessments, the district chooses from those assessments that they already use to measure student performance to measure teacher performance. The state provides those to those that are based on statewide assessments and courses; and, the district provides the remainder. Then, whatever the teacher teaches is combined.

Other questions?

M	...

K. Hebda	Okay.  One last question from Tony Demma.

T. Demma	It’s a two part question, but I can sit down after that.

K. Hebda	Your questions are always two parts.

T. Demma	Counsel can object.  In the 503 overall about instructional practices, there are a couple of places, let’s see, (6)(b)(2) and
(3) talking about district documentation of how indicators and/or domains are prioritized to present evidence that the evaluation system is designed to support effective instruction and student learning growth.  I can look at a set of the Florida Educator Accomplished Practices.  I have no idea why things must be prioritized, how they should be prioritized, or what you’re even talking about there. That shows up in that part and it’s on page 6 of that rule.

K. Hebda	Page 6...Section 6...

T. Demma          It’s hard. If I don’t go all the way back to the beginning, I can’t ever tell what section. But, I don’t really understand prioritized in that context.

Then, subsequently on that same page, at the bottom, there’s discussion of the scoring and weighting system. I don’t understand just reading it what a weighting system is you’re talking about where it comes from. What happens if you weight it and why?

K. Hebda	I’m going to get to your second question first because I know the answer to that question without finding specifically what you’re talking about in the rule.  But, I’m still looking for the other one to answer the first question.

The scoring and weighting system means the district has to determine – there is a valuation – it’s in three parts:

performance of students, instructional practice, or instructional leadership, and then professional job responsibilities.  So, how is the district going to determine a summative rating of highly effective, effective, needs improvement/developing, or unsatisfactory without combining information from all those sections. The scoring system and the weighting system is how does the district combine all of this stuff in the evaluation system to come up with a summative rating.  So, how does that happen?

Now the first part about prioritizing, I’m making sure that we’re on the same page here.  My pages aren’t numbered, so I’ll use the one off the web.  I’m sorry.

M	...

K. Hebda	It’s got to be in instructional practice. Oh, thank you.  I found it.

If you’ve got a copy of this elsewhere on the web or someplace else, then it’s in the section on instructional practice in Rule 6A- 5.030, which is in sub-section (3), under instructional practice,

which is (b) and then number 2 under that is for classroom teachers’ observation instrument. This is what you said...evidence that they’ve met the requirement under instructional practices. The first thing was a crosswalk between the district’s selected research base and...law says contemporary research and whatever framework they’re using to implement the Accomplished Practices.

The second thing is it says for classroom teachers observations instruments that include indicators organized by domain based on each of the Educator Accomplished Practices. The district must provide documentation of how indicators and/or domains are prioritized to present evidence that the evaluation system is designed to support effective instruction and student learning growth.  That means that there are a number of things in the evidence in the Educator Accomplished Practices.

The district may prioritize those things to say these pieces of this section, this domain of the evaluation system; and, domains is really just used in a manner to help organize the observation instrument, so it’s not just a laundry list of things that you may

be looking for that organize by domains or if you can think of those as guiding principles or questions. When a domain or even in the whole observation instrument, the district may put a premium on things they think have the biggest direct impact on kids’ learning, which may be those things that are specific to what happens in the classroom every day as opposed to something the teacher may do for professional development or something else that may have maybe importance, and part of the Educator Accomplished Practices that it might not be a priority of the district to help students learn. A district can choose to prioritize those things or explain how those things are prioritized to make sure that they’re getting that student learning.

T. Demma	I’ll walk up here again. I’ve got a follow up on this same question. You used the words may and can several times in your answer.  This says the district must provide documentation of how indicators are prioritized.  I read that to mean that either they’re going to feel they have to or you’re going to tell them they have to. It’s supposed to be a local assessment, which as a general concern for me is the problem with a lengthy rule like

this that has so many moving parts and as all the questions said, they don’t know what the answer is so they call and ask. It either must prioritize or they can prioritize or they can do it any which way they want. That’s what I heard you say.

K. Hebda	They must prioritize and they may prioritize any way they want.

T. Demma	But, they could tell you anything...have to say.

K. Hebda	What Mr. Demma said from his seat was that they can tell me what?  I’m sorry?

T. Demma	That everybody can prioritize everything the same. They can call it...

K. Hebda	His question was they could prioritize everything the same.  I don’t know if that qualifies prioritization.  However, I think the simple answer to your question, which is the way the text reads

today is that you must prioritize, but you may prioritize anyway you choose. That’s the way it reads today.

Okay, one other question; oh, two others, excuse me, from the web.  This is David Morrell:  “Does the automatic trigger (must be unsatisfactory if a certain level is not met) apply to highly effective and effective summative scores well...to the score on the other observable piece?”

The answer to that is no. Those work a little differently.  For effective and highly effective, the law of Florida sets the student performance part at the floor. So, for impact on summative rating that the teacher got through use with the data and their being evaluated on what they actually teach for three years,  and that goes into effect for that teacher, it would be a score on the performance of students section that would be a teacher would have to meet to be considered by the district once everything else was combined to be effective or highly effective. It’s not automatic the same way the unsatisfactory is.

The last question I think, from Karen Schaeffer...excuse me, Chris Master.  “I have a question and I had my hand up.”  I’m sorry, Chris, I didn’t see your hand up. I apologize.  “Do we need to have a rubric at the indicator level, or can they be at standard level?”

They can be at standard level and I think your evaluation system in Dade County has them at the standard level right now.  If a district determines that the rubric’s at the standard level, there are places where they need to modify those or they are having trouble getting interrelated reliability with rubrics only at the standard level, then the district can add some at the indicator level to solve the interrelated reliability problem and provide clarity.  Because, remember, what you’re trying to get in this system for the observation instrument is really good feedback. Although the rule doesn’t say you’ve got to have a separate rubric for the indicator, that would be kind of complicated, you do want to make sure that your rubric for looking at that evidence really is something that people can find consistency around and will give the teacher good feedback. That’s the whole point of even suggesting that you have rubrics.

Now Karen Schaeffer. “Has the new model...of effectiveness using confidence intervals apply using teacher...estimates at grade level and subject aggregate?”

That’s a great question. We provide districts the individual grade and subject value-added score. But, then the aggregate score is turned into a proportion of a year’s growth.  That’s where, if these standards go into place, we would use the aggregate score.

It looks like the last question is from Katherine Morrem. “Can you please clarify when the unsatisfactory VAM performance trumps the instruction practice score?”

Yes.  Based on what we have in this rule today for your consideration, is that would only happen if the teacher had both three years’ worth of data and that data was based on what she was teaching.  It wasn’t a temporary measure like in instructional team score or something like that.  So, it has to be

data on what you’re teaching and you have to have it for three years.

I think that’s all the questions. Operator, did anybody else line up?

Coordinator	Yes, we have one more from the line of Ryan Hess. Please proceed.

K. Hebda	Thank you.

R. Hess	Sorry, again, but I have those same questions I’m unclear on to follow up really on the ones you just had. For highly effective and effective, what’s the floor on those when the student growth portion triggers the other?  I’m unclear on how that works.

K. Hebda	The law...performance standards in two ways, one for consistency and then one that would actually be what somebody is now calling a trigger, but would be an automatic

impact on the summative rating. We defined what the performance standards would be. Then, we clarified when would they go into effect and for whom.

They would go into effect for everybody that teaches a course related to the state evaluation data and then districts would select equally appropriate formula to be used, and standards to be used beginning, the way the rule is written now and for your consideration, it will be next year – ’13-’14.  But, that’s just for use in the performance of students’ criterion of the evaluation. So, we all have a consistent way of using this data in the performance of student section, but then that’s combined however the rest of the evaluation went for that teacher as the district’s evaluation plan see fit.

R. Hess	So, you’re saying that it doesn’t apply for highly effective and effective. There aren’t rules on the student growth side that would override their final rating for highly effective and effective; only for unsatisfactory.

K. Hebda	All right, I think that’s a different question. We need to look into that one, too.

R. Hess	Well, that’s the question I’ve been trying to ask.  I understand the classification of employees for whom it affects; you know people have three years of data, those subject area teachers. I just don’t understand when it kicks in for those people.

K. Hebda	When it does kick in, when they have all the three years of data and everything, and if it were my evaluation and my student growth score was as we looked on – if I can get to that other page again because I want to show you the picture – on Slide
31. If my score and a 95% confidence interval were all below typical performance, which is typical state performance in my grade level and subject, then my evaluation would be unsatisfactory that year.  If all the other conditions were met, with the three years of data and I’m actually evaluated on what I’m teaching for three years and my score and the 95% confidence interval are all below zero, then that’s when and I should be unsatisfactory regardless of what else is in the evaluation.

Those standards for effective that you see on Slide 31 were either the scores above zero itself or the scores below zero, but with a confidence interval it could be considered as effective performance above zero. Then, that would mean I had met the floor for effective.  So, my district could consider me effective, could, if the rest of my evaluation played out that way.  They could make me effective or even highly effective.  But, I at least met the floor for effective.  So, I met that condition to be considered by my district as effective.  It’s the same thing for the standard of highly effective. When it impacts the final  rating, it means that I can be considered by the district to be highly effective.  It’s not an automatic the way the unsatisfactory is.

R. Hess	Let me just make sure I understand. I’m sorry for having a hard time on this.  If I got a student growth rating label of needs improvement based on what you just showed us and my rating for instructional practices was effective or even highly effective, but then the way the district had agreed and bargained out that those two halves were averaged and the final rating, it couldn’t

be above needs improvement because that’s what I got on the student performance test.  Is that right?

K. Hebda	That would be right if you were talking about unsatisfactory.

The law doesn’t address needs improvement. That only addresses unsatisfactory.

R. Hess	So, that’s right for unsatisfactory.  But, now what about the highly effective and effective part?

K. Hebda	Highly effective and effective part is just a threshold. There’s a student growth performance that would be a threshold. So, if you met the threshold, if your score was above zero and all the other conditions were met, then it enables the district to consider you as effective based on how all the rest of your evaluation went. It’s not an automatic effective.

R. Hess	But, it’s the same rule then because if I didn’t meet the threshold, then I’m needs improvement, which means I couldn’t

be effective overall regardless of what happened on the other half of the evaluation. Correct?

K. Hebda	That’s correct.

R. Hess	So, we really –

K. Hebda	If you wouldn’t...could be unsatisfactory.

R. Hess        I wouldn’t automatically.  But, I can never be higher than the student growth portion of my evaluation. Is that an accurate statement?

K. Hebda	Once those actually would be applied, yes. For the time being, it’s not going to affect anybody.  But, actually if you had three years’ worth of data and you were being evaluated on what you actually teach, then yes, that is actually the case.

R. Hess	So, effectively, the student growth half always trumps the other half in the final score given those other criteria about having three years’ worth of data and whatnot.

K. Hebda	That’s right.  Either it’s automatic when it’s unsatisfactory or it’s a threshold for effective/highly effective.

R. Hess	Is that level of requirement all in the statute or is there an additional level that’s being applied in the rule making here that’s governing the ratings above unsatisfactory?

K. Hebda	The requirements to establish those standards is on Slide 18. I went back to Slide 18 and that’s the quote from the law. The law requires that we do that. What the law doesn’t say is what those standards are. So, the standards we put in the rule of either above zero or below zero, but your confidence interval says it could be above zero.  Those kinds of things are in the rule as...the standards of the law requires us to set. So, we put those standards out there for your consideration.

R. Hess	Some teachers might be confused by this and think that the student growth portion of their evaluation is 100% of their evaluation. How can we explain that to those teachers who are getting confused in that way?

K. Hebda	I don’t know how you can explain that other than to say at some point when they have all those conditions met, if nothing changes, then it will determine their final evaluation.

R. Hess	So, just to move on to one other thing, then. I just want to clarify when you’re talking about indicators; rating at the indicator level wasn’t required. Were you talking about the teacher evaluations, the leader, or both?

K. Hebda	It’s both. In that case, it’s both.

R. Hess	So, for our leader evaluations, we do not have to rate at the indicator level?

K. Hebda	You need to have indicators of performance, otherwise it’s...not clear if anybody is collecting evidence.

R. Hess	So, we don’t have to collect evidence and rate at the indicator level?

K. Hebda	You need to rate over all in your domain...but you’ll need to somehow translate what you collected at the indicator level into a rating.  So, that’ll be your choice how you do that. But, you’ve got to figure out a way to get that evidence turned into a rating of some kind.

R. Hess	Do we have to collect evidence at the indicator level?

K. Hebda	You do need indicators of performance, right. Some people have used for instructional personnel; they’ve used the Educator Accomplished Practices themselves and split them up into parts to make them indicators. Others have added to that.

For the leadership standards, you could take leadership standards and break them into little parts to make those indicators of performance. There are ways to do it with the information in front of you.

(side conversation)

R. Hess	So, for the leadership, there are 45 indicators; on the state model, there are 45 indicators. We don’t have to rate at each of those 45 indicator levels is what you’re saying?

K. Hebda	No, you may reduce that number.  You may add to that number.

You may change those around.  You could do anything you want.  That’s just a model.

R. Hess	Thanks for that clarification. I think we might have gotten some other guides at one point or interpreted it incorrectly.  But, it helps, thanks.

K. Hebda	Good.

Coordinator	Our next question comes from the line of Mark Howard.  Please proceed.

B. Johnson	Hi, this is Bob Johnson sitting in for Mark Howard in Palm Beach County.  I just have one comment and one question. The comment was on the former discussion when the student growth factors are going to really overwhelm or dictate the final rating.

I got totally lost in that. If at some point in time that could be put in writing in some kind of form, that would be probably helpful so we could understand it better. That was the comment.

The question is this. When it came to the unsatisfactory which is what you were talking about a little bit ago in terms of it overriding any other considerations when the unsatisfactory forms in the student growth area.  My question is this.

We’re talking about three years’ worth of data.  Are we talking about three years’ consecutive unsatisfactory ratings, one for each year? Are we talking about some kind of aggregate across the three years that falls into the unsatisfactory rating? If we are talking about aggregates, the state has not yet provided the opportunities for districts to adjust student/teacher rosters going back three years.  So, how does that all fly when you’re combining multiple years to get an unsatisfactory?

K. Hebda	Thanks. I think I’m going to address the one thing first, which is the multiple years of unsatisfactory.  This doesn’t combine multiple summative ratings. This is looking at students’ performance data over three years. So, if a person was unsatisfactory a previous year, that doesn’t necessarily mean they’re unsatisfactory this year.  It’s not combining how many unsats do you have for the purpose. This was just looking at student data itself over three years.

What we’ve done so far is considered, even though we gave you historical data for modeling purposes and things like that from previous years, we consider ’11-’12 year one.  So, we

provided an aggregate VAM score for ’11-’12, and then we’ll provide one this coming year for ’11-’12 plus ’12-’13. Then, the next year we’ll provide you with one that’s ’11-’12, ’12-’13, and ’13-’14. That would be three years. So, for anybody, until that year that that standard would apply to impact the summative rating at the....

B. Johnson	Okay.  I’m still a little bit unclear.  If you have three years of data, it’s the unsatisfactory that overrides the other areas. Is that only for the latest of the three years, or does the unsatisfactory have to be somehow across all three years?

K. Hebda	No, it’s the latest of the three years. When you put all of that student data over three years together and get one answer, and it’s unsatisfactory, that’s when it’s unsatisfactory.

B. Johnson             Can districts propose ways in which to combine the three years or is that something the state’s going to do to combine the three years together?

K. Hebda	Right now, we have a process to aggregate those data. It’s in the technical report.  But, if you would like to propose different ways to provide an aggregate score and not turn it into a proportion of a year’s growth or any of those other things, then we would always want to hear that.

B. Johnson	That makes sense. Thank you.

Coordinator	Our next question comes from the line of Katherine Morrem.

Please proceed.

K. Morrem   I’m sorry.  I forgot that I could get on the phone here.  Kathy, my question is this. Again, it’s just with the more clarification on the unsatisfactory VAM trumping their overall scores.

What if they had an effective overall score in the years previous or in the former years, but again, that next year the VAM student growth score is unsatisfactory?  Is that it? Does it have to be the three years prior to that that were unsatisfactory.  I

think that’s just very similar to what the other gentleman was asking, but I’m still unclear on that.

K. Hebda	It is. We’re just talking about the performance of students’ data.

When you combine performance of students’ data for three years, don’t think about instructional practices for a second; don’t think about what my overall rating was the past two years.

K. Morrem	Okay.

K. Hebda	Only if I combined my performance with the students’ data for three years and it’s what I was actually teaching, that determines that year combining looking across all three years rolled together is unsatisfactory...evaluation’s unsatisfactory.

K. Morrem	Okay.  Three cumulative years of unsatisfactory student growth performance –

K. Hebda	No, no; not three years of unsatisfactory. Three years combined together. When you put all those data points over three years together, fresh eyes, three years of data, fresh eyes, it’s all below zero plus a confidence interval level of 95%, it’s unsatisfactory that year, that year. It does not matter what you were rated previous years.  It’s not rolling up how many unsats or how many effectives or anything like that you have. You’re just looking at your student performance data.

K. Morrem	That VAM that came out that year takes into...

K. Hebda	You were evaluated that year and that year’s over, done.  Now, we’re looking at the next year.  Then, the next year, the law says to really get a good picture of teacher performance, you should aggregate probably three years’ worth of student data to do that. So, we aggregate three years’ worth of student data and we get a performance of students’ ratings.  If it is three years’ worth of data in my content area, and all of that combined together is still below zero with a 95% confidence interval, then my overall evaluation for that year is unsatisfactory.

K. Morrem	We won’t expect to see that since 2011-2010 results count as year one, until ’14-’15?

K. Hebda	That’s right.

K. Morrem	So, 2014 there’s a release of the VAM results saying that the teacher’s VAM, just the student growth portion comes out unsatisfactory – can you finish the sentence for me?

K. Hebda	And, if that teacher’s VAM score was based on three years’ of data in the content area she was teaching those three years, then her overall evaluation is unsatisfactory, if nothing changes between now and then.

K. Morrem	Okay.  So, for districts that in the year 2011-’12 did not use the same student growth calculation formula that you are supposing here using the covariant and the standard deviation model, then that would trump our situation. Actually, this new coming year would be year one for us.

K. Hebda	It really doesn’t matter what you did in ’11-12 and ’12-’13.

We’re...in the ‘14’15 school year, take the data that we have performance of students assigned to that teacher for three years and we’re going to calculate a brand new aggregate score that looks at three years of the teacher’s data.

K. Morrem	I guess because if we were using the formula that allowed us to look at percentage of students who met or exceeded expected performance, that doesn’t matter. You’re saying that the way the VAM was released in 2011-’12, that’s what counts for year one using that variant model.

K. Hebda	The student performance from that year will be rolled up to the next two years into a brand new rating.  Remember, you’ve got to evaluate teachers every year.  So, you made a decision based on what you had and what’s in your plan for ’11-’12, and that decision’s done. ...decision for ‘12’-’13; when you finish that that’s going to be done. When you look at that data, you may consider ’11-’12 and ’12-’13 student data in that decision.

When we get to ’13-’14, we’re going to go back and look at three years of a teacher’s data, because that’s better than looking at one. When you’ve got three, you should use three.

Then, we’re going to look at three years of a teacher’s data and we’re going to give her another score for the new evaluation because we get evaluated every year.  That evaluation is made up of that year’s instructional practice and three years of student performance data. If the student performance data is unsatisfactory and it is three years of what she taught in her subject, then she’ll overall be unsatisfactory.

K. Morrem	Thank you. That clears it up.

K. Hebda	Good. We have one more question. Then, I’m going to go ahead to public comments because you only have 15 minutes left.  I know people want to say things.  If we run out of time, just let me say now real quickly, that we will of course have online, please type your comments online. We’re getting all of

that information. So, one last question from someone here in the room.

J.	Gerber	Jolene Gerber, Leon County schools.  So, for smaller districts that don’t have a library of local assessments that have been administered, in practice what this means is our 3-8 reading/math FCAT teachers will have a Value-Added Model score of ’13-’14 because they’ve made of three years of historical data that ties them in the content area they teach. Then, for all other subject area teachers, essentially what’s going to happen in practice is in ’14-’15 when we go live with our local assessments, it’s going to be three years from then. Is that correct?

K.	Hebda	That would be correct. The only thing I would say as a caveat, though, is because there are districts that, of course, are using their AP data now and all of those...assessments you already have in place. So, it could be also for AP teachers that that’s their first year where they have three years’ worth of data in their evaluation as well.

Now, we’re going to go to Part Three, operator.  I know you may still have some people in queue.  So, I’m going to ask that those people that are already in queue if you have questions, if you would submit your questions separately in writing also, so we can try to provide responses to those. Then, anybody who wants to make a comment, please get in line with the operator now.

Coordinator	(Instructions given)

K. Hebda	Anne, let me make one other comment really quickly.  Even though I read the question from the web out loud so everybody could hear them, I’m not going to read the comments that are submitted via the webinar out loud because we recorded the webinar and those will stand on their own.  So, just in the interest of time, we’ll just take the comments that are oral comments during this last 15 minutes.

Coordinator	Our first comment comes from the line of Karen Schaeffer.

Please proceed.

M	We were in queue for the question. We don’t have a comment.

Coordinator	Our next comment comes from the line of Ryan Hess. Please proceed.

R. Hess	This really is a comment.  I was thinking about the issue I was asking questions about.  I’m not asking more questions, though.

But, my comment on the rule would be to request consideration that the triggers on the student growth side that basically override the final effectiveness rating for the teacher be modified in this way.  Maybe I’m misunderstanding, but I don’t think I am.

If I don’t achieve the minimum threshold for highly effective on the student growth side, my understanding is I couldn’t be

highly effective overall.  My comment and proposal would be that that be lowered a level.  So, in other words, moving the threshold, understanding that the legislation does require thresholds, but moving the threshold that the rule embodies to say that to be highly effective overall, I don’t have to be highly effective on the student growth side, but I do have to be at least effective on the student growth side; and, then likewise all the way down.

So, I could be effective overall if I was at minimum needs improvement on the student growth side as opposed to having to be effective on the student growth side. I just think that that would make it more equitable in terms of what we’re portraying the evaluation really being 50% weighted on student growth and 50% weighted on instructional practices as opposed to my understanding of the current rule where the implementation would appear to make it 100% weighted on the student growth side. Thank you.

Coordinator	Our next comment comes from the line of Amy Bolte. Please proceed.

K. Shivley	This is Kevin Shivley in Pasco County again. We’ve got a couple of bullets and I’ll try and run through them quickly.  The first is relative to the student attendance factor that goes into the formula. We’d like to ask for consideration that that attendance be collected and applied on a period by period basis, especially at the secondary level because daily attendance doesn’t always reflect the course attendance at that level.

Relative to the performance...standards, we are asking the district still be given the opportunity to use the percentage of students meeting or exceeded, rather than just the VAM target score. We think that that’s much easier for teachers to understand and will lead to better acceptance of the system if that’s a factor that is allowed to continue.

We also fully support the Department’s plan to create that online database where teachers can access that information on

a student by student basis, so, again, they can understand where their scores are being derived.

From an overarching policy level, we’re asking for the Department’s assistance in lobbying the legislature relative to the all-inclusive process that’s going to need to take place for implementation of common core and consider either a moratorium on the FCAT or a temporary reduction in the percentage of student achievement that goes into the evaluations, at least until we can get the Park assessments implemented and get our students and teachers used to those.

Coordinator	Our next comment comes from the line of Teresa Hinckle.

Please proceed.

T. Hinckle	Piggybacking off of that, I totally agree with the reference to the student attendance at the high school/secondary level.  But, in addition, on Item (9) as far as the class size characteristics go, you’re on the right track there as far as defining the variable as the number of students assigned to the teacher.  But, I think

that needs to be more specific to include the classroom attendance in both the October FTE count and the February FTE count in other words to be held as a counter.

Coordinator	Our next comment comes from the line of Diego DeRosa.

Please proceed.

D. DeRosa	Yes.  I just wanted to add a quick comment. I know that I had asked about this, Kathy, and I know that in the...that we provided this was on there.  But, I wanted the comment in there for the record.

The difference between results that come out from school grade and A++ and evaluations, the fact that they don’t always correlate is huge, at least in our district. Some of our teachers shut down immediately when they say we went from a C to an A, our school. We did all this work.  All this is great, yet we ended up having more needs improvement teachers than any other high school; how does that make any sense?

Immediately they almost shut down and stop believing in this evaluation system at all because we’ve been drilling into their mind learning games and proficiency and graduation rate and all these things, especially at the high school level.  Having one model per school grade which is such an important piece and having a completely different model for teacher evaluation and school administrator evaluation has caused a lot of headaches, at least in Broward County.  I’m guessing in other districts as well.  At some point, the two models have to coincide. One has to be used for both, whichever one it is.

Coordinator	We have no further comments from the phones.

K. Hebda	Anybody in the room need to make comments today? Would you like to make comments...?

T. Demma	This is Tony Demma from Meyer Brooks Law Firm.  It could have been a question, but I’ll say it as a comment.

Based on my understanding of how the VAM scores will work, and particular the confidence levels that are primarily driven by how many students teachers teach, it appears to me that it will be far more difficult for a teacher who teaches smaller groups to ever get a highly effective because the range for that person will be so big the range based on two times the score, the confidence interval will almost necessarily take them below zero.  They’ll have a very hard time being highly effective.  I would like to see something looked at in that regard. Thank you.

K. Hebda	Anne, are there any other questions or any other comments in the queue?

Coordinator	No further questions from the queue.

K. Hebda	This concludes our workshop for this morning. Thank you very much.

Coordinator	Ladies and gentlemen, we thank you for your participation in today’s conference. This concludes the presentation and you may now disconnect. Have a good day.