Page 1

STATE OF FLORIDA DEPARTMENT OF EDUCATION
RULE DEVELOPMENT WORKSHOP

Broward College Central Campus 3501 Southwest Davie Road
Bldg. 22, Room 136 Davie, Florida

Monday, February 25, 2013

4:00 p.m.-6:10 p.m.

1

2	PRESENT:

3

4	Kathy Hebda, Department of Education

5	Juliana Valleado, Broward College

6	Vicky LaForte, Broward Teachers Union

7	Kenny Minchew, Broward Teachers Union

8	Eric Dwyer, Florida International University

9	Frances Costa, Broward Teachers Union

10	Ann Siwiak, Broward Teachers Union

11	Audience Members

12

13

14

15

16

17

18

19

20

21

22

23

24

25

Page 2

1	(Thereupon, the following proceedings were had

Page 3

2	MS. VALLEADO:	Good afternoon, everyone.	I would

3	like to welcome you to Broward College.	I'm Juliana Valleado,

4	and I'm associate dean of bachelor's of arts in education

5	programs at the college.	That's our bachelor's degree in

6	education.

7	We have Kathy Hebda today, who is a chancellor for

8	the state of Florida.

9	And we are here today for a rule development

10	workshop, and, hopefully, we can answer any questions that you

11	have.

12	Welcome to the college, and if you have any questions

13	about the college, or about the program, feel free to see me

14	afterwards.	Kathy?

15	MS. HEBDA:	Thank you very much for being here today,

16	can you all hear me in the back?	What a nice room.	Thank you

17	very much for helping us here, we really appreciate it.

18	I have--we are doing a real workshop today on two

19	separate rules, and we are doing them together, so they kind of

20	work hand in hand.

21	And, I have several pieces of instructions,

22	everybody, so that we get all the input we can today, for the

23	years and weeks ahead, for input, to continue the input online

24	and other places.

25	Today, we will do it face-to-face.	We do have a
1	
Page 4

1	Court Reporter here in the front of the room, who is taking

2	down everything, and you will have a transcript that will post

3	online, the same place that we posted the two webinar

4	recordings from the week before last.

5	We have this meeting today.	There's also a meeting

6	held, at the same time that we are here, there's also one in

7	Jacksonville today, and tomorrow, we'll be in Lee County, and

8	somebody else will be in Panama City.

9	And then we will have another workshop on Thursday in

10	Orlando, and then we are going to compile everything for a

11	little bit.

12	But the online opportunity to provide additional

13	input will stay up.	Even if we are not having face-to-face

14	meetings, we will continue that process.

15	I will walk you through the whole timeline in just a

16	minute, but, since we are doing a transcript, and we are not

17	doing a webcast or a video recording of the meeting, what I'm

18	going to ask is, whether it's the instructional part of the

19	meeting, or whether it's the comments section at the end, where

20	you are providing comments, if you can come close enough so

21	that the Court Reporter can hear what you are saying, we will

22	greatly, greatly appreciate it.	So, I appreciate your

23	cooperation with that.

24	So, because I'm doing a power point here, I'm going

25	to stand way back over here, so that you can see it.
1	
Page 5

1	So, is everybody able to see that.	That's the agenda

2	for today.

3	We have three parts in our rule workshop, and we have

4	a lot to accomplish today.

5	So, we will get everything, we will stay as long as

6	you still have things to say.

7	The first part is overview of the presentation, and

8	that's what I'm about to do.	We do an overview for a couple of

9	reasons.

10	First, to orient you on what the law requires that we

11	make rules on, and secondly, to give you an overview of what

12	the exact rule actually is trying to get--what we intend to do

13	as we go forward, because, one of the things we want feedback

14	on, is whether or not the written words on the page that you

15	have in front of you, right now, actually accomplishes what we

16	intend to do.	That's one of the the things we will get

17	feedback on

18	MR. MINCHEW:	Is it possible to get the power point

19	emailed to us?

20	MS. HEBDA:	Yes, it's actually posted online.

21	there's a web address that I am going to show you, and

22	everything you need is on that website for these rules.

And

In

23	fact, if you put fldoe.org, and the right hand side of that

24	page, just put that in, and it will take you there.

25	So, if we need to go through this, and what we are
1	

1	trying to do is see that you understand the Statute.

Page 6

2	Part two, not everybody understands the rule, or at

3	least not to comment yet.

4	First of all, do you understand that we mean what we

5	say, or what happens here, what happens there.

6	So, there's a part in our rule book that we have just

7	for questions and answers, because we do have a Court Reporter

8	here in the front, if you have a question, if you can speak

9	loudly enough, or to come forward more, so we can be sure that

10	we get those questions, and to have the questions answered

11	today on the record.

12	It may not be that we can answer your questions, but

13	at least we want to capture your question, and if we can't

14	answer it, we will be able to go back to it and give you a

15	response.

16	After that, there's part three, and that's public

17	comments.

18	This is the portion where we have the interaction.

19	We want you to give us your comments.

20	If you are trying to make a comment at that point in

21	time, we don't want you to feel that somebody is going to

22	debate you, your comment can just stay on the record.

23	So, we will try to give you a response.

24	What we will do, when we are on part three, if you

25	are filling out a speaker card, what I would like you to do, if
1	
Page 7

1	you are making a comment, I will take it from you, you walk

2	forward, so the Court Reporter can hear you, and we will take

3	it here.

4	And then we will go around to the back of the room.

5	Any questions about the agenda before we start?

6	Okay.	Let's go.

7	Okay.	Input and participation.

8	And, we are going to the issue, one more time.	When

9	asking a question or making a comment, please state your name

10	and where you are from, your name, if you are from a school

11	district, association, or anything like that.

12	If you are making a comment, like with the webinar,

13	fill out a speaker card, so that we can have your name

14	correctly spelled, so that we don't have it phonetically, for

15	the record.

16	Then, if you have a lot of comments, we will time

17	them, but we don't have a huge number of people here, so we

18	have a good amount of time, and we can listen to anyone that

19	wants to speak.

20	And, if we get done with all the questions, and

21	somebody wants to speak, you can come forward, and make a

22	comment on the record, or fill out the card, and you can make a

23	comment online, so, we will get everything you say today, and

24	then go to the website, you see here, and give us your

25	comment.
1	
Page 8

1	Now, I mentioned earlier, there's a couple of types

2	of input we are looking for.

3	So, basically, what we are hoping you will tell us is

4	things you see in the rule texts, in front of you, one rule or

5	the other or both, that you think needs to be added or deleted

6	or changed, any of those things.

7	We have already received some comments from some

8	people that say, I think you should reword that second

9	paragraph, like this.

10	That's hugely positive.	One of the things we want to

11	know, you may not agree with the contents of the rule, but you

12	may say, I want you to add or say something, but, you also may

13	feel, after you have seen the presentation, that you want us to

14	do something that you think we are going to do, but the

15	paragraph, you think it says that, but it doesn't.

16	So, if you any it should say something different,

17	give us the comment.	At this point in time, this is just draft

18	texts for your consideration.

19	This is not the stated rule for consideration or any

20	of those things yet.	This is some draft texts for

21	consideration, so that we can get feedback.

22	And, here's our timeline.

23	Okay.	Right now, we are having workshops, on the

24	draft text.

25	We are having, you know, like I said, we are having a
1	
Page 9

1	rules text right now, last week with the webinar, we are having

2	face to face with the teachers.

3	And then, in March and April, we are going to review

4	the public comments.

5	One of the things we will do, is monitor legislative

6	sessions and see if anything happens that will affect what is

7	contained in the exact rules text.

8	We will also be compiling your comments in the rules

9	workshop, and additional comments coming online during those

10	months.

11	That's an important thing to happen.

12	And I think, you know, if you look at it, it's not

13	the exact rules now, it's a draft text.	The point is, many

14	times, a lot of people don't give us feedback, unless they have

15	something to react to.	So, if something does change, other

16	than what we have in the rules text, we still get a lot more

17	feedback and interaction from people, if we show something like

18	this, and then, you can say, we hope you adopt those rules.

19	We don't get feedback that way.	So, this is proposed

20	language right now, but, we may have to revise it, but we need

21	to get your feedback now, even though we have to hold off on

22	redoing it.

23	So, in May, we will revise the rule text and publish

24	for consideration by state board.

25	And in June, it's the state board meeting for
1	

1	consideration of the rules for adoption.

Page 10

2	And, so, we want you to understand, that what we are

3	talking about now are rules that will go in effect for the

4	following school year.

5	So, the rules that we will be discussing now, that

6	will be for the following school year.	They will go in effect

7	for the 2013-14 school year.

8	And, if there's anything that changes, it won't

9	affect it for the current school year, it will not be until

10	next year, at the earliest.

11	Any questions about the timeline, before we go into

12	the substantive rules?

13	MR. MINCHEW:	So, right now, and you are saying that

14	you are going to listen to us now, as to what we think should

15	be done.

16	MS. HEBDA:	I don't understand your question.

17	MR. MINCHEW:	I appreciate your doing this, but I

18	wonder right now, if the state is going to do what they want to

19	do anyway.

20	MS. HEBDA:	Well, I don't know if I can tell you

21	that.	The only option we have is to collect the input, if we

22	have no input, there's no opportunity.

23	MR. MINCHEW:	I'm sure you understand my question.

24	MS. HEBDA:	I do.

25	MR. MINCHEW:	Thank you.
1	
Page 11

1	MS. HEBDA:	But, when we get to input, I will let you

2	tell us what you would like to.

3	MR. MINCHEW:	Thank you.

4	MS. HEBDA:	Okay.	Contents of the law, and contents

5	of the rule.	We will take one rule at a time, and the first

6	rule, you should have it in front of you.	The first rule we

7	are going to look at is the rule numbered 6A-5.030.

8	There's district instructional personnel and school

9	administrator evaluation systems.	That's the first rule we are

10	going to look at.

11	As I said, the first thing we want to know and need

12	to know, what does the law say we have to put into the rule.

13	You have to have specific rule making authority, in

14	order to write a rule on something.

15	So, in this case, it's section 1012.34, Florida

16	Statutes.

17	It says to establish uniform procedures for the

18	submission, review, and approval of district evaluation

19	systems.

20	Okay.	And, let me hold off.	We have a couple more

21	people coming in.	Welcome.

22	If you are coming in now, there's a sign in sheet,

23	somewhere, making its way around the room, or it's in the back

24	of the room.

25	If you would, sometime before you leave, sign up on
1	

1	that sheet, I would appreciate it.

Page 12

2	And, there are handouts in the back, and you should

3	take one of each.	There's two separate handouts.

4	AUDIENCE MEMBER:	One of them says 1012.34, the other

5	one says the same thing.

6	MS. HEBDA:	Okay.

7	MS. VALLEADO:	They should look like this, two

8	separate ones.

9	MR. MINCHEW:	The sign in sheet is available.

10	MS. HEBDA:	Yes, it should be in the back.	If you

11	have the sign in sheet with you, would you raise your hand.

12	MS. VALLEADO:	It made it all the way back here.

13	MS. HEBDA:	Okay.

14	Okay.	I think everybody has the handout.

15	So, back to the rule authority, okay.

16	The rule authority, section 1012.34, Florida

17	Statutes.

18	We need to establish a couple of things in this

19	particular rule.	The first thing is, establish uniform

20	procedures for the submission, review, and approval of district

21	evaluation systems.

22	Reporting requirements for the annual evaluation of

23	instructional personnel, and school administrators.

24	And we submit this to the DOE.

25	Second thing is reporting requirements, for reporting
1	
Page 13

1	annual evaluation results, to instructional personnel and

2	school administrators.

3	And the third one is a process for monitoring school

4	district implementation of evaluation systems in accordance

5	with this section.

6	And, we have many school evaluation systems in place,

7	and the department's responsibility is to monitor the

8	evaluation system, and DOE-approved evaluation system.

9	So, we need to remind folks about what we need to do

10	under the rule authority.	When I talk about the department's

11	responsibilities, and we need to have an evaluation system.

12	There are evaluation systems that need to be implemented, and

13	there's specific rules as to what they can do in the evaluation

14	systems.

15	And we will go through some of what those things are.

16	Those are things that the department has to say in

17	the rules, how are we going to make sure that this person meets

18	the district requirements.

19	How are we going to require that they are teaching

20	the courses that are required by law.	So, we have to put a

21	process in the rule, to let the district know, here is what we

22	are going to expect from you, to send to us, so we can make a

23	determination, whether or not your system is in accordance with

24	the law.

25	Then, here's what we are going to do to monitor that
1	
Page 14

1	system, so that you know, in effect, we are monitoring and

2	complying.

3	And, here's some other things that need to be

4	included in the District evaluation system, that the department

5	needs to provide, how we are going to look at these things

6	here.

7	The first thing is, the progress as they act on it

8	and rise to the top.

9	Teacher evaluations are designed to support effective

10	instruction and student learning growth.	Results are used when

11	developing district and school improvement plans, and results

12	are used to identify professional development, and other human

13	resource decisions for instructional personnel and school

14	administrators.

15	So, those are the things that we need to be looking

16	for to make sure the requirements are met within the law.

17	Another thing we need to look for is whether the

18	District meets the four levels of performance.

19	So, we need to decide how we need to go about

20	implementing this system.

21	The law says evaluations have three measures and

22	criteria.	Performance of students, instructional practice, and

23	professional and job responsibility, and for school

24	administrators, it includes leadership.

25	So, we need to make sure whether the District is
1	
Page 15

1	setting forth and using those evaluations of the different

2	systems.

3	And, also, it needs to be based on sound educational

4	principals and contemporary research in effective practices in

5	these three major areas, so, we need to make sure that in fact

6	they are in place.	And we don't just check if they are using

7	it, but to make sure that it meets with legal standards.

8	So, what do we do in instructional practice.

9	It says, for classroom teachers, they have to have

10	the Florida Educator Accomplished Practices.

11	For instructional personnel who do not have classroom

12	responsibilities, specifically, they need to include specific

13	job expectations related to student support for that job.

14	And, they need to know whether it's something

15	specific for their position.

16	And for school leaders, again, there's a lot of

17	things that they are responsible for.	The effectiveness of

18	classroom teachers in the school.

19	They need to make sure that the administrator

20	is making appropriate use of evaluation criteria and

21	procedures.

22	And, he needs to meet recruitment, and retention of

23	effective and highly effective classroom teachers, improvement

24	in the percentage increase of instructional personnel, and we

25	need to evaluate the percentage of those increases in the
1	

1	schools.

2

Page 16

The next criteria to be considered is the evaluation

3	of performance of students.

4	And, this is just an overview of what the performance

5	of students is under the law, and what you would be looking

6	for.	The first part, as you can see, is that performance of

7	students is to be at least fifty percent of a performance

8	evaluation, based upon data and indicators of student learning

9	growth, assessed annually, or measured by statewide assessment,

10	or for subject and grade levels, not measured by statewide

11	assessment,

12	The overall percentage for any individual can be

13	reduced to 40 percent in the absence of three years of data.

14	The percentage of non-classroom instructional

15	personnel that can be split between statewide assessment

16	results and other local measures of student outcomes based on

17	job responsibilities.

18	Also, districts can choose to combine state and local

19	assessments, based on the individual teaching assignments, and

20	can choose performance measures of other learning growth, when

21	they find it to be more appropriate.

22	For example, they need to measure if, in fact, the

23	grade level is not appropriate, they need to make an assessment

24	on that.

25	And, sometimes, they just don't make an appropriate
3	

1	evaluation, and, remember, at least fifty percent of the

Page 17

2	evaluation must be based upon data and indicators.	So, that's

3	something more to be considered.

4	Performance of the students, they have reduced that

5	now, that there are teachers of courses for which the district

6	has not implemented appropriate assessment, and also, they are

7	not doing an appropriate evaluation.

8	For example, if they may have-- if they have not

9	selected an assessment that they want to use, so, they may not

10	be using it appropriately.

11	So, then, for those courses, those assessments are

12	not appropriate, and they need to look into that.	And, it

13	cannot be used for a local assessment at that point in time,

14	and they need to assess the students on a statewide level.

15	For example, if you have a course, where the student

16	is not keeping up, then the district has to use a school

17	improvement plan, and a plan approved by the school principal.

18	And, then there's another option, a district school

19	superintendent may assign instructional personnel to an

20	instructional team, for those individuals who are not coming up

21	to the required growth, for evaluation purposes.

22	So, if you look at this paragraph, there's a lot of

23	things that they must use, or in the following school year,

24	these things may no longer be available.	So, again, this is

25	something the district has to have in their evaluation system,
1	
Page 18

1	this is something that we will be looking for, to make sure

2	that the students are evaluated properly.

3	And these are other components that the law required,

4	under the educator evaluation systems.

5	One is observation instruments, with indicators, that

6	they are actually using the correct practices, or that they no

7	longer meet the standards for the school administrators.	At

8	least two evaluations per year are conducted, for newly hired

9	personnel.

10	They provide for training of evaluators in the

11	system.

12	They are informing personnel about the district

13	evaluation system, and its components, what the procedures

14	are.

15	And this has included multiple data sources in the

16	evaluation system.

17	They are given the data for what the professional

18	development and school improvement needs to be.

19	Again, it's not just to use multiple sources, but

20	what those sources are.

21	And the use of teaching fields needing special

22	procedures, like I mentioned earlier.	Parental input.

23	We need to have the parental input, and the district

24	can tell you how that happens, whether it requires a certain

25	method, or a certain number of times to do it.
1	

1	And, then, teaching fields needing special

2	procedures.

Page 19

3	Sometimes, there are areas that require special

4	procedures, in the evaluations.

5	So, it depends on the circumstances, in different

6	teaching fields.

7	And, they need to be implemented, now how they use

8	them, but the fact they have been used.

9	The district is required to review the evaluation

10	system annually.

11	Then there's the additional options for the district,

12	including peer review, assistance processes, input from

13	additional personnel, who do the evaluations, and to be used by

14	the evaluator, if it's provided, and amendments to the

15	evaluation, and additional professional responsibilities beyond

16	the state board rule.

17	So, if they require additional responsibility, by the

18	state board rule, and for our part, that's the state board of

19	ethics and the professional code of conduct.

20	Those are the state board's responsibility, and they

21	at least have to be established every year.

22	So, that's everything that needs to be used in the

23	educator evaluation systems.

24	So, basically, this tells you what's going to be

25	required by the district, and the elements, I have reviewed
1	
Page 20

1	those elements, and then also, like I mentioned before, and let

2	me go back.

3	Here we are, and what do we do.	What's the

4	monitoring of the systems, how are those resources used, so, if

5	you look at the rules, what you will see, okay, on the second

6	page of the handout, that you have been given.

7	You will see, right in the comments section, and

8	there's some components that might be used a certain way in

9	this rule, for clarification purposes.

10	Okay.	But, for purposes of this rule, that's what it

11	means.

12	And then, if you go over to, when you finish the

13	definitions, the submission process, which is two pages over,

14	near the back, section two.

15	So, how does the district use this evaluation system

16	for review.

17	Do they provide a website, and different kinds of

18	things, the same way they did two years ago, when we first

19	started doing this.

20	The next section, part three, is where you get to all

21	that content, that I just talked about, that the law required

22	be included in the evaluation system.

23	And we try to explain here, what kind of evidence we

24	are going to expect, for us to see and say, yes, the district

25	addressed that.
1	
Page 21

1	No, I don't see that, and I need more information.

2	That's what section three is all about, all that kind of thing,

3	and you can provide that evidence.

4	And, also, I went through that content.	Then,

5	section four, let me see.	It should be in the back of the

6	page.

7	In section four, it's about five pages over, in the

8	middle of the page, and it called the initial review process.

9	So, once you consider it, get all the evidence, then

10	what are we going to do about that.

11	What's our process to review it.

12	So, then, do we have a timeline for reviewing the

13	evaluation systems, so the district will know what our

14	timeframe is, to get back to them.

15	And, then, in section five, we talk about approval.

16	So, what would be the results of our input, and we

17	set up several approval options that's very similar to what's

18	already in the rules.

19	You can be fully approved, you can be conditionally

20	approved, or you can be denied.

21	And, we have provided those definitions, of how you

22	can achieve each one of those things.

23	Then, after that, we have section six, because the

24	law says the district can make modifications to those systems,

25	anytime they want.
1	
Page 22

1	But, if the modifications are substantive, then they

2	come to us for review and approval.	If they are not

3	substantive, then the district just makes some other changes.

4	For instance, school board members change.	If you

5	get a new cover sheet, with new school board members, that's a

6	substantive change.

7	A substantive change means we show something that's a

8	whole new system.	You are not using this contemporary set of

9	rules, you are going to use a new one.

10	We need to define it for the school district in the

11	rules, so they know what the change actually was, and what the

12	substantive change was.

13	Then, as I promised, we have to establish a

14	monitoring system, which is new.	The department has been

15	required to review evaluation systems for over a decade, but we

16	have not been required to monitor it.	So, there's a whole new

17	process.

18	In the monitoring section, we try to monitor in the

19	same way, that we already review a professional development

20	system.

21	Once every five years, through the school district,

22	with the team, and we know it's been changed in the process,

23	and we do onsite reviews.

24	So, we set up this monitoring process in the same

25	way.
1	
Page 23

1	And there are annual requirements that are included

2	in the rules, because it's required in the law.

3	But, the report, it has to be made December 1st,

4	based on those district reports, and we only come on-site to

5	monitor them every five years.	That's the way the monitoring

6	process is set up, so, then the last section is the reporting

7	section, and it's evident that it has everything to do with

8	review and approval.

9	And then another one that we included in section 9,

10	which is the last page of the rule, the next to the last page,

11	are the guidelines for the evaluation.	Th only thing the law

12	says, the submitting person has to have the set of rules.

13	And they have to consider the guidelines placed by

14	the department.

15	They have to use the guidelines and rules, so the

16	district knows where they are, and that everthing is put in

17	place, and you can use everything you need, it's right in the

18	rules.	So, that's that section.

19	Now, let's go to the other rule.

20	I am going to keep moving, so that we get through

21	this part, and we have more time for questions and comments.

22	So, let me keep going.

23	Okay.	Go to the second rule, that's 6A-5.411.

24	In making the first rule, we are still looking at

25	whether or not the district has included performance of student
1	
Page 24

1	criteria in the law in this evaluation.	So, one of the things

2	the department need to do is provide a few things in its own

3	rules to supplement that rule.

4	And, that's going to performance of students.

5	So, the rule, it's a separate rule, it's a companion

6	rule.

7	It's the rule authority and contents for 6A-5.411,

8	and the first thing is that each formula for measuring student

9	growth has to be approved by the commissioner.

10	The second point is, and you can see there, it's the

11	measurement of student learning growth and associated

12	implementation procedures.

13	If, for example, if the department provided those

14	estimates, and what would be contained in there, for the

15	district to include these things in their evaluation systems,

16	so those kinds of things are included in this rule.

17	And then, again, a process by which teachers may

18	review their class rosters.

19	It says there's a process by which the teachers may

20	review it, because the district has its own process, so each

21	teacher has to have rosters, and make sure that they write down

22	things that they are using to calculate it.

23	But the department needs to provide an avenue for the

24	teachers to do that, to make sure that we have the

25	verification, too.
1	
Page 25

1	Then, the second part of this section of the rule, on

2	this rule, included more things.

3	And, we are going to put it all in, because these are

4	the things that I need a lot of feedback from you on.

5	We have to establish performance level standards.

6	So, they have the results of performance levels.

7	If you have negative results, and positive results,

8	so, what do you need to do there.

9	And, how do I use those numbers in the evaluation

10	system.	Well, in the current year, 20l3, this is the way we

11	are using it right now.

12	In this place, the law says that the state has to

13	make determinations, the state says, you get a rule together

14	for performance standards to be used statewide.

15	And it's for the purpose of establishing specific,

16	concrete standards for each performance level to insure clear

17	and sufficient differentiation.	So, this is what I need your

18	feedback on, and you need to let us know, if it's the wrong

19	ones, the right ones, and if it needs to be supplemented, all

20	that information, we need.

21	What are the next two rules, because they are very

22	important as well.

23	One is to establish performance level standards, for

24	student learning growth.

25	So, if you get the performance of the students
1	
Page 26

1	themselves, and then as to all the other evaluations following

2	that.

3	And we have to establish learning growth standards,

4	that if a student's performance is in the lowest threshold,

5	then the entire evaluation must be declared unsatisfactory,

6	right.

7	So, we have two kinds of performance standards, and

8	one is for the students, and that's what we are talking about

9	just now.

10	And the first reason, if everybody uses the same

11	standard, and the performance issue is part of the evaluation,

12	but, the second issue is to have standards that are necessary

13	to establish, and if certain conditions are met, that the

14	entire evaluation will be thrown out, if the threshold is not

15	met.

16	Or, conversely, if you find someone to be

17	unsatisfactory, or highly unsatisfactory, then, what do you

18	need to use on the performance of student criteria, if the

19	performance is to be affected by the lower study.

20	AUDIENCE MEMBER:	So, are you saying that this is

21	explained here in the rule?

22	MS. HEBDA:	Yes, it is, and I will point that out to

23	you.

24	AUDIENCE MEMBER:	I didn't get the question.	If the

25	criteria is in there--
1	

1

2	that.

3

4

5

Page 27

MS. HEBDA:	Yes, it is, and I will walk you through

Okay.

AUDIENCE MEMBER:	So, it's in there.

MS. HEBDA:	This is what the law says we have to put

6	within the rules.	The law doesn't put the performance standard

7	in the rule.	The department has to establish what the growth

8	standards are.	And that's what the proposed rule is, for

9	consideration in establishing the rule next year.

10	That's why it's very important, that we get some

11	feedback on this.

12	So, let's look at what else we have here.

13	First thing to remember. growth formulas.

14	Remember, we talked about teachers that teach some

15	grades, and then we talk about teachers who teach other

16	grades.

17	So, this is the state law, these are the ones that we

18	will all use the same way, across the board.

19	And, the department has to determine what those rules

20	are, and we only have the one that we established on June 1, of

21	2011, and then we have growth formulas to be determined by

22	local school districts.

23	But, the department is responsible, for growth

24	formulas, not just the ones that we have to use, but to provide

25	examples, and also to look within the rules, what it means to
6	
Page 28

1	have equally important formulas, because we are trying to do

2	things that apply to everybody in the district, so, what does

3	the district have to do, to make all the different growth

4	standards, for different assessments, equally appropriate.

5	Section one, FCAT.

6	So, in the rules, rule, we have to keep in mind, that

7	we have to adopt the measures of student learning growth, such

8	as FCAT.	So, when we started working with this rule, this is

9	what we first came up with.

10	And, this is a value added model, and this is a high

11	level remainder of what the content is, to begin with the

12	rules.

13	So, and this is the value added model developed by

14	Florida educators.

15	And, we have to keep in mind, the student growth

16	implementation committee, all those things.

17	And, we have to remember, on June of 2011, the SGIC

18	developed the FCAT model, exactly as it was proposed.

19	So, then, what did they do.	They explored eight

20	different types of value added models.	That means, you have

21	different factors to include in the models, to get down to what

22	the teacher's contribution has to be.

23	We look at eight different models to do that, and

24	what the factors should be, to be included in the models.

25	And, remember, we kept going back to, you are not
1	

1	developing it for the entire school day.

Page 29

2	But you are developing this model for individual

3	teacher's evaluations.	So, it's important to them, that what

4	they do in teaching, will be important to the individual

5	teacher.

6	So, that's very important, and you can see, that we

7	need these models, for every type of student, and educators and

8	administrators.

9	It's a lot of hours that went into this.

10	So, they talk about teachers, school board members,

11	and parent representatives, and representatives from the

12	community.

13	And, they go through a lot of steps, to see which is

14	the best way to do it.

15	So, everybody should be mindful of the growth

16	factors, in different states.	But, they have to go with the

17	one that works with FCAT, not somebody else's model, so, then,

18	you need to look at the different models.

19	And then, in this example, okay, and, we are going to

20	give you an example with teacher X.

21	Okay.	What you are looking at is the one student,

22	not the one teacher, but the one student.

23	And we are talking about students, all assigned to a

24	teacher.

25	But, what it does, it looks at prior performance, and
1	

1	this is FCAT.

Page 30

2	And it looks at how the student actually does this

3	year, so, last year in school, this year in school.

4	And we are talking about developmental, not level

5	one, two, three, four, five.

6	So, if you are going one through five, use this

7	model.	It's already there.

8	And, then, based on the things we know about the

9	students, including their performance, and the other

10	characteristics that I will go through, and then, that's for

11	the teachers to understand, how should the student do, based on

12	what we know from that student, and from other students, with

13	the same characteristics, how they perform around the state.

14	Can do it by subject, does it mean that all seventh

15	grade students around the state, starting from here, if they

16	have every single characteristic we will go through, and that

17	determines where we expect that student to grow.	How much do

18	they grow, based on the things we know about them.

19	And then if you look at their performance, to see how

20	they actually did, the difference is then attributable to the

21	teacher, because, everything else that we know about the

22	student, has gone in here, so, then all of these differences,

23	from all of the students, and going back to the teacher, for

24	the '11-'12 data, not this year, but, it's all going back to

25	the teacher, not just for that school, but for that teacher.
1	

1

2	that.

3

Page 31

Now, let me show you some factors having to do with

Okay.	The student characteristics.	So, what we see,

4	we look at the rules, and you supplement what we have here,

5	because we have to describe this first model in the rule book.

6	So, then, you look at student characteristics that

7	should be taken into account, classroom characteristics to be

8	taken into account, and school characteristics to be taken into

9	account.

10	And, it explains, up to two prior years of

11	achievement scores.

12	So, if the student has a prior resource FCAT, but

13	they had another one the previous year, we will use that,

14	okay.

15	We will consult with that. and then, the number of

16	subject relevant courses, that's going to FCAT, how they are

17	doing on FCAT, we will go back to that.

18	And it states where they are going to go.	And, these

19	other things, we know that we consider them a little bit, in

20	the technical report, and it will show you exactly how much, by

21	subject, by grade.

22	So, even though, for example, and let's say,

23	different from model age in grades.	That means, they are the

24	same grade, but they have other abilities, and you can look at

25	the data, and see if that's a good model, and if you can see
4	

1	how they score next year.

Page 32

2	So, that data may not be statistically significant,

3	the statewide data, but, from there, you will see how a lot of

4	students have done on it, or if a lot of students are in that

5	group, it might make a difference to you.

6	And remember, we are doing it for an individual

7	teacher's evaluation, so, you may look at the back of that

8	technical report, and you may see some of these things on here,

9	don't seem to matter, they don't seem to make much difference

10	between one student and another, but the teacher needs to know,

11	so, the committee decides to include it.

12	Those are the different characteristics, to do with

13	disability as well, and that's coming into another question.

14	The committee worked with the department of education

15	office, and we got into the list of disabilities, and they

16	included all of them independently.

17	If you look at the back of the technical report, and

18	we see the difference, there's a difference, like having a

19	learning disability, might be quite different, than a student

20	who has some other disability.

21	AUDIENCE MEMBER:	As opposed to a physical

22	disability.

23	MS. HEBDA:	Right, absolutely.	So, they added all of

24	these in there, but not everyone is the same.

25	AUDIENCE MEMBER:	So, that teacher is getting a
1	

1	report.

2

Page 33

MS. HEBDA:	Well, a report.	But you are looking at

3	how the student is supposed to do, and then look at the

4	teacher.

5	And then looking at that model.

6	Then, based on how other students do, then you see

7	that same characteristic.

8	So, then, you just adjust it up or down, and with

9	everybody else from the state, you use that same characteristic

10	here, yes, Ma'am.

11	MS. COSTA:	And you are talking about, like,

12	differences with the student's score, on the FCAT, from year to

13	year.

14	MS. HEBDA:	There's only one student score.	The

15	score for each student, it's the differences between all of her

16	students, and then it comes back to the teacher.

17	MS. COSTA:	And, then, you use it from last year.

18	MS. HEBDA:	That's right.	And we use the last source

19	of data that we have, but if I have a source of data for the

20	teacher, not the student, but for the teacher, then the teacher

21	can reduce the amount of fails they get in that evaluation.

22	MS. COSTA:	And you are talking about the percentage

23	on that.	And, could that be a student that's above or below

24	that?

25	You are saying, if it's lower than the average, and
3	
Page 34

1	then, that means, okay, that maybe not all of my students score

2	below the average, and some students are better than that.

3	MS. HEBDA:	Right, many of you probably would use

4	your own running data, after you look at FCAT.

5	But, then you have levels one, two, three, four, and

6	five.

7	But, again, levels one through five, don't count for

8	this, so, this is guided by these levels over here, as opposed

9	to one, two, three, four, and five, so, when the state is

10	getting the grades, last year, from the FCAT, that were quite

11	low, how does that affect my value added score, it doesn't.

12	Now, how do the students come up.

13	A lot of students came up to here.	So, if they are

14	up to here, then every other student starts here, and then

15	whatever they can do, to improve, that's where you make up for

16	the deficit, and then, that difference, the difference between

17	all those students, and the teachers, are combined for the

18	school.	Yes, ma'am.

19	AUDIENCE MEMBER:	Then, you are saying, how they can

20	come up, the scores.	But, then, if some kids are getting help

21	at home, doing their homework, some students do not have help

22	at home.

23	MS. HEBDA:	That's a great question, and the

24	committee talks about a lot of things, that are not absolute,

25	that we don't collect data on, and we don't know in the
1	

1	database, if the students get help at home.

Page 35

2	So, then, what happens, and with the FCAT model, it

3	always go by the growth pattern, and even the ones we put

4	together in the future, we may have a harder time with it.

5	But, in the FCAT model, okay, it's going from one

6	through five, all the way through.

7	But, that prior score is the biggest predictor of how

8	you expect it to be next year.

9	In fact, we put out-- after we went through all the

10	information, we ran an impact data, to see how all this turned

11	out.

12	And then, we ran it to see whether someone had a one

13	hundred percent of students with disabilities, what if they

14	have fifty percent, what if they have fifty percent English

15	language learners.

16	And, we look at it, more or less, on these kinds of

17	students, as to how it affects the scores, if a higher number

18	of teachers have higher scores, or if a higher number of

19	teachers have lower scores, and we didn't find that.

20	And some of the committee members hypothesized that

21	it wouldn't turn out that way.

22	Basically, the committee, we actually don't just look

23	at the composite scores of studentss with disabilities.

24	But there are things like attendance and mobility,

25	those are some of the things that also tend to show up.
1	
Page 36

1	So, and then there are students who are struggling

2	otherwise, and may not be getting help at home, based on the

3	other things that we do have, that might otherwise account for

4	that.

5	So, again, that's not always the case.	It's not

6	always the case, but in our case, it impacted the data very,

7	very positively, based on many, many things, and based on

8	things that we don't account for explicitly, in predicting

9	scores, such as race, and necessity.

10	So, anyone of these things are accounted for.

11	But, because we had a really strong predictor score,

12	and everything was included, including class size and

13	everything else, it worked out, as a predictor.	So, you can't

14	tell the district on a number of highs and lows, with teachers

15	that have zero percent students in poverty, to one hundred

16	percent students in poverty, along with all the other things we

17	talked about.

18	So, there's other things that we need to consider

19	from school to school.

20	AUDIENCE MEMBER:	And when you are using the FCAT,

21	like the scores in math, and then, are you comparing that to

22	this?

23	MS. HEBDA:	That's a great question.	Because I'm

24	talking about the FCAT model right now, that's the one we have

25	adopted.
1	
Page 37

1	AUDIENCE MEMBER:	Right, because we don't have that

2	information to always be included.

3	But, then, am I grading on algebra, in order for me

4	to know where my scores would be coming from, or where it is

5	not coming from.

6	MS. HEBDA:	Okay.	They have not recommended-- SGIC

7	has not recommended this specifically for FCAT.

8	But, they gave us information on algebra one source

9	model, and biology source model.

10	AUDIENCE MEMBER:	So, then, how are we getting the

11	base scores.	How does it work out?

12	MS. HEBDA:	That's a great question.	Unless the

13	state has a model that applies to everything you do, like this

14	one, then the district has to decide how they use that data in

15	the evaluation system.	So, just like the difference between

16	Palm Beach County and Broward County.

17	It's a statewide adopted growth model, that means

18	everybody has to use it the same way in the district.

19	MS. COSTA:	So, then, you are saying when you

20	implement it, that's the whole state.

21	MS. HEBDA:	For the FCAT model, yes.

22	MS. COSTA:	We are not doing FCAT, it's over, after

23	2014.	There's no more FCAT.

24	MS. HEBDA:	FCAT 2.0, that you are using right now,

25	with algebra.
1	
Page 38

1	MS. COSTA:	After 2014, we will not be giving the

2	FCAT anymore.

3	MS. HEBDA:	Right, so, it's not gone yet.

4	MS. COSTA:	But, why are they going to implement

5	something statewide, if they don't even have the data to

6	implement it with.

7	MS. HEBDA:	This is beginning next year, which is

8	2014, so, when it comes to that, something has to happen with

9	the rules, for 2014, and many others, when it comes into play,

10	to do some things for the evaluation processes.

11	So, this is what we have right now.

12	MS. COSTA:	We collect data also on RGI, I don't know

13	if it's reported, and BMP's, and that doesn't seem to be

14	included.

15	Because, that would affect the performance.

16	MS. HEBDA:	Right.	The question was, the data that's

17	not reported that's collected in RGI, for example, and other

18	things-- the committee didn't discuss RGI data, and, I think

19	they just captured all the data there, and we had somebody do

20	that.

21	But, there's a website, if that's what you want, that

22	has the scores, for the FCAT students, and what some of those

23	sticking points were.

24	And they did bring up RGI, because they were

25	concerned people weren't scoring it the same way, and this was
1	

1	something that was statewide.

Page 39

2	MS. COSTA:	Why don't they establish the consistent

3	way, wouldn't that be more effective, and more, you know, more

4	useful on the proper evaluation of each student?

5	Because, that does affect a child's performance, you

6	know.

7	MS. HEBDA:	So, when we get to the comments section,

8	okay, that's something that we would-- that would qualify as

9	the kind of comment that we would like.

10	MR. MINCHEW:	And I'm concerned, that when a child is

11	sitting in the classroom, but they are not necessarily

12	capturing it, or not there.

13	MS. HEBDA:	And, that's included, and we discussed

14	certain circumstances in another room, they had to have some

15	special circumstances, and things like that.

16	There's some data points that we don't know, not just

17	RGI, we collect data, in every single way.

18	And some students are transitions, in and out of

19	schools, or even if it's in the same school.

20	We collect that in the state.	What we don't have is

21	course by course attendance.	Only the district has course by

22	course attendance.

23	But, if you are talking about something like that,

24	where the student was actually present for the day, but they

25	weren't in anyone's class, then, that would be only something
1	

1	that only the district could have, in using it for the

Page 40

2	evaluation system, not in the gross model necessarily, but in

3	the evaluation system, to comment on that data.

4	MS. COSTA:	And how do they do that?

5	MS. HEBDA:	It's up to the district.	You know,

6	there's some places where they set up some facilities for

7	that.

8	MS. COSTA:	Online, there's a lot of variables.

9	MR. MINCHEW:	There's a lot of unknowns.

10	MS. HEBDA:	Okay.	Let me finish this off, and then

11	we will get to your questions.

12	Okay.	And, one of the things you have to put in your

13	rules is the implementation procedures, so you list the kind of

14	data you get from the value added results.

15	And we talked about the scores.	And, like they were

16	saying, that we also can tell that some schools have had

17	individuals that excelled.	We can tell which students exceeded

18	the predictions for that school, so, we do it by district.

19	Okay.	And, it was other things that we get.	In a

20	value added result, zero indicates typically that the student

21	performs no percent or worse than expected in the state.

22	Remember, what you are looking at is numbers above

23	zero, positive numbers, and negative numbers.	So, positive

24	numbers mean, that for each teacher, her students overall as a

25	group outperformed the expectations.	Maybe some didn't, and
1	
Page 41

1	some did.	But when you put them all together in that school,

2	the school can show that they did.

3	And the overall scores, when you put them all

4	together, if it's below zero, on average, they performed less

5	than what you expect them to do as a group.

6	And that's important, for you to look at the

7	standard, based on that relationship.

8	And then, percentage of expectations.	Performance

9	expectations, one of the things you consider, you know,

10	remember, the green bar, and the red bar, you can see how the

11	difference is, between the typical school, and that school.

12	Okay.	When it comes to the expectations, it's either

13	a yes or no.	Either they exceed it, or they don't exceed.	And

14	that's got to be from school to school.

15	So, one other thing to consider is the standard

16	error, and, statistically, the students that are assigned to a

17	teacher, they all should have the standard error.

18	And, basically, it's like voters, for example.

19	Instead of voting -- they will vote for one thing,

20	and they will turn back and say, most of the voters, okay,

21	overall, they voted at a fifty percent rate, plus or minus,

22	three points.	That plus or minus three points doesn't mean

23	that they don't typically vote that way; they did.

24	What it means is, if I had sampled another set of

25	voters, with a set of characteristics, then I'm sure that the
1	
Page 42

1	results would have been somewhere between 47 and 52, that's

2	plus or minus three points.

3	If I change the samples, I change the students

4	assigned to a teacher, that have those characteristics, that's

5	how sure I would be that that score is right.

6	So, one of the things that we provide to the district

7	are standard errors.

8	And that standard error is to effectively decide, how

9	close they are to that number.

10	For example, in a presidential sample, the plus or

11	minus three points.

12	And, I can be confident, given another set of

13	students with similar characteristics, that it would have been

14	within this range of scores.	That's what that means.

15	And, it's the same as any other statistical sample,

16	having to do with the population or anything else.	Basically,

17	what we are trying to do, we wanted to use this model, because

18	it has so many other things, that go into whether or not a

19	student is learning or not.

20	And it's a statistical model, if you are going to use

21	a statistical model, you need to use it correctly.

22	So, you are using it for an individual student or

23	teacher's evaluation.

24	So, if you are confident, that the information you

25	have is correct statistically, you use it.
1	

1	So, how does that work.

Page 43

2

3	before.

4

And, keep in mind some of the things that we said

We talked about the fact, that if everything goes on

5	schedule, and then we go through all the meetings, the rule

6	would still not be effective until the following school year,

7	so, 20l3-14 would be the first year that what I'm about to

8	propose to you, for your consideration, will be used, and it

9	will be used for the performances and the student criteria.

10	Performance of student criteria.

11	So, we would all use-- we are using all this data,

12	all this performance classification, for this performance

13	evaluation, and then, it will be used for whatever the school

14	would need it to set up the rules.

15	So, by using the performance of students, we are

16	trying to set up standards that will be used consistently

17	throughout the state.	If that's the data, you use the data the

18	same way around the state.

19	Now, the other thing we are trying to do, which is a

20	little trickier, but we think it's important, we want to make

21	sure, if we can-- we want these standards that we set to be a

22	criterion reference standard, meaning, every year, we don't

23	want to have a rank of who is the worst, and who was the best.

24	Now, if you get a value added result, you get the

25	district-- the highest score and the lowest score, you get that
5	

1	every single time.

Page 44

2	But, the actual score itself, the score is for an

3	average of the user's work, and we know what the average data

4	is, every single year, with this assessment.

5	So, what we have proposed, for your consideration, is

6	that we begin with the data, the growth, the average growth by

7	grade level every year, and that becomes the standard.	So,

8	each year, everybody can start moving above that standard each

9	year, instead of every year, having the same distribution, and

10	being the top two percent or bottom two percent, or whatever

11	that is, and working from there.	It's harder to do.	It's much

12	easier, just to get the distribution and say, here it is.

13	But, we need collaboration, so everybody is working

14	together, so everybody can get better, that's what you want,

15	but we are trying, and I don't know if this is a better way to

16	do it, when you get the feedback on it.

17	And, we need to establish whether it's going to be

18	the best way to do it, by working together, and not just

19	getting the percentage of growth and everything else, but

20	everybody needs to use it.	So that's the goal on that.

21	Now, let's go to the classification options.

22	Okay.	Here's the classification options.	We talked

23	about the scores, and we talked about a confidential interval.

24	Okay.	And, there's the number here, it goes to here,

25	and then, using one standard of error, okay, and two standard
1	
Page 45

1	errors for measurement, and you use two different confidential

2	intervals, okay, first, a confidential interval of plus or

3	minus 68 percent.	I have 68 percent confidence, with a similar

4	set of students, that this person's score in the school would

5	have been somewhere in this area.

6	Now, remember, we are using one confidential

7	interval, based on one set of standards.

8	Or, if I have two standard errors, plus or minus two,

9	given the same group of students, that the score would have

10	been between here and here, for sure.

11	You can almost think of this, you know, using that

12	example, of plus or minus three.

13	Okay.	You can look at it, plus or minus two, and

14	plus or minus five, okay.

15	So, then, I'm confident that the bigger this is going

16	to be--because, I have to be really, really sure, that it's

17	going to be between here and here.

18	So, those are the things to think about, going to the

19	confidence of what the score is going to be.

20	Okay.	And, roll it up from 68 percent, from a

21	confidence sample of 68 percent confident, and one of 95

22	percent confident.	That's really, really good, okay.

23	Now, what do the percentages look like, if we use the

24	statistics that we should use.

25	What might it look like.	Okay.	And this is for your
1	

1	consideration, think about these.

2	Go home, talk about them.

Page 46

3	And, this is the running data, and these had been in

4	place before.	How does it work.

5	And this line up here, okay, this is from zero.

6	Okay.	And, that's where we start from, and this is

7	typical, right here.	Okay.	And the red bar is the 68 percent

8	confident, and the yellow bar is 95 percent confident.

9	So, then, how do you get to the top.

10	Okay.	Let's talk about the standard for highly

11	effective.

12	Okay.	And some of these are--and, the graphs are

13	really different, so you can see the difference.	In this case,

14	okay, by this standard, it means that it's highly effective,

15	and that's at 95 percent confident.	Meaning, with two standard

16	errors, that our average, all this group of students, and, if

17	you go by the item scored, will be somewhere between here and

18	here, and down to here, and the most important thing to think

19	about is that we have 95 percent confidence, we are entirely 95

20	percent confident, that, where you are looking-- if you look at

21	where the score could be, even if it's at the bottom of where

22	that score should be, it's still at typical performance, that's

23	what it means, typical performance.	So, here is somebody, and

24	each student has to go through this.

25	Okay.	But, then, I will hazard everybody's score
1	
Page 47

1	would be so good, that it would be entirely, even with 95

2	percent confidence, above typical growth, with that standard.

3	right.

4	Now, what about these two right here, actually, we

5	can probably do the opposite, standards for unsatisfactory.

6	I'm 95 percent confident, that when you put all these

7	statistical scores together, all of these, that the entire

8	possibilities, the range of scores, is completely below typical

9	growth.

10	Okay.	It's completely below.

11	So, we have the two extremes.	Okay.	And, right now,

12	this is just the performance that I have calculated.

13	And, you need to combine that with whatever the

14	district--the other part of the process, and then that's

15	combined, in the system, to determine whatever the final result

16	is.

17	And, there's one way to calculate it, with 68 percent

18	confidence.

19	Okay.	So, I have two sets here.	And, this is the

20	standard for being effective.

21	And, the law requires the standard to be for

22	effective, highly effective, and unsatisfactory.

23	So, I want you to think of the two ways that we've

24	defined these effectiveness standards.	The first one is,

25	whatever the baseline score is.	Whatever the result is for
1	
Page 48

1	that year, don't worry about how confident we are, but it could

2	have been something else, with the kid, but what was it for

3	that year.

4	Okay.	So, look at this one here, so, her class that

5	year is high performing, typical performance for the state.

6	Okay.	Now, the difference between this one and this

7	one, is that, these are not, these are 95 percent confident,

8	but it always is going to be higher, for the student.

9	And, then, there's a chance it might have been below

10	this, even if I apply 95 percent confidence level for that

11	school.

12	So, likely, for this one part of the evaluation, it's

13	going to be between highly effective and effective.

14	And the other definition for effective is, the score

15	is actually below zero.

16	So, on average, for that one year, the student did

17	perform less than effective.

18	But, then the other confidence level, right here, it

19	turns out with that 68 percent confident, it could have been

20	higher than that, if it had been the student with similar

21	characteristics.

22	So, to account for the fact, that the confidence

23	level would rise, you want to give somebody the higher score,

24	because, when we use statistics, and we should, in a

25	statistical model, we want to be sure we make the right
1	
Page 49

1	judgment, and the best judgment we use, is that the ratio.

2	And the reason that you have multiple models, you

3	have a lot of people making judgments, and with different

4	professional responsibilities, so you are going to have

5	different models.

6	But, with this criteria, then you employ the

7	statistics, and it could get the ratio at what it could be, and

8	then to get it to what it should have been.	Yes, Ma'am.

9	AUDIENCE MEMBER:	I have a question.	One of the

10	concerns that we have, is that model for teachers who teach

11	11th grade.

12	And the concern that I have, the model doesn't take

13	into account, giving the teachers enough credit.

14	And you have the same factors for students.	But what

15	do you have for a teacher that has been teaching for ten years,

16	who is using the same subject?

17	You are evaluating them with a teacher that has

18	taught that subject for 15 years.

19	MS. HEBDA:	That's a great point, you want me to

20	address the second one first, would that be all right?

21	In this case, these are the things that, again, based

22	on the the guidelines, is that everybody would use these

23	standards once they are adopted, or whatever it turns out, it's

24	quite like this.

25	You still have, the district still has that
1	
Page 50

1	discretion on these in special circumstances, and special

2	teaching fields, to do something like that, if that's what they

3	chose to do.	They sure can.

4	A lot of districts don't want to do that, but they

5	could.	But it's something that we could make allotment for.

6	But, I don't know that we would want to at the state,

7	and say, here's what we are looking for to be in the first

8	year, second year, third year, fourth year, fifth, all that

9	yet.	We can set the standard, and the district could use it,

10	and account for special circumstances, largely, using it the

11	same way, but special circumstances to be accounted for.

12	And your first question was about the variance in the

13	test?	That's the hard part about the criterion reference, and

14	we discussed that in the meeting, just a few minutes ago, even

15	with FCAT, okay, I mean, it's really nice, we have talked about

16	it some tonight, we haven't given it enough analysis, to know

17	how much variability there is in the test form, all those

18	things that happen, to make it worse.

19	I don't mean to say, we don't do all those things,

20	and we do everything we are supposed to, by all the FCAT

21	standards.

22	But, one way to do it, is to get the distribution for

23	that year, and that's the one good thing to it.

24	So, I think, that there's a conclusion around that,

25	because we don't just want to do a distribution on it.
1	
Page 51

1	And, you don't know what you are shooting for, you

2	know.	And, we need to know what you are shooting for.

MS. COSTA:	And, can you get back to the 60 percent	

on your evaluation.	

MS. HEBDA:	
I don't know what 60 percent, is.	

MS. COSTA:	
Well, I mean, you can be as low as six	

3

4

5

6

7	percent, in the scores.	Only six percent variability or

8	something.	I'm looking at the standard for effectiveness,

9	okay.	And the second one that's below that.

10	MS. HEBDA:	Right, there's no confidence level there,

11	it goes above, typically.

12	So, we are not positive that this result is going to

13	be above that percentag.	Is that what that means?	Are you in

14	Broward County?

15	MS. COSTA:	Yes.

16	MS. HEBDA:	If you have anymore questions.	Okay.

17	And she asked about a six percent.

18	AUDIENCE MEMBER:	And that's going to student

19	expectations.	And, what we are looking at, is calculating a

20	percentage with an expectation for all teachers at that grade

21	level, and then, to bring it up to a 99 percentile, and base

22	the scores on that.

23	And the sixth percentile corresponded to the success

24	rate.	So, then, 35 percent of the students meeting

25	expectations, there's about six percentile that are lower than
7	
Page 52

1	that.	So, basically, you have to look at it as a whole, if you

2	are going to use those stantards.

3	MS. HEBDA:	Okay.	Now, I just wanted to look at one

4	thing, before we move forward.

5	There's something I want to point out, before we move

6	to something else.

7	Okay.	And, right now, there's a lot of districts

8	that don't use this quota.

9	They use the number of students, the percentage of

10	students that are above at that school.

11	But, that's another way of doing it, but this is the

12	requirement, but it's not using it for itself.

13	And they are using it for different reasons, and they

14	could.

15	They have a right to use it for whatever they want.

16	So, if this were to go for another round of changes,

17	we wouldn't have to stop using it at schools.	But it would be

18	used as a baseline.

19	So, I think what we are talking about, if you are

20	going back and seeing if you want to make a different comment

21	on it at some point, and what suggestions you have, we have a

22	couple of different suggestions.

23	And it may appeal to you, it may give you an idea of

24	something else.

25	Right now, we are only using this typical
1	
Page 53

1	performance, and then looking at all the different checkpoints,

2	around typical performance, and this has to do with confidence,

3	and other stuff.

4	One the things you might say, that's one way of

5	looking at it.

6	And then you might use, for example, highly

7	effective, and even though your whole confidence level doesn't

8	rise above zero, your score itself might have exceeded another

9	bar.

10	So, essentially, we know the average year growth, at

11	this year, and we know that, for example, mathematics teachers,

12	on the whole, and we put them all together, in grades four

13	through eight, on average, are teachers that were rated highly

14	effective in mathematics each year, overall, actually had value

15	added scores, on average, where their students scored 20

16	percent more than typical.

17	So, instead of a usual growth, their students scored

18	20 percent above that.	That was on average.	Some schools did

19	better, some did not.	So, that's an example.	So, what you

20	might want to do, or someone suggested that we do, is have this

21	as one set of definitions, but also give a teacher, who doesn't

22	have a growing number of students, okay, and then they remain

23	stagnant, that's why they don't use the data, and it is more

24	data point based, and they have confidence in the predictions--

25	if someone had a really, really high score like this, but their
1	
Page 54

1	number of students is low, then their confidence range is

2	fairly large, and it goes well above zero.

3	So, we know the whole thing is above zero, but you

4	also want to give somebody credit, if they have a high score

5	like this, even though, I'm not 95 percent confident, it might

6	have been that same school, that scored high.

7	You see what I mean, and then, we have another check

8	point, where somebody might have been highly effective, for

9	example.

10	That's one thought for consideration.	And another

11	thought is the expectations.

12	So, if you are not confident about the score itself,

13	because you have a second standard that comes in, that looks at

14	how many kids are assigned to the teacher, actually are really

15	attending the school-- if it's a higher number, would that be

16	moving another way, highly effective, or more effective.	So,

17	what you want to do is go back to what you want people to have

18	to achieve in the evaluation system.	What is it that you want

19	to point out, and what do you want people to do, if they are

20	not quite there yet, so, we are not opposed to multiple

21	definitions for these things, at all.

22	So, there could be other ways to do this.

23	So, this is just one way to do it for your

24	consideration.

25	Okay.	What I should do now is talk about the other
1	

1	part of the rule, and this has to do with performance

2	standard's impact on summative ratings.

Page 55

3	And then, we will apply that second and third area

4	that I talked about, for the whole evaluation.

5	So, then, we have them there for those two

6	performances, to assess the performance of students, they also

7	have a standard that, if not met, would result in an

8	unsatisfactory overall evaluation.

9	So, what we tried to do in the rule was use the

10	condition already in the law, to see, when would that occur.

11	So, even though, if everything goes according to

12	schedule, next year, we would actually use the standards,

13	whatever they end up being, for performance of the student

14	population, and the rest of the population does whatever it

15	does.

16	Only under those certain conditions would those

17	standards be found effective overall.

18	And, we will use it under these criteria.

19	So, for those standards, if the entire evaluation is

20	satisfactory, the teachers would have to be evaluated on what

21	they are teaching, the area they are teaching in is evaluated

22	based on the measures in 1012.34, and they would have to have

23	three years of student performance data that must be available

24	for the teacher or administrator.	That's what we would need,

25	for that teacher or administrator, to go through the whole
1	

1	evaluation.

Page 56

2	And, these are used in the recent evaluations, that

3	we were talking about, and we know these are only temporary

4	standards, or temporary data, we don't have the end data.

5	So, basically, you need to add the performance of

6	students in here.

7	And we also proposed that the non-classroom

8	instructors, that don't teach the students directly, be

9	included in the standards, so, they should not be applied in

10	those evaluations, if they are non-classroom.

11	MS. COSTA:	What about primary teachers, pre-school

12	teachers, pre-K, they are not being evaluated, they don't have

13	three years of student performance data.

14	MS. HEBDA:	I don't know who does and who doesn't

15	have it.	Some teachers may end up having students for the data

16	at some point, although they certainly don't know, so, if you

17	don't know, it doesn't apply to you.	Even if you are not a

18	teacher, it doesn't apply, so, there will be students that

19	don't have three years worth of data, but then it doesn't

20	really apply to you.

21	And, remember, these standards right here, are only

22	for that one requirement that we have that's related to the

23	FCAT data.

24	And what we tried to do in the rules also, how did

25	the district set those confidence standards, that we feel are
1	

1	appropriate, and I will go over that with you in just a

2	second.

3	MS. COSTA:	Right now, this is my confusion.

4	MS. HEBDA:	Right.

Page 57

5	MS. COSTA:	At this point right now, primary teacher,

6	since there's not going to be FCAT, and the other evaluations

7	are not available, okay, there's no students that go through

8	FCAT.

9	MS. HEBDA:	That's the part in the law, that the

10	superintendent has addressed, that was a choice the district

11	made.

12	MS. COSTA:	But that's changing as well.

13	MS. HEBDA:	In 2015.

14	MS. COSTA:	So, they are going to find another way to

15	evaluate the primary teachers?

16	MS. HEBDA:	Yes, correct.	Yes, ma'am.

17	AUDIENCE MEMBER:	And when you use the data, that you

18	are talking about, in three years, you have to have three years

19	of data.	So, in three years, okay, are you saying that they

20	are going to be available to be evaluated, or that it's not

21	going to be enough, for the teacher to be evaluated, and in

22	fact, it can't be evaluated?

23	MS. HEBDA:	It's a rolling thing.	And it's a good

24	question, and in a different way, it goes to what needs to be

25	done.
1	
Page 58

1	In the evaluation system, you also can use it for

2	performance of students, if you have it, you should use at

3	least three years of student performance data for that teacher,

4	so, three years of my students, okay, and right now, we don't

5	have that data, but eventually, we will.

6	But, it doesn't go for professional practice.

7	Professional practice is just for that year, whatever you

8	collect evidence on that year.

9	Okay.	So, let's say, that they are there for a

10	couple of years.	In the '14-'15 school year, you will have had

11	evaluations for at least three years by then.

12	And then, to look back to the performance data for

13	'11-'12-- their evaluations, for '11-'12, whatever it was, for

14	'12-'13, whatever it was, '13-'14, for example, that's the

15	third year.

16	So, you don't go back and change what their data was,

17	you know what it was, at the time.

18	But, when you do their evaluation for the l3-14

19	school year, and 13-14, you look at those teachers and

20	students, all rolled up together, for three years.

21	You don't go back and say, whatever was being active

22	that year.

23	No, because that was a different evaluation.

24	But, I collect all my data, okay, and then I see, the

25	students, how did they do.
1	
Page 59

1	And then, it would be brand new, for the next year,

2	brand new.

3	AUDIENCE MEMBER:	Right, so you go back to zero.

4	MS. HEBDA:	That's right.	So, whatever you did

5	already, has not changed.

6	AUDIENCE MEMBER:	So, you have to calculate it over

7	again.

8	MS. HEBDA:	That's right, then for 14-15, you have

9	the 13-14, and 12-l3, you add them together.	And then you have

10	a brand new one for that year.

11	But you have a three year trend for students, with

12	that teacher, okay.	Okay.	That was very, very important,

13	yes.	This is very important.

14	So, it doesn't matter, okay, if you didn't have any

15	more, because that was the evaluation for that year.

16	But, all those years are rolled up together, once you

17	have three years of data.

18	Okay.	Great question.	So, this is the way that it

19	works with these conditions.	You may think of other

20	conditions.

21	And, it's important to have these things in place,

22	because they work, they all work together.

23	So, if you think of other conditions, that you think

24	we should be looking at, then let us know what those are.

25	All right.	That's a lot of information.	You know, I
1	
Page 60

1	do wnat to state in the rule--in looking at the rule, and I

2	will do it pretty quickly.

3	Okay.	Again, if you will take a look at the second

4	rule.

5	MS. COSTA:	Are the teachers going to know the cutoff

6	points of what you are doing, before they leave, or do you know

7	after?

8	MS. HEBDA:	You have to do it after.	Because, the

9	information is collected that year, you can't go for some prior

10	year, so it's important to know exactly.

11	So, that's why I emphasize the technical reports, so,

12	how much does it matter, you know, to be ELL, in 7th grade.

13	How much does that matter.	So, you can actually see those

14	things, and get an idea.

15	It's kind of like the same thing of--

16	MS. COSTA:	I look at it this way, you tell me what

17	you want me to delve at, each year, my goal.

18	MS. HEBDA:	Right.

19	MS. COSTA:	You know what I mean.

20	MS. HEBDA:	Kind of like doing it now, without

21	knowing.

22	MS. COSTA:	How do you know, you know, when you don't

23	even know what the cutoff point for that child is.	You know, I

24	don't get how--

25	MS. HEBDA:	I think, just like now, you know, at
1	
Page 61

1	sixth grade, okay, we had a course, and whether the child was

2	okay for that course--I don't know.

3	Maybe others do, but I don't know exactly how many

4	questions on the FCAT my students have to get right, to

5	necessarily mean that I have done this X, Y, and Z.	I'm

6	teaching the whole course.	You are differentiating the

7	questions.	You don't have all those things you do as a great

8	teacher.

9	MS. COSTA:	Okay.	And, if you go into education, to

10	help students, because that's what we've done--

11	Okay.	Then, I need to let them know, how many

12	questions they need to get right, and do we know all of that.

13	I'm not saying that's exactly what you teach.	But,

14	you know, it's interconnected to your career, and your

15	professionalism.	So, shouldn't we have that information.

16	MS. HEBDA:	Great question, great comments.

17	MS. COSTA:	Because, we do need it.	When we add

18	statistics to it, they are making an assumption.	When you add

19	certain, certain learning stragegies that you say will be high

20	yield strategies, not high instructional, what did you say.

21	The high probability, the high probability of that strategy,

22	not high yield, you are making it scientific.	So, I think we

23	should have more information, because I think that's the

24	biggest frustration, as teachers, that they are working

25	blindly.
1	
Page 62

1	Okay.	Sometimes, they don't know what they are

2	working for.

3	They believe it's supposed to be X, Y, and Z, but

4	they are not getting that information.

5	MS. HEBDA:	Exactly.	When we get to the comments

6	section, that's a great comment to make.	I want to do one more

7	thing, before we get to the comments section.

8	And that is, show you the part about the, if you

9	numbered your pages, it would be--it would be page seven.

10	Okay.	And that's the district established scores.

11	If you go to what would be page 7, that third

12	paragraph down, a little C on it.

13	And it says the district established performance

14	level standard for local assessment.

15	Okay.	And, where you see the C.

16	And it gives a couple of examples, as to how

17	districts can make those standards for their local assessments

18	comparable.

19	And then you are given a third option, the way it

20	works, and tell us what you think is instructionally valid and

21	turn that in.

22	Okay.	So, when you are thinking about district

23	comparable level standards, that's something else we try to

24	look for here, because this is only going to cover a few

25	people, or a certain percentage of folks.
1	
Page 63

1	So, there's a lot of other things that people teach

2	during the day, and other courses, and standards, and things

3	like that.

4	So, it's going to matter how districts work with

5	those kinds of data, yes, Ma'am.

6	AUDIENCE MEMBER:	Wouldn't they eventually have a DOC

7	standard for every course in the state?

8	MS. HEBDA:	They only have DOC for certain subjects,

9	geometry, history, and physics, so those are the things that

10	are covered.

11	But, the other things are still locally developed.

12	And, you have to go by the FCAT systems, that there's

13	certain items right there, there's four hundred some teachers

14	that were trained across the state to write assessment items in

15	K through l2 courses, and there are districts that were awarded

16	grants as to what they need to measure as far as physical

17	education, and foreign languages, and some of the other courses

18	where we don't have any certification, so, those things can be

19	available to everybody, but they will be developed and

20	determined by the districts.

21	So, then, you will be provided the items to develop

22	the kinds of tests that you want, but the district still makes

23	the tests.

24	AUDIENCE MEMBER:	But, then, you are saying, what's

25	the point of what should be used throughout the state, what's
1	

1	the assessment?

Page 64

2	MS. HEBDA:	That's a great question.	Some people

3	want more assessments, and teachers want less, so, if they are

4	being used by everybody, why don't you make it statewide.	Like

5	Hillsborough, Broward county, okay, if Broward does it

6	different from Dade.

7	AUDIENCE MEMBER:	And it would be fair, more fair, if

8	it's done statewide, DOC, that's just a comment.

9	MS. HEBDA:	Okay.	That's a comment to make.	That's

10	a very good comment to make.	Thank you.

11	Are there anymore questions, yes, Ma'am.

12	AUDIENCE MEMBER:	And I just want to know.	I need to

13	know, that if you have other methods that are being used, as

14	far as instructional methods, for example, when it comes to the

15	FCAT.

16	If they have some resources, for the instructional

17	methods, that need to be more uniform, because I'm not sure

18	they do right now.

19	MS. HEBDA:	I would say, yes, they really should,

20	yes.

21	AUDIENCE MEMBER:	And, what about if it's something,

22	for example, that one person is teaching it somewhat

23	differently, than another teacher.

24	MS. HEBDA:	I see your concern, that's a great

25	question, and I don't think there's anything in here, that
1	
Page 65

1	addresses that, since it went into effect, because, the law

2	said, that these are the options to use, to use as a

3	proficiency measure, instead of a growth measure.

4	So, the only thing that's required in here, is that

5	you would get the comparable figure, which could be a

6	comparable proficiency figure, based on the expectations, or

7	however you want to set that up, maybe some other way.	So, it

8	is a local assessment, that will still be determining what the

9	assessments are, and what the standards are.

10	AUDIENCE MEMBER:	What about the assessment, where

11	you would try to make an assessment, that it has to be a

12	standard, to establish the proficiency, and how well that they

13	are doing, if different people do things differently?

14	MS. HEBDA:	There's no requirement that they do

15	anything differently, to answer that question.

16	I think one of the things that we did do, for the

17	instructional model, that the state supervisor developed, the

18	state supervisors were the school counselors, school

19	psychologist, and school administrators, got together, and

20	decided, what a good way to do it, what's a good way to get

21	this accomplished, and there is a good model system on the

22	website.

23	I would say, it's somewhere in the forties.	They

24	actually look for examples of things that you can do, for other

25	student outcomes, and how to use, excuse me, either some
1	
Page 66

1	student assessment results, in a statewide assessment, and

2	there's some suggestions in there, as to how you should do it.

3	AUDIENCE MEMBER:	So, if I can ask you another

4	question.	And are you saying that they had a manual on that?

5	MS. HEBDA:	They did, they did.	But, yes, the state

6	manual was available for everybody to use, but it was sort of a

7	pilot.

8	AUDIENCE MEMBER:	Right, but, I don't know if they

9	are still using it.

10	MS. HEBDA:	Yes, we did use it.	This is turned into

11	a file.	And as I said, this is what we are using for the whole

12	year.

13	And, there are some districts that don't even use

14	it.	They have their own verification process, and there's

15	still some that want to use their own process, and they are

16	free to do that.

17	AUDIENCE MEMBER:	So, is there going to be a way of

18	evaluating them in a standard way, if we have been using the

19	traditional methods?

20	MS. HEBDA:	Yes.

21	AUDIENCE MEMBER:	And, I don't really know if there's

22	anything uniform.	Do you know if there's going to be

23	anything--

24	MS. HEBDA:	Okay.	The rules say you have to do your

25	input.
1	

1	AUDIENCE MEMBER:	Right.

Page 67

2	MS. HEBDA:	It doesn't say how, it doesn't say under

3	what circumstances, it doesn't say any of those things, so,

4	there's quite a variety of different categories and input out

5	there, for those evaluations.

6	AUDIENCE MEMBER:	Right, but that's what I was

7	wondering about.

8	MS. HEBDA:	Okay.	Somehow.

9	MS. COSTA:	Does that go into the district's

10	evaluation, is that what it would be?

11	MS. HEBDA:	For the performance evaluation systems.

12	MS. COSTA:	Okay.	The individual evaluation

13	components, that's part of the district, not the teachers.

14	MS. HEBDA	The teacher, administrator, and

15	supervisor.	They are all required to have their own parental

16	input.

17	MS. COSTA:	The teachers will have parental input.

18	MS. HEBDA:	The district has it in their system right

19	now, again, there's no requirement on how to do it.

20	Some of those things are said, it says, appropriate

21	parental input into the system.	So, the districts cover that

22	right now.

23	MS. COSTA:	And the reasons for-- if I can just ask,

24	the reasons for it, going back to 2012, would be, just to use

25	it as an example, okay, Kenny here is in the 5th grade, in
1	

1	2012, he takes the FCAT.

2	MS. HEBDA:	Okay.

Page 68

3	MS. COSTA:	Then in 6th grade, you take the FCAT, and

4	then the following year.

5	Exactly, how are they going to use that information,

6	to check it with other scores, looking for him.

7	MS. HEBDA:	Okay.	Well, what you do, you look

8	at--okay, he was in 5th grade, then he went to 6th grade, then

9	he went to 7th grade, so that's a couple of years, so, the 5th

10	grade score is the learning baseline, and then we add some,

11	some level of information, in the 6th grade score, too.

12	Okay.	The biggest thing is the 6th grade score.	But

13	there's information gleaned from the 5th grade score.	There's

14	information gleaned about him, he might have been an ELL

15	student for two years, now he is not.

16	And all of that information is rolled up into that

17	green book, that book is for him.

18	And then you look at at in the 7th grade, how did he

19	actually do.

20	Then you take the difference of how you expect him to

21	do, and how he actually did.

22	Okay.	And you can't always go by the prior score and

23	everything else.

24	It's what goes in there, with all the other kids.

25	MS. COSTA:	And are you saying, the green book, the
1	
Page 69

1	green bar?	Isn't the green bar going to be set after he takes

2	the test?

3	MS. HEBDA:	Okay.	Some of the things we move the

4	green bar for, are things that are captured during that very

5	school year, during his attendance, his attendance for the

6	whole year, his mobility that year.	His test scores that

7	year.

8	MS. COSTA:	So, they are not thinking of making the

9	green bar first.	And then making adjustments fit.

10	MS. HEBDA:	No, it all happens at one time.	It all

11	happens at one time.	You don't think about it, for saying, you

12	know, that they do the whole score, and go through it from

13	there.

14	It's all related, it's all together, including

15	attendance, and everything else.	Yes, Ma'am.

16	AUDIENCE MEMBER:	And, then, if you are looking at

17	how to score that, and how you look at the green bar--

18	MS. HEBDA:	If you go to fldoe.org.

19	Okay.	If you look at the bottom of this one, fldoe.

20	org.

21	When you go to that page, one of the things you will

22	see is state model information systems, if you hit on that

23	link, then you will see the instructionals.	Yes, sir.

24	AUDIENCE MEMBER:	Okay.	And, you are talking about

25	the math scores.	If one score is different, does it really
1	
Page 70

1	make it much different than another one, just one difficult

2	performance?

3	MS. HEBDA:	That's one of the things I was talking

4	about, one of the things you look at in the technical report,

5	if you look at the ratio between reading and math, one of the

6	things that you may notice, which is an assessment that we have

7	read, that even across the country, reading assessments have a

8	harder time showing differentiation, than math assessments.

9	There could be a lot of different reasons for that,

10	but that's the case.

11	So, that seems to influence, also, what you are

12	looking for, in terms of reading and mathematics.

13	Now, as to whether or not that makes a difference in

14	the standard error, so that he can fall above or below, the 95

15	percent confidence level, that's something you have to

16	examine.

17	So, it may be, when you look at your district data,

18	if you see something that you think needs to be taken in the

19	standard, we need to know that, so it can go in the rule,

20	that's a great point.

21	AUDIENCE MEMBER:	There's a lot of school level data,

22	okay, and that's something to consider.

23	MS. HEBDA:	Right, to put that all in there, right.

24	Any other questions, before we have comments.

25	Okay.	Now, we only have 15 minutes for comments.
1	
Page 71

1	But, let me see a show of hands if you want to make a comment

2	today on the record.	There's at least two, three, four.

3	Others

4	Excuse me, sorry.

5	So, one, two, three, four, five, six, 7, anyone in

6	the back.

7	Okay.	7, here's what we will do.

8	We will do two minutes per person, but, you can say,

9	you can write as long as you want, on the website.

10	So, think about what you want to do, what you want to

11	say today, so it can be in the record, but anything you give us

12	in the website is also on the record.	We will start with you.

13	Okay.	If you want to make a comment, make it, and

14	that's it.

15	MS. LAFORTE:	I'm Vicky LaForte, I'm a teacher.

16	Okay, and, I would like to go through what I'm going

17	through.

18	I'm a teacher, I used to work at a technical center.

19	Okay.	And then, we had 52 percent, that were doing

20	above the standard, okay, and it's not great, but we do pretty

21	well as a group.

22	We do very well, 75 percent, and then, I know that 75

23	percent is pretty good, 71 percent is better than overall, and

24	90 percent meet the writing standard, and 84 percent are at a

25	level 2 or higher on reading.
1	
Page 72

1	So, then, I just want to make sure, we are doing well

2	with the state FCAT data.

3	Okay.	And, according to the district, we are at 99

4	percent, with our college readiness rate, and 92 percent meet

5	the FCAT requirement for reading.

6	Okay.	And FCAT last year, the highest percent that

7	did well in the district, third highest for reading in the

8	district as well.

9	And, let's see what else.	And then, we have gotten

10	an award for excellence.

11	So, we have gotten it before.	And I'm saying that,

12	because, we are just doing very well in the district.

13	However, according to the statistics, we are the

14	worse high school in the district.	The worst high school in

15	the district.

16	How can we do this, and be the worst at the same

17	time.

18	And I don't have confidence, because I think we have

19	a good school.	We do the right things.	We do well with our

20	students.	We do very well with SGIC, and SAT.

21	And, we really do well on the SAT, when they take it

22	the first year.

23	And, then, we do the right thing, and I have a lot of

24	great students, and, I have one had a 308, one got a 309, and

25	one got a 304.
1	
Page 73

1	And that's pretty darn high, when you are talking

2	about our school.

3	Now, we also have a number of students, who had a

4	score higher than two hundred.

5	Okay.	Basically, you know, you are not going to see

6	how much better we are doing, based on what they are telling

7	us.

8	MS. HEBDA:	You have 30 seconds.

9	MS. LAFORTE:	And we were told it's very, very

10	difficult when a student scores higher than the state actually

11	says they did.	However, that's one out of every 38 students,

12	one out of every 40 students, that's very bad.

13	And, in addition, it's because we do the right thing,

14	and we have been told that if one of the variables, if you

15	score high on three or more courses, and, that's covering the

16	previous test scores.

17	But, that variable is in there currently.	And then,

18	we score higher, and everybody is really trying to do their

19	best, and that's kind of what we work out.	So, I tell them, if

20	you do the right thing, that you will become successful.

21	And, I'm a teacher, telling them that they are highly

22	effectuve, and that we still need improvement.

23	MS. HEBDA:	I am sorry, that's it.	Sorry, out of

24	time.

25	MR. MINCHEW:	Okay.	I'm just going to read it for
1	
Page 74

1	the record.	I do test scores, and evaluate teachers, in the

2	program, and we haven't had many problems.

3	Here's the one thing, influences.	Test scores are

4	influenced by many things, including incoming level, including

5	the teachers, the attitude of teachers, one cannot immediately

6	separate the influence that a particular teacher or program

7	might have on the variables.

8	And, polls, polls take samples, they cover a small

9	selection, and don't even give a name.	A student's scores

10	represent how much has been learned on all materials, but his

11	tests, like polls, can be misleading.

12	And multiple tests, if you measure it, and some do it

13	better than many, in their dealings as teachers.

14	Attitude, engagement, and then on one's own, it is

15	difficult to measure the tests.

16	In some cases, it would be most important to do other

17	measures than the tests.

18	In addition, and, if you can consider, that the tests

19	have demonstrated, what is widely ignored by many, things that

20	are ignored by many in the education establishment.

21	Every teacher knows that providing strategy for test

22	taking, in their performance, and knowing their curriculum,

23	that will be on the tests, can have greater effect.

24	And, sometimes, their effect is substantial.

25	And, one cannot focus on test scores, while at the
1	
Page 75

1	same time, wondering about student loans.	And that affects a

2	student's achievement.

3	And, it's hard to determine an individual teacher's

4	performance, because a small amount of student scores produces

5	a lot of errors.

6	Even with three years of test data, the department of

7	education estimates that scores will be wrong 25 percent of the

8	time.

9	It could be 20 percent one year, and 57 the next,

10	therefore, they are not reliable, and not a sufficiently valid

11	measurement for judging teacher effectiveness and teacher

12	performance.	Thank you.

13	MS. HEBDA:	Thank you.	And do you want to give me a

14	copy of that, too?

15	MR. MINCHEW:	Yes, I guess I could.

16	MS. HEBDA:	Thank you.

17	MR. DWYER:	Okay.	And, I'm a supervisor of education

18	at FIU.

19	And I received my doctorate in advanced education,

20	and I have been on the board of directors since 2000.

21	And I'm worried about the fact about scheduling

22	requirements, for the curriculum development, and then,

23	specifically, for it to relate to those, who don't have

24	sufficient credentials.

25	So, I just want to say that there should be
1	
Page 76

1	credentials for teachers, that require more training for them,

2	they are each required to have fifty hours of training

3	themselves.

4	In other words, they need to have actually more hours

5	of training, than the teachers who gave them the training.

6	Okay.

7	And, teachers need to have sound, effective teacher

8	training.	But, they don't.	And even though they may have a

9	stamp on a transcript, why am I worried about this.	There's no

10	evidence in the last ten years that the student graduation rate

11	has gone higher, it's 67 percent, and it's been that way for

12	the last ten years.	And, the only requirement that meets our

13	credentials are with minority students, who are expected to do

14	much better, on at least improve as to the graduation rate.

15	And there's no specification, that teachers who have

16	ESOL training, are teaching ESOL in class.

17	In fact, we don't do that at all.	We teach science,

18	and we see a lot of them who need it.

19	And, sometimes, ESOL students only pass the FCAT on

20	the 33rd percetile, when it should be mainstream.

21	But, there's recent evidence, that the rates are low,

22	but in other cases, ESOL is at the mainstream.

23	So, don't be surprised if teachers in classes,

24	without this variable, are then strapped with teaching ESOL,

25	and hoping for change
1	
Page 77

1	But, with the hopes that, if they do this, we hope

2	that they consider ESOL performance beyond the mainstream.

3	Hopefully, scoring well beyond the 50th percentile.

4	MS. HEBDA:	Thirty seconds.

5	MR. DWYER:	That's okay.	I think we can evaluate

6	ESOL, with baseline curriculum.	And, in fact, the teachers who

7	have the endorsement must be considered for teaching with the

8	ESOL evaluation.

9	And, actually, they need to be classified with the

10	evaluation, and then, if they pass that, they can get the

11	endorsement.	As such, the whole conversation, about value

12	added, is really absurd.

13	MS. HEBDA:	Thank you.

14	MS. COSTA:	Okay.	My first question is, okay.	How

15	is this going on so long.	Okay.	This is going back to more

16	questions.

17	I mean, if teachers are going to be evaluated by

18	students, and student scores, then the student date needs to be

19	provided prior to the testing, not after the testing.

20	And, actually, we need to know what their goals are,

21	and we need to have clear goals, on the tests.

22	It seems, all we are concerned about is this testing,

23	we are not concerned about what they will be learning, we are

24	concerned about how they are going to perform on the tests.

25	So, actually, that has to be-- that has to be actually put in
1	
Page 78

1	for the students, that we are expecting their expectations to

2	score higher.	You know, you have to be prepared, you have to

3	do this.

4	Okay.	My other concern is, my other concern, I'm not

5	convinced since 2011, using 2011-2012 testing data, is a valid

6	place to begin using the data, and using that model.

7	And, my last concern is the cost, and impact that it

8	will have on dollars spent in the actual classroom.

9	We already know our technology is very poor.

10	And, I feel like I'm spending my money in the wrong

11	place, I really do.	So, I think we should be spending more

12	money on professional development, and on technology, in the

13	classroom.	And that's my comment.	And there's one other thing

14	I would like to read, and this is more from myself, because we

15	are getting to the point, where we are being told to do certain

16	things, and we just don't have these miracle children.

17	There are no high yield instructional strategies,

18	there are only high probability strategies.	And the simple

19	presence or absence of an instructional strategy does not

20	define effectiveness but it's a teacher's expertise in adopting

21	the strategy in the classroom, within the context, of correct

22	teaching, that produces gains.	So, thank you very much.

23	AUDIENCE MEMBER:	And, I think we have gotten carried

24	away.

25	MS. SIWIAK:	Thank you.
1	
Page 79

1	And I would like to tell you, that I'm a teacher.	In

2	the Broward County system, there are schools that have

3	assistant principals, who have yet to have a formal evaluation

4	of their teachers.

5	There are two schools that have 18 teachers before

6	them, and, to my knowledge, they are in the office, that to my

7	knowledge, that's all they do, no student discipline, who have

8	been living in small rooms since August.

9	This is tantamount to harassment, and then, let's not

10	forget the VOI.

11	Take a hard look at the district, and the

12	implementation of certain criteria, because, in Broward

13	schools, none of this is being done.

14	And, that's what concerns me.	Also, the fact that

15	this is being used as a weapon, to get rid of expensive

16	teachers, with a lot of years of experience.	Teacher aides.

17	MS. WILLIAMS:	Okay.	And, I will come forward, so

18	that I can read it to you.

19	Okay.	So, what I would like to say, for you to

20	consider, I have several questions.

21	The first question, okay, the models based on two

22	years of prior scores.

23	And I think going back to that early rate, of those

24	two years, that those are constant, that those identical, on

25	that information, and that's very relevant.
1	
Page 80

1	And, I'm teaching at a school, where 64 percent of my

2	school is Hispanic, and about 15 percent, is just other

3	minorities.	So, we have a lot of language issues that happen.

4	And, we are not taking that into account, many

5	immigration issues, deaths of their parents.

6	I feel like something needs to be stated, so the

7	transcript wouldn't look so hard, but it's something that the

8	teachers can go to help our students, something that takes into

9	account, life events, you know, that this student doesn't have

10	help in the home, but, definitely, they don't have the

11	resources that other students might have, so, that's for your

12	consideration.

13	And my second question is about student disabilities,

14	who take the FCAT, the students with disabilities who take the

15	same FCAT.	And, then, students with disabilities, who don't do

16	well on math, and on their language levels, and that means that

17	on these scores, when you look at them-- how does it tie into

18	the students themselves.

19	And, you have to look at their grades, but

20	academically, they are at a second grade level.

21	They can't do well on the FCAT, they are not going to

22	pass it, because they are at a second grade level.

23	How does that information help or impact the

24	students, more importantly?

25	More importantly, how does it help the teacher, the
1	
Page 81

1	teacher gets a school score.	But, there's progress that that

2	teacher may have made, from a low second grade, to almost at

3	third grade, we need to take a look at the differences of

4	students, the testing differences of students, and give more

5	appropriate studies, that would be more beneficial to the

6	students.

7	MS. HEBDA:	Thank you.	Is there anyone else.	Anyone

8	else.	Okay.	That's it.

9	Thank you very, very, very much for being here today,

10	and for asking the questions, and giving your comments.

11	Thank you, okay, and thank you all for being here.

12	(Thereupon, the meeting was concluded).

13

14

15

16

17

18

19

20

21

22

23

24

25

1

2

3

4	STATE OF FLORIDA:

5	COUNTY	OF	DADE:

CERTIFICATE

Page 82

6	I, NEYSA SOSA, Registered Professional Reporter, certify

7	that I was authorized to and did stenographically report the

8	foregoing meeting; and that the transcript is a true record

9	of the proceedings.

10	I further certify that I am not a relative, employee,

11	attorney, or counsel of any of the parties, nor am I a

12	relative or employee of any of the parties' attorney or

13	counsel connected with the action, nor am I financially

14	interested in the action.

15	DATED this	day of March, 2013.

16

17

18	NEYSA SOSA, RPR, FPR

Page 83

 	A 	 abilities (1) 31:24
able (2) 5:1 6:14
absence (2) 16:13
78:19
absolute (1) 34:24
absolutely (1) 32:23
absurd (1) 77:12
academically (1)
80:20
accomplish (1) 5:4
accomplished (2)
15:10 65:21
accomplishes (1) 5:15
account (10) 31:7,8,9
36:3,8 48:22 49:13
50:10 80:4,9
accounted (2) 36:10
50:11
achieve (2) 21:22
54:18
achievement (2)
31:11 75:2
act (1) 14:7
action (2) 82:13,14
active (1) 58:21
actual (2) 44:2 78:8
add (6) 8:12 56:5
59:9 61:17,18
68:10
added (11) 8:5 28:10
28:13,20 32:23
34:11 40:14,20
43:24 53:15 77:12
addition (2) 73:13
74:18
additional (6) 4:12
9:9 19:11,13,15,17

agree (1) 8:11	assistance (1) 19:12	19:24 35:22 41:18	certain (13) 18:24,25
ahead (1) 3:23	assistant (1) 79:3	42:16 52:1 56:5	20:8 26:13 39:14
aides (1) 79:16	associate (1) 3:4	73:5	55:16 61:19,19
algebra (3) 37:3,8,25	associated (1) 24:11	beach (1) 37:16	62:25 63:8,13
allotment (1) 50:5	association (1) 7:11	bebecause (1) 45:16	78:15 79:12
amendments (1)	assumption (1) 61:18	beginning (1) 38:7	certainly (1) 56:16
19:14	atokay (1) 68:8	beit (1) 62:9	certificate (1) 82:3
amount (3) 7:18	attendance (6) 35:24	believe (1) 62:3	certification (1)
33:21 75:4	39:21,22 69:5,5,15	beneficial (1) 81:5	63:18
analysis (1) 50:16	attending (1) 54:15	best (5) 29:14 43:23	certify (2) 82:6,10
ann (1) 2:10	attitude (2) 74:5,14	44:18 49:1 73:19	chance (1) 48:9
annual (3) 12:22 13:1	attorney (2) 82:11,12	better (8) 34:2 44:14	chancellor (1) 3:7
23:1	attributable (1) 30:20	44:15 53:19 71:23	change (10) 9:15 22:4
annually (2) 16:9	audience (32) 2:11	73:6 74:13 76:14	22:6,7,11,12 42:3,3
19:10	12:4 26:20,24 27:4	beyond (3) 19:15	58:16 76:25
answer (4) 3:10 6:12	32:21,25 34:19	77:2,3	changed (3) 8:6 22:22
6:14 65:15	36:20 37:1,10 49:9	bigger (1) 45:15	59:5
answered (1) 6:10	51:18 57:17 59:3,6	biggest (3) 35:7 61:24	changes (3) 10:8 22:3
answers (1) 6:7	63:6,24 64:7,12,21	68:12	52:16
anymore (3) 38:2	65:10 66:3,8,17,21	biology (1) 37:9	changing (1) 57:12
51:16 64:11	67:1,6 69:16,24	bit (2) 4:11 31:19	characteristic (3)
anyones (1) 39:25	70:21 78:23	bldg (1) 1:12	30:16 33:7,9
anytime (1) 21:25	august (1) 79:8	blindly (1) 61:25	characteristics (11)
anyway (1) 10:19	authority (5) 11:13	bmps (1) 38:13	30:10,13 31:3,6,7,8
appeal (1) 52:23	12:15,16 13:10	board (10) 9:24,25	32:12 41:25 42:4
applied (1) 56:9	24:7	19:16,18,18 22:4,5	42:13 48:21
applies (1) 37:13	authorized (1) 82:7	27:18 29:10 75:20	check (3) 15:6 54:7
apply (6) 28:2 48:10	available (7) 12:9	boards (1) 19:20	68:6
55:3 56:17,18,20	17:24 55:23 57:7	book (5) 6:6 31:5	checkpoints (1) 53:1
appreciate (5) 3:17	57:20 63:19 66:6	68:17,17,25	child (3) 39:10 60:23
4:22,22 10:17 12:1	avenue (1) 24:23	bottom (3) 44:10	61:1
appropriate (11)	average (12) 33:25	46:21 69:19	children (1) 78:16
15:20 16:21,23,25	34:2 41:4 44:3,3,6	brand (3) 59:1,2,10	childs (1) 39:5
17:6,7,12 28:4 57:1	46:16 48:16 53:10	bring (2) 38:24 51:21	choice (1) 57:10
67:20 81:5	53:13,15,18	broward (13) 1:10	choose (2) 16:18,20
appropriately (1)	award (1) 72:10	2:5,6,7,9,10 3:3	chose (1) 50:3
17:10	awarded (1) 63:15	37:16 51:14 64:5,5	circumstances (7)
approval (6) 11:18			79:2,12	19:5 39:14,15 50:1

address (2) 5:21

12:20 21:15,17

 	B 	    	

50:10,11 67:3

49:20

22:2 23:8

bachelors (2) 3:4,5

 	C 	

city (1) 4:8

addressed (2) 20:25
57:10
addresses (1) 65:1
adjust (1) 33:8
adjustments (1) 69:9
administrator (5)
11:9 15:19 55:24
55:25 67:14
administrators (7)
12:23 13:2 14:14
14:24 18:7 29:8
65:19
adopt (2) 9:18 28:7
adopted (3) 36:25
37:17 49:23
adopting (1) 78:20
adoption (1) 10:1
advanced (1) 75:19
affect (5) 9:6 10:9
34:11 38:15 39:5
afternoon (1) 3:2
age (1) 31:23
agenda (2) 5:1 7:5
ago (2) 20:18 50:14

approved (4) 17:17
21:19,20 24:9
april (1) 9:3
area (3) 45:5 55:3,21
areand (1) 46:12
areas (2) 15:5 19:3
arts (1) 3:4
asked (1) 51:17
asking (2) 7:9 81:10
assess (2) 17:14 55:6
assessed (1) 16:9
assessment (17) 16:9
16:11,15,23 17:6,9
17:13 44:4 62:14
63:14 64:1 65:8,10
65:11 66:1,1 70:6
assessments (8) 16:19
17:11 28:4 62:17
64:3 65:9 70:7,8
assign (1) 17:19
assigned (4) 29:23
41:16 42:4 54:14
assignments (1)
16:19

back (32) 3:16 4:25	c (2) 62:12,15
6:14 7:4 11:23 12:2	calculate (3) 24:22
12:10,12,15 20:2,14	47:17 59:6
21:5,14 28:25	calculated (1) 47:12
30:23,24 31:17	calculating (1) 51:19
32:7,17 33:16	called (1) 21:8
41:20 51:3 52:20	campus (1) 1:10
54:17 58:12,16,21	cant (6) 6:13 36:13
59:3 67:24 71:6	57:22 60:9 68:22
77:15 79:23	80:21
bad (1) 73:12	capture (1) 6:13
bar (10) 41:10,10	captured (2) 38:19
46:7,8 53:9 69:1,1	69:4
69:4,9,17	capturing (1) 39:12
base (2) 37:11 51:21	card (3) 6:25 7:13,22
based (21) 15:3 16:8	career (1) 61:14
16:16,19 17:2 23:4	carried (1) 78:23
30:8,11,18 33:6	case (7) 11:15 36:5,6
36:2,7,7 41:7 45:7	36:6 46:13 49:21
49:21 53:24 55:22	70:10
65:6 73:6 79:21	cases (2) 74:16 76:22
baseline (4) 47:25	categories (1) 67:4
52:18 68:10 77:6	center (1) 71:18
basically (8) 8:3	central (1) 1:10

clarification (1) 20:9
class (5) 24:18 36:12
39:25 48:4 76:16
classes (1) 76:23
classification (3)
43:12 44:21,22
classified (1) 77:9
classroom (9) 15:9,11
15:18,23 31:7
39:11 78:8,13,21
clear (2) 25:16 77:21
close (2) 4:20 42:9
code (1) 19:19
collaboration (1)
44:13
collect (7) 10:21
34:25 38:12 39:17
39:20 58:8,24
collected (2) 38:17
60:9
college (7) 1:10 2:5
3:3,5,12,13 72:4
combine (2) 16:18
47:13

Page 84

combined (2) 34:17	42:24 45:15,21,22	11:20 12:18 52:22	decides (1) 32:11	57:24 58:23 64:6
47:15	46:8,8,15,20 47:6	58:10 62:16 68:9	decisions (1) 14:13	65:13 67:4 69:25
come (8) 4:20 6:9	48:1,7,19 54:5,12	course (8) 17:15	declared (1) 26:5	70:1,9
7:21 22:2 23:4	confidential (4) 44:23	39:21,21,21,22 61:1	deficit (1) 34:16	differentiating (1)
34:12,20 79:17	45:1,2,6	61:6 63:7	define (2) 22:10	61:6
comes (6) 33:16 38:8	confusion (1) 57:3	coursei (1) 61:2	78:20	differentiation (2)
38:9 41:12 54:13	connected (1) 82:13	courses (8) 13:20	defined (1) 47:24	25:17 70:8
64:14	consider (10) 21:9	17:5,11 31:16 63:2	definitely (1) 80:10	differently (3) 64:23
coming (7) 9:9 11:21	23:13 31:19 36:18	63:15,17 73:15	definition (1) 48:14	65:13,15
11:22 17:20 32:13	41:9,15 70:22	court (4) 4:1,21 6:7	definitions (4) 20:13	difficult (3) 70:1
37:4,5	74:18 77:2 79:20	7:2	21:21 53:21 54:21	73:10 74:15
comment (20) 6:3,20	consideration (12)	cover (4) 22:5 62:24	degree (1) 3:5	directly (1) 56:8
6:22 7:1,9,12,22,23	8:18,19,21 9:24	67:21 74:8	deleted (1) 8:5	directors (1) 75:20
7:25 8:17 39:9 40:3	10:1 27:9 43:8 44:5	covered (1) 63:10	delve (1) 60:17	disabilities (6) 32:15
52:20 62:6 64:8,9	46:1 54:10,24	covering (1) 73:15	demonstrated (1)	35:13,23 80:13,14
64:10 71:1,13	80:12	credentials (3) 75:24	74:19	80:15
78:13	considered (3) 16:2	76:1,13	denied (1) 21:20	disability (4) 32:13
comments (18) 4:19	17:3 77:7	credit (2) 49:13 54:4	department (14) 1:4	32:19,20,22
4:20 6:17,19 7:16	consistent (1) 39:2	criteria (11) 14:22	2:4 13:16 14:4	discipline (1) 79:7
8:7 9:4,8,9 20:7	consistently (1) 43:16	15:20 16:2 24:1	22:14 23:14 24:2	discretion (1) 50:1
23:21 39:7 61:16	constant (1) 79:24	26:18,25 43:9,10	24:13,23 27:7,19,23	discuss (1) 38:18
62:5,7 70:24,25	consult (1) 31:15	49:6 55:18 79:12	32:14 75:6	discussed (2) 39:13
81:10	contained (2) 9:7	criterion (2) 43:22	departments (2) 13:7	50:14
commissioner (1)	24:14	50:13	13:10	discussing (1) 10:5
24:9	contemporary (2)	current (2) 10:9	depends (1) 19:5	distribution (4) 44:9
committee (7) 28:16	15:4 22:8	25:10	describe (1) 31:5	44:12 50:22,25
32:11,14 34:24	content (3) 20:21	currently (1) 73:17	designed (1) 14:9	district (61) 7:11 11:8
35:20,22 38:18	21:4 28:11	curriculum (3) 74:22	determination (1)	11:18 12:20 13:4
community (1) 29:12	contents (4) 8:11 11:4	75:22 77:6	13:23	13:18,21 14:4,11,18
companion (1) 24:5	11:4 24:7	cutoff (2) 60:5,23	determinations (1)	14:25 17:5,16,18,25
comparable (4) 62:18	context (1) 78:21			25:13	18:12,23 19:9,11,25

62:23 65:5,6	continue (2) 3:23
comparing (1) 36:21	4:14
compile (1) 4:10	contribution (1)
compiling (1) 9:8	28:22
completely (2) 47:8	conversation (1)
47:10	77:11
complying (1) 14:2	conversely (1) 26:16
components (4) 18:3	convinced (1) 78:5
18:13 20:8 67:13	cooperation (1) 4:23
composite (1) 35:23	copy (1) 75:14
concern (5) 49:12	correct (4) 18:6 42:25
64:24 78:4,4,7	57:16 78:21
concerned (5) 38:25	correctly (2) 7:14
39:10 77:22,23,24	42:21
concerns (2) 49:10	corresponded (1)
79:14	51:23
concluded (1) 81:12	cost (1) 78:7
conclusion (1) 50:24	costa (34) 2:9 33:11
concrete (1) 25:16	33:17,22 37:19,22
condition (1) 55:10	38:1,4,12 39:2 40:4
conditionally (1)	40:8 51:3,6,15
21:19	56:11 57:3,5,12,14
conditions (5) 26:13	60:5,16,19,22 61:9
55:16 59:19,20,23	61:17 67:9,12,17,23
conduct (1) 19:19	68:3,25 69:8 77:14
conducted (1) 18:8	counsel (2) 82:11,13
confidence (17) 45:3	counselors (1) 65:18
45:19,21 46:19	count (1) 34:7
47:2,18 48:10,18,22	country (1) 70:7
51:10 53:2,7,24	county (7) 4:7 37:16
54:1 56:25 70:15	37:16 51:14 64:5
72:18	79:2 82:5
confident (15) 42:12	couple (8) 5:8 8:1

 	D 	
dade (2) 64:6 82:5
darn (1) 73:1
data (55) 16:8,13
17:2 18:15,17
30:24 31:25 32:2,3
33:19,19 34:4,25
35:10 36:6 37:14
38:5,12,16,18,19
39:16,17 40:3,14
43:11,17,17 44:3,6
46:3 53:23,24
55:23 56:4,4,13,15
56:19,23 57:17,19
58:3,5,12,16,24
59:17 63:5 70:17
70:21 72:2 75:6
78:5,6
database (1) 35:1
date (1) 77:18
dated (1) 82:15
davie (2) 1:11,13
day (4) 29:1 39:24
63:2 82:15
dealings (1) 74:13
dean (1) 3:4
deaths (1) 80:5
debate (1) 6:22
decade (1) 22:15
december (1) 23:3
decide (3) 14:19
37:14 42:8
decided (1) 65:20

determine (3) 27:19	20:15,24 21:13,24
47:15 75:3	22:3,10,21 23:4,16
determined (2) 27:21	23:25 24:15,20
63:20	28:2,3 36:14 37:14
determines (1) 30:17	37:18 39:21 40:1,5
determining (1) 65:8	40:18 42:6 43:25
develop (1) 63:21	49:25 50:9 56:25
developed (5) 28:13	57:10 62:10,13,22
28:18 63:11,19	63:22 67:13,18
65:17	70:17 72:3,7,8,12
developing (3) 14:11	72:14,15 79:11
29:1,2	districts (11) 16:18
development (7) 1:5	27:22 50:4 52:7
3:9 14:12 18:18	62:17 63:4,15,20
22:19 75:22 78:12	66:13 67:9,21
developmental (1)	districtthe (1) 47:14
30:4	doc (3) 63:6,8 64:8
didnt (5) 26:24 35:19	doctorate (1) 75:19
38:18 40:25 59:14	doe (1) 12:24
difference (13) 30:20	doeapproved (1) 13:8
32:5,9,18,18 34:16	doesnt (18) 8:15 27:6
34:16 37:15 41:11	34:11 38:13 41:22
46:13 48:6 68:20	49:12 53:7,21
70:13	56:14,17,18,19 58:6
differences (5) 30:22	59:14 67:2,2,3 80:9
33:12,15 81:3,4	doing (20) 3:18,19
different (30) 8:16	4:16,17,24 10:17
15:1 19:5 20:17	17:7 20:19 31:17
28:3,4,20,21,23	32:6 34:21 37:22
29:16,18 31:23	52:11 60:6,20
32:12,19 45:1	65:13 71:19 72:1
46:13 49:3,5 52:13	72:12 73:6
52:20,22 53:1	dollars (1) 78:8

Page 85

dont (69) 6:21 7:14	endorsement (2) 77:7	63:6	factors (5) 28:21,24	foregoing (1) 82:8
7:17 9:14,19 10:16	77:11	everthing (1) 23:16	29:16 31:1 49:14	foreign (1) 63:17
10:20 15:6 16:25	engagement (1) 74:14	everybody (18) 3:22	fails (1) 33:21	forget (1) 79:10
21:1 32:9,9 34:7,25	english (1) 35:14	5:1 6:2 12:14 26:10	fair (2) 64:7,7	form (1) 50:17
34:25 35:22 36:8	entire (5) 26:5,14	28:2 29:15 33:9	fairly (1) 54:2	formal (1) 79:3
37:1 38:5,12 39:2	29:1 47:7 55:19	37:18 44:8,13,14,20	fall (1) 70:14	formula (1) 24:8
39:16,20 41:13,23	entirely (2) 46:19	49:22 63:19 64:4	far (2) 63:16 64:14	formulas (4) 27:13,21
43:22 44:15 48:1	47:1	66:6 73:18	fcat (39) 28:5,8,18	27:24 28:1
50:4,6,19,19,25	equally (2) 28:1,4	everybodys (1) 46:25	29:17 30:1 31:12	forth (1) 15:1
51:1,5 52:8 53:23	eric (1) 2:8	evidence (6) 20:23	31:16,17 33:12	forties (1) 65:23
56:4,8,12,14,16,17	error (5) 41:16,17	21:3,9 58:8 76:10	34:4,10 35:2,5	forward (6) 5:13 6:9
56:19 58:4,16,21	42:8 44:25 70:14	76:21	36:20,24 37:7,21,22	7:2,21 52:4 79:17
60:22,24 61:2,3,7	errors (5) 42:7 45:1,8	evident (1) 23:7	37:23,24 38:2,22	found (1) 55:17
62:1 63:18 64:4,25	46:16 75:5	exact (3) 5:12 9:7,13	50:15,20 56:23	four (10) 14:18 21:5,7
66:8,13,21 69:11	esol (8) 76:16,16,19	exactly (7) 28:18	57:6,8 61:4 63:12	30:5 34:5,9 53:12
72:18 74:9 75:23	76:22,24 77:2,6,8	31:20 60:10 61:3	64:15 68:1,3 72:2,5	63:13 71:2,5
76:8,17,23 78:16	essentially (1) 53:10	61:13 62:5 68:5	72:6 76:19 80:14	fourth (1) 50:8
80:10,15	establish (12) 11:17	examine (1) 70:16	80:15,21	fpr (1) 82:18
draft (4) 8:17,20,24	12:18,19 22:13	example (19) 16:22	february (1) 1:18	frances (1) 2:9
9:13	25:5,23 26:3,13	17:8,15 24:13	feedback (11) 5:13,17	free (2) 3:13 66:16
dwyer (3) 2:8 75:17	27:7 39:2 44:17	29:19,20 31:22	8:21 9:14,17,19,21	front (5) 4:1 5:15 6:8
77:5	65:12	38:17 41:18 42:10	25:4,18 27:11	8:4 11:6
 	   established (4) 19:21	45:12 53:6,11,19	44:16	frustration (1) 61:24
 	E		27:20 62:10,13	54:9 58:14 64:14	feel (6) 3:13 6:21 8:13	fully (1) 21:19

earlier (2) 8:1 18:22
earliest (1) 10:10
early (1) 79:23

establishing (2) 25:15	64:22 67:25	56:25 78:10 80:6	further (1) 82:10
27:9	examples (3) 27:25	fields (4) 18:21 19:1,6	future (1) 35:4
establishment (1)	62:16 65:24	50:2

easier (1) 44:12
education (11) 1:4
2:4 3:4,6 32:14
61:9 63:17 74:20
75:7,17,19
educational (1) 15:3
educator (3) 15:10
18:4 19:23
educators (2) 28:14
29:7
effect (6) 10:3,6 14:1
65:1 74:23,24
effective (22) 14:9
15:4,23,23 39:3
43:6 46:11,14
47:20,22,22 48:13
48:13,14,17 53:7,14
54:8,16,16 55:17
76:7
effectively (1) 42:8
effectiveness (5)
15:17 47:24 51:8
75:11 78:20
effectuve (1) 73:22
eight (3) 28:19,23
53:13
either (3) 41:12,13

74:20	exceed (2) 41:13,13	fifth (1) 50:8
estimates (2) 24:14	exceeded (2) 40:17	fifty (6) 16:7 17:1
75:7	53:8	35:14,14 41:21
ethics (1) 19:19	excelled (1) 40:17	76:2
evaluate (4) 15:25	excellence (1) 72:10	figure (2) 65:5,6
57:15 74:1 77:5	excuse (2) 65:25 71:4	file (1) 66:11
evaluated (8) 18:2	expect (6) 13:22	fill (2) 7:13,22
55:20,21 56:12	20:24 30:17 35:8	filling (1) 6:25
57:20,21,22 77:17	41:5 68:20	final (1) 47:15
evaluating (2) 49:17	expectation (1) 51:20	financially (1) 82:13
66:18	expectations (10)	find (4) 16:21 26:16
evaluation (64) 11:9	15:13 40:25 41:8,9	35:19 57:14
11:18 12:21,22	41:12 51:19,25	finish (2) 20:12 40:10
13:1,4,6,8,8,11,12	54:11 65:6 78:1	first (27) 5:7,10 6:4
13:13 14:4 15:20	expected (2) 40:21	11:5,6,9,11 12:19
16:2,8 17:1,2,7,21	76:13	14:7 16:6 20:18
17:25 18:4,13,16	expecting (1) 78:1	23:24 24:8 26:10
19:9,15,23 20:15,22	expensive (1) 79:15	27:13 28:9 31:5
21:13 22:15 23:11	experience (1) 79:16	43:7 45:2 47:24
24:1,15 25:9 26:5	expertise (1) 78:20	49:20 50:7,12 69:9
26:11,14 32:7	explain (1) 20:23	72:22 77:14 79:21
33:21 37:15 38:10	explained (1) 26:21	fit (1) 69:9
39:4 40:2,3 42:23	explains (1) 31:10	fiu (1) 75:18
43:13 48:12 51:4	explicitly (1) 36:8	five (12) 21:7,15
54:18 55:4,8,19	explored (1) 28:19	22:21 23:5 30:5,6
56:1 58:1,18,23	extremes (1) 47:11	34:6,7,9 35:6 45:14
59:15 67:10,11,12			71:5

 	G 	
gains (1) 78:22
geometry (1) 63:9
getting (8) 32:25
34:10,20 36:2
37:10 44:19 62:4
78:15
getwhat (1) 5:12
give (16) 5:11 6:14,19
6:23 7:24 8:17 9:14
29:20 48:23 52:23
53:21 54:4 71:11
74:9 75:13 81:4
given (6) 18:17 20:6
42:12 45:9 50:16
62:19
gives (1) 62:16
giving (3) 38:1 49:13
81:10
gleaned (2) 68:13,14
go (49) 5:13,25 6:14
7:4,6,24 10:3,6,11
13:15 14:19 20:2
20:12 23:19,23
29:13,16 30:10,16
31:17,18 35:3
42:18 43:5 44:21

65:25
elements (2) 19:25
20:1
ell (2) 60:12 68:14
elses (1) 29:17
emailed (1) 5:19
emphasize (1) 60:11
employ (1) 49:6
employee (2) 82:10
82:12

77:8,10 79:3
evaluations (14) 14:9
14:21 15:1 18:8
19:4,13 26:1 29:3
56:2,10 57:6 58:11
58:13 67:5
evaluator (1) 19:14
evaluators (1) 18:10
events (1) 80:9
eventually (2) 58:5

 	F 	
face (2) 9:2,2
facetoface (2) 3:25
4:13
facilities (1) 40:6
fact (12) 5:23 15:5
16:22 19:8 35:9
43:4 48:22 57:22
75:21 76:17 77:6
79:14

fldoe (3) 5:23 69:18
69:19
florida (9) 1:3,13 2:8
3:8 11:15 12:16
15:10 28:14 82:4
focus (1) 74:25
folks (2) 13:9 62:25
following (7) 3:1 10:4
10:6 17:23 26:1
43:6 68:4

46:2,17,24 52:16
54:17 55:25 57:1,7
58:6,16,21 59:3
60:9 61:9 62:11
63:12 67:9 68:22
69:12,18,21 70:19
71:16 80:8
goal (2) 44:20 60:17
goals (2) 77:20,21
goes (7) 43:4 44:24

Page 86

51:11 54:2 55:11	47:2,9 53:10,17	78:2	46:18 59:12,13,21	instructions (1) 3:21
57:24 68:24	65:3	highest (3) 43:25 72:6	60:10 74:16	instructors (1) 56:8
going (68) 4:10,18,24	guess (1) 75:15	72:7	importantly (2) 80:24	instruments (1) 18:5
5:21 6:21 7:8 8:14	guided (1) 34:8	highly (11) 15:23	80:25	insure (1) 25:16
9:3 10:14,18 11:7	guidelines (4) 23:11	26:17 46:10,14	improve (2) 34:15	intend (2) 5:12,16
11:10 13:17,19,22	23:13,15 49:22	47:22 48:13 53:6	76:14	interaction (2) 6:18
13:25 14:5 19:24			53:13 54:8,16	improvement (5)	9:17

20:24 21:10 22:9
23:20,22 24:4 25:3
28:25 29:19 30:6
30:23,24 31:16,18
35:5 38:4 42:20
44:17 45:15,17,18
45:19 48:8,13 49:4
51:12,18 52:2,20
57:6,14,20,21 60:5
62:24 63:4 66:17
66:22 67:24 68:5
69:1 71:16 73:5,25
77:15,15,17,24

 	H 	
hand (4) 3:20,20 5:23
12:11
handout (2) 12:14
20:6
handouts (2) 12:2,3
hands (1) 71:1
happen (4) 9:11 38:8
50:18 80:3
happens (7) 6:5,5 9:6
18:24 35:2 69:10
69:11
harassment (1) 79:9

73:21	14:11 15:23 17:17	interconnected (1)
highs (1) 36:14	18:18 73:22	61:14
hillsborough (1) 64:5	include (4) 15:12	interested (1) 82:14
hired (1) 18:8	24:15 28:21 32:11	international (1) 2:8
hispanic (1) 80:2	included (15) 14:4	interval (3) 44:23
history (1) 63:9	18:15 20:22 23:1,9	45:2,7
hit (1) 69:22	23:25 24:16 25:2	intervals (1) 45:2
hold (2) 9:21 11:20	28:24 32:16 36:12	isnt (1) 69:1
home (6) 34:21,22	37:2 38:14 39:13	issue (3) 7:8 26:11,12
35:1 36:2 46:2	56:9	issues (2) 80:3,5
80:10	includes (1) 14:24	item (1) 46:17
homework (1) 34:21	including (6) 19:12	items (3) 63:13,14,21
hope (2) 9:18 77:1	30:9 36:12 69:14	 	

79:23 80:21
good (13) 3:2 7:18
31:25 45:22 47:1
50:23 57:23 64:10
65:20,20,21 71:23
72:19
gotten (3) 72:9,11
78:23
grade (24) 16:10,23
30:15 31:21,24

hard (4) 50:13 75:3
79:11 80:7
harder (3) 35:4 44:11
70:8
havent (2) 50:16 74:2
havewe (1) 3:18
hazard (1) 46:25
hear (3) 3:16 4:21 7:2
hebda (81) 2:4 3:7,15
5:20 10:16,20,24

hopefully (2) 3:10	74:4,4
77:3	incoming (1) 74:4
hopes (1) 77:1	increase (1) 15:24
hoping (2) 8:3 76:25	increases (1) 15:25
hours (3) 29:9 76:2,4	independently (1)
huge (1) 7:17	32:16
hugely (1) 8:10	indicates (1) 40:20
human (1) 14:12	indicators (3) 16:8
hundred (4) 35:13	17:2 18:5
36:15 63:13 73:4	individual (8) 16:12

 	J 	
jacksonville (1) 4:7
job (4) 14:23 15:13
15:13 16:17
judging (1) 75:11
judgment (2) 49:1,1
judgments (1) 49:3
juliana (2) 2:5 3:3
june (3) 9:25 27:20
28:17

44:7 49:11 51:20
60:12 61:1 67:25

11:1,4 12:6,10,13
26:22 27:1,5 32:23

hypothesized (1)	16:19 29:2,4 32:6	 	
35:20	42:22 67:12 75:3		K 	

68:3,8,8,9,10,11,12

33:2,14,18 34:3,23	 	

individuals (2) 17:20

k (1) 63:15

68:13,18 80:20,22

36:23 37:6,12,21,24

 	I 	

40:17

kathy (3) 2:4 3:7,14

81:2,3
grades (6) 27:15,16
31:23 34:10 53:12
80:19
grading (1) 37:3
graduation (2) 76:10
76:14
grants (1) 63:16
graphs (1) 46:12
great (14) 34:23
36:23 37:12 49:19
59:18 61:7,16,16
62:6 64:2,24 70:20
71:20 72:24
greater (1) 74:23
greatly (2) 4:22,22
green (8) 41:10 68:17
68:25 69:1,1,4,9,17
gross (1) 40:2
group (6) 32:5 40:25
41:5 45:9 46:16
71:21
grow (2) 30:17,18
growing (1) 53:22
growth (26) 14:10
16:9,20 17:21 24:9
24:11 25:24 26:3
27:7,13,21,23 28:3
28:7,15 29:15 35:3
37:17 44:6,6,19

38:3,7,16 39:7,13	idea (2) 52:23 60:14
40:5,10 49:19 51:5	identical (1) 79:24
51:10,16 52:3	identify (1) 14:12
56:14 57:4,9,13,16	ignored (2) 74:19,20
57:23 59:4,8 60:8	im (31) 3:3,4 4:17,24
60:18,20,25 61:16	4:24 5:8 10:23
62:5 63:8 64:2,9,19	36:23 39:10 41:25
64:24 65:14 66:5	43:7 45:15 47:6
66:10,20,24 67:2,8	51:8 54:5 61:5,13
67:11,14,18 68:2,7	64:17 71:15,15,16
69:3,10,18 70:3,23	71:18 72:11 73:21
73:8,23 75:13,16	73:25 75:17,21
77:4,13 81:7	78:4,10 79:1 80:1
held (1) 4:6	immediately (1) 74:5
help (9) 34:20,21 35:1	immigration (1) 80:5
36:2 61:10 80:8,10	impact (4) 35:10 55:2
80:23,25	78:7 80:23
helping (1) 3:17	impacted (1) 36:6
heres (7) 8:22 13:25	implement (3) 37:20
14:3 44:22 50:7	38:4,6
71:7 74:3	implementation (5)
high (16) 28:10 48:5	13:4 24:12 28:16
53:25 54:4,6 61:19	40:13 79:12
61:20,21,21,22	implemented (3)
72:14,14 73:1,15	13:12 17:6 19:7
78:17,18	implementing (1)
higher (13) 35:17,18	14:20
35:18 48:8,20,23	important (15) 9:11
54:15 71:25 73:4	25:22 27:10 28:1
73:10,18 76:11	29:3,4,6 41:6 43:20

influence (2) 70:11
74:6
influenced (1) 74:4
influences (1) 74:3
information (19) 21:1
25:20 35:10 37:2,8
42:24 59:25 60:9
61:15,23 62:4 68:5
68:11,13,14,16
69:22 79:25 80:23
informing (1) 18:12
initial (1) 21:8
input (18) 3:22,23,23
4:13 7:7 8:2 10:21
10:22 11:1 18:22
18:23 19:12 21:16
66:25 67:4,16,17,21
instance (1) 22:4
instruction (1) 14:10
instructional (18)
4:18 11:8 12:23
13:1 14:13,22 15:8
15:11,24 16:14
17:19,20 61:20
64:14,16 65:17
78:17,19
instructionally (1)
62:20
instructionals (1)
69:23

keep (5) 23:20,22
28:6,15 43:2
keeping (1) 17:16
kenny (2) 2:7 67:25
kept (1) 28:25
kid (1) 48:2
kids (3) 34:20 54:14
68:24
kind (8) 3:19 20:23
21:2 39:9 40:13
60:15,20 73:19
kinds (6) 20:17 24:16
26:7 35:16 63:5,22
know (75) 8:11,25
9:12 10:20 11:11
11:12 13:21 14:1
15:14 21:13 22:11
22:22 25:18 30:8
30:12,18,21 31:19
32:10 34:25 37:4
38:12 39:3,6,16
40:5 41:9 44:3,15
45:11 50:6,16 51:1
51:2,2,5 53:10,11
54:3 56:3,14,16,17
58:17 59:24,25
60:5,6,10,12,19,22
60:22,23,23,25 61:2
61:3,11,12,14 62:1
64:12,13 66:8,21,22

Page 87

69:12 70:19 71:22	longer (2) 17:24 18:7	60:19 61:5 77:17	model (29) 28:10,13	newly (1) 18:8
73:5 77:20 78:2,9	look (49) 9:12 11:7,10	meaning (2) 43:22	28:18 29:2,17 30:7	neysa (2) 82:6,18
80:9	12:7 14:5,17 17:12	46:15	31:5,23,25 33:5	nice (2) 3:16 50:15
knowing (2) 60:21	17:22 20:5 27:12	means (13) 20:11	35:2,5 36:24 37:9,9	nonclassroom (3)
74:22	27:25 28:23 29:18	22:7 27:25 28:20	37:13,17,21 40:2	16:14 56:7,10
knowledge (2) 79:6,7	30:19 31:4,6,24	31:23 34:1 37:17	42:17,20,21 48:25	notice (1) 70:6
knows (2) 23:16	32:7,17 33:3 34:4	41:24 42:14 46:14	49:10,12 65:17,21	number (13) 7:17
74:21	35:16,22 41:6	46:23 51:13 80:16	69:22 78:6	18:25 31:15 35:17
 		45:13,23,25 46:20	measure (6) 16:22	models (9) 28:20,21	35:18 36:14 42:9
 	L		48:4 52:1,3 58:12	63:16 65:3,3 74:12	28:23,24 29:7,18	44:24 52:9 53:22

l2 (1) 63:15
l314 (1) 58:18
laforte (4) 2:6 71:15
71:15 73:9
language (4) 9:20
35:15 80:3,16

58:19 60:3,16	74:15	49:2,5 79:21	54:1,15 73:3
62:24 65:24 68:7	measured (2) 16:9,10	modifications (2)	numbered (2) 11:7
68:18 69:17,19	measurement (3)	21:24 22:1	62:9
70:4,5,17 79:11	24:11 45:1 75:11	monday (1) 1:18	numbers (5) 25:9
80:7,17,19 81:3	measures (6) 14:21	money (2) 78:10,12	40:22,23,23,24
looking (20) 8:2	16:16,20 28:7	monitor (6) 9:5 13:7	 	

languages (1) 63:17
large (1) 54:2
largely (1) 50:10
law (22) 5:10 11:4,12
13:20,24 14:16,21
16:5 18:3 20:21
21:24 23:2,11 24:1
25:12 27:5,6,17
47:21 55:10 57:9
65:1
leaders (1) 15:16
leadership (1) 14:24

14:15 16:5 18:1	55:22 74:17	13:25 22:16,18
23:24 29:21 33:2,5	measuring (1) 24:8	23:5
40:22 46:20 50:7	meet (4) 15:22 18:7	monitoring (7) 13:3
51:8,19 53:1,5	71:24 72:4	14:1 20:4 22:14,18
59:24 60:1 68:6	meeting (9) 4:5,5,17	22:24 23:5
69:16 70:12	4:19 9:25 50:14	months (1) 9:10
looks (3) 29:25 30:2	51:24 81:12 82:8	move (3) 52:4,5 69:3
54:13	meetings (2) 4:14	moving (3) 23:20
lot (27) 5:4 7:16 9:14	43:5	44:8 54:16
9:16 15:16 17:22	meets (4) 13:17 14:18	multiple (5) 18:15,19
25:4 29:9,13 32:3,4	15:7 76:12	49:2 54:20 74:12
34:13,24 40:8,9	member (31) 12:4	 	

 	O 	
observation (1) 18:5
occur (1) 55:10
office (2) 32:15 79:6
okay (108) 7:6,7 8:23
11:4,20 12:6,13,14
12:15 20:5,10
23:23 27:3 29:19
29:21 31:3,14 34:1
35:5 37:6 39:8
40:10,12,19 41:12
41:20 44:22,24,25

learned (1) 74:10

49:3 50:4 52:7	26:20,24 27:4

 	N 	

45:2,13,14,20,22,25

learners (1) 35:15
learning (12) 14:10
16:8,20 24:11
25:24 26:3 28:7
32:19 42:19 61:19
68:10 77:23
leave (2) 11:25 60:6
lee (1) 4:7
legal (1) 15:7
legislative (1) 9:5

59:25 63:1 70:9,21	32:21,25 34:19
72:23 75:5 76:18	36:20 37:1,10 49:9
79:16 80:3	51:18 57:17 59:3,6
loudly (1) 6:9	63:6,24 64:7,12,21
low (5) 34:11 51:6	65:10 66:3,8,17,21
54:1 76:21 81:2	67:1,6 69:16,24
lower (4) 26:19 33:25	70:21 78:23
35:19 51:25	members (5) 2:11
lowest (2) 26:4 43:25	22:4,5 29:10 35:20
lows (1) 36:14	mentioned (3) 8:1

name (4) 7:9,10,13	46:5,6,7,10,12,14
74:9	46:25 47:10,11,19
near (1) 20:14	48:4,6 50:15 51:9
necessarily (3) 39:11	51:16 52:3,7 53:22
40:2 61:5	54:25 57:7,19 58:4
necessary (1) 26:12	58:9,24 59:12,12,14
necessity (1) 36:9	59:18 60:3 61:1,2,9
need (62) 5:22,25	61:11 62:1,10,15,22
9:20 11:11 12:18	64:5,9 66:24 67:8
13:9,9,11,12 14:3	67:12,25 68:2,7,12

level (23) 16:23 17:14	 	

18:22 20:1

14:15,17,19,19,25	68:22 69:3,19,24

25:5,16,23 28:11

 	M 	   met (4) 14:16 26:13

15:5,12,14,19,25	70:22,25 71:7,13,16

30:4 44:7 48:10,18	m (2) 1:19,19
48:23 51:10,21	maam (7) 33:10
53:7 62:14,23	34:18 49:8 57:16
68:11 70:15,21	63:5 64:11 69:15
71:25 74:4 80:20	mainstream (3) 76:20
80:22	76:22 77:2
levels (7) 14:18 16:10	major (1) 15:5
25:6 34:5,7,8 80:16	making (12) 7:1,9,12
life (1) 80:9	11:13,23 15:20
line (1) 46:5	23:24 49:3 61:18
link (1) 69:23	61:22 69:8,9
list (2) 32:15 40:13	manual (2) 66:4,6
listen (2) 7:18 10:14	march (2) 9:3 82:15
little (4) 4:11 31:19	materials (1) 74:10
43:20 62:12	math (5) 36:21 69:25
living (1) 79:8	70:5,8 80:16
loans (1) 75:1	mathematics (3)
local (7) 16:16,18	53:11,14 70:12
17:13 27:22 62:14	matter (5) 32:9 59:14
62:17 65:8	60:12,13 63:4
locally (1) 63:11	mean (11) 6:4 30:14
long (3) 5:5 71:9	40:24 41:22 50:15
77:15	50:19 51:6 54:7

26:15 55:7
method (1) 18:25
methods (4) 64:13,14
64:17 66:19
middle (1) 21:8
minchew (12) 2:7
5:18 10:13,17,23,25
11:3 12:9 39:10
40:9 73:25 75:15
mind (3) 28:6,15 43:2
mindful (1) 29:15
minorities (1) 80:3
minority (1) 76:13
minus (9) 41:21,22
42:2,11 45:3,8,12
45:13,14
minute (1) 4:16
minutes (3) 50:14
70:25 71:8
miracle (1) 78:16
misleading (1) 74:11
mobility (2) 35:24
69:6

16:22,23 17:12,14	71:19,20 72:3,6
18:23 19:7 21:1	73:5,25 75:17 76:6
22:10 23:17 24:2	77:5,14,14,15 78:4
25:4,8,17,18,20	79:17,19,21 81:8,11
26:18 29:7,18	once (4) 21:9 22:21
36:18 42:21 43:14	49:23 59:16
44:13,17 47:13	ones (7) 12:8 25:19
51:2 55:24 56:5	25:19 27:17,24
61:11,12,17 63:16	35:3 74:14
64:12,17 70:19	online (7) 3:23 4:3,12
73:22 76:4,7,18	5:20 7:23 9:9 40:8
77:9,20,21 81:3	onsite (2) 22:23 23:4
needing (2) 18:21	opportunity (2) 4:12
19:1	10:22
needs (14) 8:5 14:5	opposed (3) 32:21
15:3,22 18:18	34:8 54:20
19:22 24:23 25:19	opposite (1) 47:5
32:10 44:20 57:24	option (3) 10:21
70:18 77:18 80:6	17:18 62:19
negative (2) 25:7	options (5) 19:11
40:23	21:17 44:21,22
new (9) 22:5,5,8,9,14	65:2
22:16 59:1,2,10	order (2) 11:14 37:3

Page 88

org (3) 5:23 69:18,20	percentage (10)	39:16 41:22,22	produces (2) 75:4		R 	
orient (1) 5:10	15:24,25 16:12,14	42:2,11 60:6	78:22	race (1) 36:9
orlando (1) 4:10	33:22 41:8 44:19	polls (3) 74:8,8,11	professional (11)	raise (1) 12:11
outcomes (2) 16:16	51:20 52:9 62:25	poor (1) 78:9	14:12,23 18:17	ran (2) 35:10,12
65:25	percentages (1) 45:23	population (3) 42:16	19:15,19 22:19	range (3) 42:14 47:8
outperformed (1)	percentile (4) 51:21	55:14,14	49:4 58:6,7 78:12	54:1
40:25	51:23,25 77:3	portion (1) 6:18	82:6	rank (1) 43:23
overall (8) 16:12	percetile (1) 76:20	position (1) 15:15	professionalism (1)	rate (6) 41:21 51:24
40:24 41:3,21	perform (3) 30:13	positive (5) 8:10 25:7	61:15	72:4 76:10,14
53:14 55:8,17	48:17 77:24	40:23,23 51:12	proficiency (3) 65:3,6	79:23
71:23	performance (54)	positively (1) 36:7	65:12	rated (1) 53:13
overview (4) 5:7,8,11	14:18,22 16:3,4,6,7	possibilities (1) 47:8	program (3) 3:13	rates (1) 76:21
16:4	16:20 17:4 23:25	possible (1) 5:18	74:2,6	ratings (1) 55:2
 		24:4 25:5,6,14,16	post (1) 4:2	programs (1) 3:5	ratio (3) 49:1,7 70:5
 	P		25:23,25 26:4,7,11	posted (2) 4:3 5:20	progress (2) 14:7	react (1) 9:15

p (2) 1:19,19
page (10) 5:14,24
20:6 21:6,8 23:10
23:10 62:9,11
69:21
pages (3) 20:13 21:7
62:9
palm (1) 37:16
panama (1) 4:8
paragraph (4) 8:9,15
17:22 62:12
parent (1) 29:11
parental (5) 18:22,23
67:15,17,21
parents (1) 80:5
part (19) 4:18 5:7 6:2
6:6,16,24 16:6
19:18 20:20 23:21
25:1 26:11 47:14
48:12 50:13 55:1
57:9 62:8 67:13
participation (1) 7:7
particular (2) 12:19
74:6
parties (2) 82:11,12
parts (1) 5:3
pass (3) 76:19 77:10
80:22
pattern (1) 35:3

26:18,19 27:6	poverty (2) 36:15,16	81:1
29:25 30:9,19	power (2) 4:24 5:18	promised (1) 22:13
38:15 39:5 41:8	practice (4) 14:22	proper (1) 39:4
43:10,12,12,15	15:8 58:6,7	properly (1) 18:2
46:22,23 47:12	practices (3) 15:4,10	propose (1) 43:8
48:5 53:1,2 55:1,6	18:6	proposed (5) 9:19
55:13,23 56:5,13	predicting (1) 36:8	27:8 28:18 44:5
58:2,3,12 62:13	predictions (2) 40:18	56:7
67:11 70:2 74:22	53:24	provide (9) 4:12 14:5
75:4,12 77:2	predictor (3) 35:7	18:10 20:17 21:3
performances (2)	36:11,13	24:2,23 27:24 42:6
43:9 55:6	prek (1) 56:12	provided (5) 19:14
performed (1) 41:4	prepared (1) 78:2	21:21 24:13 63:21
performing (1) 48:5	preschool (1) 56:11	77:19
performs (1) 40:21	presence (1) 78:19	providing (2) 4:20
person (4) 13:17	present (2) 2:2 39:24	74:21
23:12 64:22 71:8	presentation (2) 5:7	psychologist (1)
personnel (11) 11:8	8:13	65:19
12:23 13:1 14:13	presidential (1) 42:10	public (2) 6:16 9:4
15:11,24 16:15	pretty (4) 60:2 71:20	publish (1) 9:23
17:19 18:9,12	71:23 73:1	purpose (1) 25:15
19:13	previous (2) 31:13	purposes (3) 17:21
persons (1) 45:4	73:16	20:9,10
phonetically (1) 7:14	primary (3) 56:11	put (17) 5:23,24
physical (2) 32:21	57:5,15	11:12 13:20 23:16
63:16	principal (1) 17:17	25:3 27:5,6 35:3,9
physics (1) 63:9	principals (2) 15:4	40:12 41:1,3 47:6
pieces (1) 3:21	79:3	53:12 70:23 77:25
pilot (1) 66:7	prior (8) 29:25 31:10	 	

read (4) 70:7 73:25
78:14 79:18
readiness (1) 72:4
reading (6) 70:5,7,12
71:25 72:5,7
real (1) 3:18
really (19) 3:17 36:11
45:16,16,22,22
46:13 50:15 53:25
53:25 54:14 56:20
64:19 66:21 69:25
72:21 73:18 77:12
78:11
reason (2) 26:10 49:2
reasons (5) 5:9 52:13
67:23,24 70:9
received (2) 8:7 75:19
recommended (2)
37:6,7
record (9) 6:11,22
7:15,22 71:2,11,12
74:1 82:8
recording (1) 4:17
recordings (1) 4:4
recruitment (1) 15:22
red (2) 41:10 46:7
redoing (1) 9:22
reduce (1) 33:21
reduced (2) 16:13

peer (1) 19:12

place (9) 4:3 13:6	31:12 35:7 60:9

 	Q 	

17:4

people (13) 7:17 8:8
9:14,17 11:21
38:25 49:3 54:17
54:19 62:25 63:1
64:2 65:13
percent (52) 16:7,13
17:1 35:13,14,14
36:15,16 40:21
41:21 44:10,10
45:3,3,20,21,22
46:7,8,15,19,20
47:2,6,17 48:7,10
48:19 51:3,5,7,7,17
51:24 53:16,18
54:5 70:15 71:19
71:22,23,23,24,24
72:4,4,6 75:7,9
76:11 80:1,2
percentag (1) 51:13

15:6 23:17 25:12	68:22 77:19 79:22
46:4 59:21 78:6,11	probability (3) 61:21
placed (1) 23:13	61:21 78:18
places (2) 3:24 40:6	probably (2) 34:3
plan (2) 17:17,17	47:5
plans (1) 14:11	problems (1) 74:2
play (1) 38:9	procedures (9) 11:17
please (1) 7:9	12:20 15:21 18:13
plus (9) 41:21,22 42:2	18:22 19:2,4 24:12
42:10 45:2,8,12,13	40:13
45:14	proceedings (2) 3:1
point (20) 4:24 5:18	82:9
6:20 8:17 9:13	process (16) 4:14
17:13 24:10 26:22	13:3,21 20:13 21:8
49:19 52:5,21	21:11 22:17,22,24
53:24 54:8,19	23:6 24:17,19,20
56:16 57:5 60:23	47:14 66:14,15
63:25 70:20 78:15	processes (2) 19:12
points (7) 38:23	38:10

qualify (1) 39:8	reference (2) 43:22
question (23) 6:8,13	50:13
7:9 10:16,23 26:24	registered (1) 82:6
32:13 34:23 36:23	relate (1) 75:23
37:12 38:16 49:9	related (3) 15:13
50:12 57:24 59:18	56:22 69:14
61:16 64:2,25	relationship (1) 41:7
65:15 66:4 77:14	relative (2) 82:10,12
79:21 80:13	relevant (2) 31:16
questions (20) 3:10	79:25
3:12 6:7,10,10,12	reliable (1) 75:10
7:5,20 10:11 23:21	remain (1) 53:22
40:11 51:16 61:4,7	remainder (1) 28:11
61:12 64:11 70:24	remember (10) 17:1
77:16 79:20 81:10	27:13,14 28:17,25
quickly (1) 60:2	32:6 40:22 41:10
quite (5) 32:19 34:10	45:6 56:21
49:24 54:20 67:4	remind (1) 13:9
quota (1) 52:8	report (8) 23:3 31:20

Page 89
32:8,17 33:1,2 70:4	8:23 9:1,20 10:13	57:19 61:13 63:24	seconds (2) 73:8 77:4	79:8
82:7	10:18 20:7 23:17	66:4 68:25 69:11	section (22) 4:19	somebody (9) 4:8
reported (2) 38:13,17	25:11,19 26:6	72:11	11:15 12:16 13:5	6:21 7:21 29:17
reporter (5) 4:1,21	32:23 33:18 34:3	says (15) 8:15 11:17	20:7,14,20 21:2,5,7	38:19 46:23 48:23
6:7 7:2 82:6	36:24 37:1,24 38:3	12:4,5 14:21 15:9	21:15,23 22:18	54:4,8
reporting (4) 12:22	38:11,16 42:5 46:7	21:24 23:12 24:19	23:6,7,9,18 25:1	somewhat (1) 64:22
12:25,25 23:6	47:3,4,11 48:18,25	25:12,13 27:5	28:5 39:7 62:6,7	sorry (3) 71:4 73:23
reports (2) 23:4	49:20 51:10 52:7	62:13 67:20 73:11	see (41) 3:13 4:25 5:1	73:23
60:11	52:15,25 56:21	schedule (2) 43:5	6:1 7:24 8:4 9:6	sort (1) 66:6
represent (1) 74:10	57:3,4,5 58:4 59:3	55:12	16:6 20:5,7,24 21:1	sosa (2) 82:6,18
representatives (2)	59:4,8,25 60:18	scheduling (1) 75:21	21:5 24:10 29:6,13	sound (2) 15:3 76:7
29:11,11	61:4,12 63:13	school (64) 7:10 10:4	30:19 31:3,25,25	source (4) 33:18,19
require (4) 13:19	64:18 66:8 67:1,6	10:6,7,9 11:8 12:23	32:3,8,18 33:6	37:8,9
19:3,17 76:1	67:18,22 70:23,23	13:2,3,6 14:11,13	35:10,12 41:10	sources (3) 18:15,19
required (12) 13:20	72:19,23 73:13,20	14:23 15:16,18	46:13 54:7 55:10	18:20
17:21 18:3 19:9,25	rise (3) 14:8 48:23	17:16,17,18,23 18:7	58:24 60:13 62:15	southwest (1) 1:11
20:21 22:15,16	53:8	18:18 22:4,5,10,21	64:24 69:22,23	speak (3) 6:8 7:19,21
23:2 65:4 67:15	road (1) 1:11	27:22 29:1,10 30:3	70:18 71:1 72:9	speaker (2) 6:25 7:13
76:2	roll (1) 45:20	30:3,25 31:8 34:18	73:5 76:18	special (8) 18:21 19:1
requirement (6)	rolled (3) 58:20 59:16	36:19,19 39:19	seeing (1) 52:20	19:3 39:15 50:1,1
52:12 56:22 65:14	68:16	40:18 41:1,2,11,11	seen (1) 8:13	50:10,11
67:19 72:5 76:12	rolling (1) 57:23	41:14,14 43:6,13	selected (1) 17:9	specific (5) 11:13
requirements (6)	room (7) 1:12 3:16	45:4 48:11 52:10	selection (1) 74:9	13:13 15:12,15
12:22,25 13:18	4:1 7:4 11:23,24	54:6,15 58:10,19	send (1) 13:22	25:15
14:16 23:1 75:22	39:14	65:18,18,19 69:5	separate (5) 3:19	specifically (3) 15:12
requires (3) 5:10	rooms (1) 79:8	70:21 72:14,14,19	12:3,8 24:5 74:6	37:7 75:23
18:24 47:21	rosters (2) 24:18,21	73:2 80:1,2 81:1	sessions (1) 9:6	specification (1)
research (1) 15:4	round (1) 52:16	schools (8) 16:1 39:19	set (19) 21:17 22:8,24	76:15
resource (2) 14:13	rpr (1) 82:18	40:16 52:17 53:18	23:6,12 40:6 41:24	spelled (1) 7:14
31:12	rule (55) 1:5 3:9 5:3	79:2,5,13	41:25 42:12 43:14	spending (2) 78:10,11
resources (3) 20:4	5:12 6:2,6 8:4,4,11	science (1) 76:17	43:16,21 45:4,7	spent (1) 78:8
64:16 80:11	8:19 9:23 11:5,5,6	scientific (1) 61:22	50:9 53:21 56:25	split (1) 16:15
response (2) 6:15,23	11:6,7,9,12,13,14	score (39) 32:1 33:12	65:7 69:1	stagnant (1) 53:23
responsibilities (5)	12:15,16,19 13:10	33:14,15 34:1,11	sets (1) 47:19	stamp (1) 76:9
13:11 15:12 16:17	13:21 19:16,18	35:7 36:11 42:5	setting (1) 15:1	stand (1) 4:25
19:15 49:4	20:9,10 23:10,19,23	43:25,25 44:2,2	seven (1) 62:9	standard (31) 26:11
responsibility (4)	23:24 24:3,5,5,6,7	45:4,9,19 46:21,22	seventh (1) 30:14	27:6 41:7,15,17
13:7 14:23 19:17	24:16 25:1,2,13	46:25 47:25 48:14	sgic (3) 28:17 37:6	42:7,8 43:22 44:7,8
19:20	26:21 27:7,8,9 28:6	48:23 53:8,25 54:4	72:20	44:25,25 45:8
responsible (2) 15:17	28:8 31:5 43:5 55:1	54:12 68:10,11,12	sheet (5) 11:22 12:1,9	46:10,14,15 47:2,20
27:23	55:9 60:1,4 70:19	68:13,22 69:12,17	12:11 22:5	47:21 50:9 51:8
rest (1) 55:14	rulein (1) 60:1	69:25 73:4,15,18	shooting (2) 51:1,2	54:13 55:7 62:14
result (6) 40:20 43:24	rules (36) 3:19 5:11	78:2 81:1	shouldnt (1) 61:15	63:7 65:12 66:18
47:15,25 51:12	5:22 9:1,7,8,13,16	scored (4) 46:17	show (9) 5:21 9:17	70:14,19 71:20,24
55:7	9:18 10:1,3,5,12	53:15,17 54:6	22:7 31:1,20 35:25	standards (30) 15:7
results (11) 13:1	13:13,17 20:5	scores (35) 31:11	41:2 62:8 71:1	18:7 25:5,14,16,23
14:10,11 16:16	21:18 22:9,11 23:2	34:20 35:17,18,19	showing (1) 70:8	26:3,7,12 27:8 28:4
21:16 25:6,7,7	23:12,15,18 24:3	35:23 36:9,21 37:4	side (1) 5:23	43:16,21 45:7 47:5
40:14 42:1 66:1	25:21 27:6,19,25	37:11 38:22 40:15	sign (4) 11:22,25 12:9	47:24 49:23 50:21
retention (1) 15:22	28:6,12 31:4 38:9	41:3 42:14 44:23	12:11	55:2,12,17,19 56:4
review (14) 9:3 11:18	40:13 43:14 56:24	47:7,8 51:7,22	significant (1) 32:2	56:9,21,25 62:17,23
12:20 19:9,12	66:24	53:15 62:10 68:6	similar (4) 21:17	63:2 65:9
20:16 21:8,11 22:2	running (2) 34:4 46:3	69:6,25 73:10,16	42:13 45:3 48:20	stantards (1) 52:2
22:15,19 23:8			74:1,3,9,25 75:4,7	simple (1) 78:18	start (4) 7:5 44:8 46:6

24:18,20
reviewed (1) 19:25
reviewing (1) 21:12
reviews (1) 22:23
revise (2) 9:20,23
reword (1) 8:8
rgi (5) 38:12,17,18,24
39:17
rid (1) 79:15
right (58) 5:15,23

 	S 	
sample (3) 42:10,15
45:21
sampled (1) 41:24
samples (2) 42:3 74:8
sat (2) 72:20,21
satisfactory (1) 55:20
saying (14) 4:21
10:13 26:20 33:25
34:19 37:19 40:16

77:18 79:22 80:17	single (4) 30:16 39:17	71:12
scoring (2) 38:25	44:1,4	started (2) 20:19 28:8
77:3	sir (1) 69:23	starting (1) 30:15
second (18) 8:8 12:25	sitting (1) 39:11	starts (1) 34:14
20:5 23:23 24:10	siwiak (2) 2:10 78:25	state (37) 1:3 3:8 7:9
25:1 26:12 49:20	six (6) 21:23 51:6,7	9:24,25 10:18
50:8 51:9 54:13	51:17,25 71:5	16:18 19:16,18,18
55:3 57:2 60:3	sixth (2) 51:23 61:1	19:20 25:12,13
80:13,20,22 81:2	size (1) 36:12	27:17 30:13,15
secondly (1) 5:11	small (3) 74:8 75:4	33:9 34:9 37:13,20

Page 90

39:20 40:21 43:17	40:24 41:16 42:3	54:18 58:1 65:21	75:3 76:1,5,7,15,23	65:4 67:6,13 68:9
43:18 48:5 50:6	42:13 43:15 45:4,9	67:18,21 79:2	77:6,17 78:20 79:4	70:3,10,15,20,22
60:1 63:7,14,25	46:16 49:14 51:24	systems (18) 11:9,19	79:5,16 80:8	71:14 73:1,11,12,15
65:17,18 66:5	52:9,10 53:15,17,22	12:21 13:4,6,12,14	teaching (17) 13:19	73:19,23 77:5
69:22 72:2 73:10	54:1 55:6 56:6,8,15	15:2 18:4 19:23	16:19 18:21 19:1,6	78:13 79:7,14,25
82:4	56:18 57:7 58:2,4	20:4 21:13,24	29:4 49:15 50:2	80:11 81:8
stated (2) 8:19 80:6	58:20,25 59:11	22:15 24:15 63:12	55:21,21 61:6	theres (58) 4:5,6 5:21
states (2) 29:16 31:18	61:4,10 72:20,24	67:11 69:22	64:22 76:16,24	6:6,16 8:1 10:8,22
statewide (12) 16:9	73:3,11,12 74:9			77:7 78:22 80:1	11:8,22 12:3 13:13

16:10,15 17:14	75:2 76:13,19
25:14 32:3 37:17	77:18 78:1 80:8,11
38:5 39:1 64:4,8	80:14,15,18,24 81:4
66:1	81:4,6
statistical (5) 42:15	studentss (1) 35:23
42:20,21 47:7	studies (1) 81:5
48:25	study (1) 26:19
statistically (3) 32:2	stuff (1) 53:3
41:16 42:25	subject (6) 16:10
statistics (5) 45:24	30:14 31:16,21
48:24 49:7 61:18	49:16,18
72:13	subjects (1) 63:8
statute (1) 6:1	submission (3) 11:18
statutes (2) 11:16	12:20 20:13
12:17	submit (1) 12:24
stay (3) 4:13 5:5 6:22	submitting (1) 23:12
stenographically (1)	substantial (1) 74:24
82:7	substantive (6) 10:12
steps (1) 29:13	22:1,3,6,7,12
sticking (1) 38:23	success (1) 51:23
stop (1) 52:17	successful (1) 73:20
stragegies (1) 61:19	sufficient (2) 25:17
strapped (1) 76:24	75:24
strategies (3) 61:20	sufficiently (1) 75:10
78:17,18	suggested (1) 53:20
strategy (4) 61:21	suggestions (3) 52:21
74:21 78:19,21	52:22 66:2
strong (1) 36:11	summative (1) 55:2
struggling (1) 36:1	superintendent (2)
student (59) 14:10	17:19 57:10
15:13 16:8,16	supervisor (3) 65:17
17:15 23:25 24:8	67:15 75:17
24:11 25:24 26:18	supervisors (1) 65:18
28:7,15 29:7,21,22	supplement (2) 24:3
30:2,11,12,17,22	31:4
31:3,6,12 32:10,19	supplemented (1)
33:3,14,15,20,23	25:19
34:14 39:4,24	support (2) 14:9
40:20 42:19,22	15:13
43:9,10 46:24 48:8	supposed (3) 33:3
48:16,20 51:18	50:20 62:3
55:13,23 56:13	sure (20) 6:9 10:23
58:3 65:25 66:1	13:17 14:16,25
68:15 73:10 75:1,4	15:5,7,19 18:1
76:10 77:18,18	24:21,24 41:25
79:7 80:9,13	42:5 43:21 45:10
students (85) 14:22	45:16 48:25 50:3
16:3,5,7 17:4,14	64:17 72:1
18:2 24:4 25:25	surprised (1) 76:23
26:4,8 29:23 30:9	system (28) 13:8,8,11
30:12,15,23 32:4,4	13:23 14:1,4,20
33:6,12,16 34:1,2	17:25 18:11,13,16
34:12,13,17,21 35:1	19:10 20:15,22
35:13,17 36:1,15,16	22:8,14,20 25:10
38:22 39:18 40:17	37:15 40:2,3 47:15

 	T 	
take (15) 5:24 7:1,2
11:5 12:3 49:12
60:3 68:3,20 72:21
74:8 79:11 80:14
80:14 81:3
taken (4) 31:7,8,8
70:18
takes (3) 68:1 69:1
80:8
talk (7) 13:10 21:15
27:15 29:10 46:2
46:10 54:25
talked (9) 20:21
27:14 36:17 40:15
43:4 44:22,23
50:15 55:4
talking (14) 10:3 26:8
29:23 30:4 33:11
33:22 36:24 39:23
52:19 56:3 57:18
69:24 70:3 73:1
talks (1) 34:24
tantamount (1) 79:9
taught (1) 49:18
teach (7) 27:14,15
49:10 56:8 61:13
63:1 76:17
teacher (47) 14:9
24:21 29:5,20,22,24
30:21,23,25,25
32:10,25 33:4,16,20
33:20,20 40:24
41:17 42:4 49:15
49:17 53:21 54:14
55:24,25 56:18
57:5,21 58:3 59:12
61:8 64:23 67:14
71:15,18 73:21
74:6,21 75:11,11
76:7 79:1,16 80:25
81:1,2
teachers (58) 2:6,7,9
2:10 9:2 15:9,18,23
17:5 24:17,19,24
27:14,15 28:22
29:3,10 30:11 32:7
34:17 35:18,19
36:14 42:23 49:10
49:13 51:20 53:11
53:13 55:20 56:11
56:12,15 57:15
58:19 60:5 61:24
63:13 64:3 67:13
67:17 74:1,5,5,13

team (2) 17:20 22:22	15:16 17:18,22
technical (6) 31:20	19:11 20:8 22:16
32:8,17 60:11 70:4	24:19 32:18 33:14
71:18	36:18 37:23 38:21
technology (2) 78:9	39:16 40:6,8,9
78:12	44:24 47:17 48:9
tell (11) 8:3 10:20	50:24 51:10,25
11:2 18:24 36:14	52:5,7 57:6,7 63:1
40:16,17 60:16	63:12,13 64:25
62:20 73:19 79:1	65:14 66:2,14,21,22
telling (2) 73:6,21	67:4,19 68:13,13
tells (1) 19:24	70:21 71:2 76:9,15
temporary (2) 56:3,4	76:21 78:13 81:1
ten (3) 49:15 76:10,12	thing (28) 9:11 11:11
tend (1) 35:25	12:5,19,25 14:7,17
terms (1) 70:12	21:2 23:11 24:8
test (10) 50:13,17	27:13 41:15,19
69:2,6 73:16 74:1,3	43:19 46:18 50:23
74:21,25 75:6	52:4 54:3 57:23
testing (5) 77:19,19	60:15 62:7 65:4
77:22 78:5 81:4	68:12 72:23 73:13
tests (10) 63:22,23	73:20 74:3 78:13
74:11,12,15,17,18	things (79) 5:6,13,16
74:23 77:21,24	8:4,6,10,20 9:5
text (6) 8:24 9:1,7,13	12:18 13:15,16
9:16,23	14:3,5,15 15:17
texts (3) 8:4,18,20	17:23,24 20:18
th (1) 23:11	21:22 24:1,2,15,16
thank (15) 3:15,16	24:22 25:2,4 28:2
10:25 11:3 64:10	28:16 30:8,18
75:12,13,16 77:13	31:19 32:8 34:24
78:22,25 81:7,9,11	35:24,25 36:3,7,8
81:11	36:10,16,18 38:10
thats (109) 3:5 5:1,8	38:18 39:15 40:12
5:16 6:16 8:10 9:11	40:19 41:9 42:6,18
11:9 17:2 19:18,22	43:2 45:18 49:21
20:10 21:2,17 22:5	50:18,19 53:4
22:7 23:5,18,23	54:21 59:21 60:14
24:4 26:8 27:8,10	61:7 63:1,2,9,11,18
29:6 30:10 31:16	65:13,16,24 67:3,20
31:25 32:13 33:18	69:3,4,21 70:3,4,6
33:23 34:15,23	72:19 74:4,19
36:5,23,24 37:12,20	78:16
38:16,17,21 39:8,13	think (33) 8:5,8,14,15
41:6,14 42:1,4,14	9:12 10:14 12:14
43:17 44:14,20	38:18 43:20 45:11
45:22 46:6,15,22	45:18 46:1,18
47:14 49:19 50:2	47:23 50:24 52:19
50:13,23 51:9,18	59:19,23,23 60:25
52:11 53:4,19,23	61:22,23 62:20
54:10 55:24 56:22	64:25 65:16 69:11
57:9,12 58:14 59:4	70:18 71:10 72:18
59:8,25 60:11	77:5 78:11,23
61:10,13,23 62:6,10	79:23
62:23 64:2,8,9,9,24	thinking (2) 62:22

Page 91

69:8	45:1,8,8,13 46:15	video (1) 4:17	75:1	yellow (1) 46:8
third (8) 13:3 50:8	47:4,11,19,23 55:5	voi (1) 79:10	wont (1) 10:8	yield (3) 61:20,22
55:3 58:15 62:11	68:15 71:2,5,8 73:4	vote (2) 41:19,23	words (2) 5:14 76:4	78:17
62:19 72:7 81:3	79:5,21,24	voted (1) 41:21	work (10) 3:20 37:11	 	

thirty (1) 77:4	type (1) 29:7	voters (3) 41:18,20,25	43:1 44:3 46:4

 	Z 	

thought (2) 54:10,11	types (2) 8:1 28:20	voting (1) 41:19	59:22,22 63:4	z (2) 61:5 62:3
three (32) 5:3 6:16,24	typical (10) 41:11			71:18 73:19	zero (10) 36:15 40:20

14:21 15:5 16:13	46:7,22,23 47:2,8

 	W 	

worked (2) 32:14	40:23 41:4 46:5

20:20 21:2 30:5	48:5 52:25 53:2,16
34:5,9 41:22,22	typically (3) 40:20

walk (3) 4:15 7:1 27:1
want (49) 5:13 6:13

36:13
working (6) 28:8

48:15 53:8 54:2,3
59:3

42:2,11 45:12	41:23 51:11
55:23 56:13,19	 	

6:19,21 8:10,12,13
10:2,18 11:11 17:9

44:11,13,18 61:24	 	
62:2		0 	

57:18,18,19 58:3,4

 	U 	

21:25 38:21 43:20

works (3) 29:17

0 (1) 37:24

58:11,20 59:11,17
71:2,5 73:15 75:6

understand (6) 6:1,4	43:21,23 44:14
10:2,16,23 30:11	47:23 48:23,25

59:19 62:20
workshop (6) 1:5

00 (1) 1:19
030 (1) 11:7

threshold (2) 26:4,14

understands (1) 6:2	49:19 50:4,6,25

3:10,18 4:9 5:3 9:9	 	

thrown (1) 26:14
thursday (1) 4:9
tie (1) 80:17
time (19) 4:6 6:21 7:8
7:16,18 8:17 11:5
17:13 23:21 35:4
44:1 58:17 69:10
69:11 70:8 72:17
73:24 75:1,8
timeframe (1) 21:14
timeline (4) 4:15 8:22
10:11 21:12
times (2) 9:14 18:25
today (15) 3:7,9,15
3:18,22,25 4:5,7
5:2,4 6:11 7:23
71:2,11 81:9
told (3) 73:9,14 78:15

uniform (4) 11:17	52:5,15,20 53:20
12:19 64:17 66:22	54:4,17,17,18,19
union (4) 2:6,7,9,10	60:17 62:6 63:22
university (1) 2:8	64:3,3,12 65:7
unknowns (1) 40:9	66:15 71:1,9,10,10
unsatisfactory (6)	71:13 72:1 75:13
26:5,17,17 47:5,22	75:25
55:8	wanted (2) 42:17
use (58) 15:20 17:9	52:3
17:16,23 18:19,21	wants (2) 7:19,21
19:7 20:15 22:9	way (40) 4:25 9:19
23:15,17 25:9	11:23 12:12 20:8
26:18 27:18,24	20:18 22:19,25
30:6 31:13 33:9,17	23:5 25:10 27:18
33:18 34:3 37:14	29:14 35:6,21
37:18 42:17,20,21	37:18 38:25 39:3
42:25 43:11,17	39:17 41:23 43:18
44:20 45:1,23,24	44:15,18 47:17

workshops (1) 8:23
worried (2) 75:21
76:9
worry (1) 48:1
worse (3) 40:21 50:18
72:14
worst (3) 43:23 72:14
72:16
worth (1) 56:19
wouldnt (5) 35:21
39:3 52:17 63:6
80:7
write (4) 11:14 24:21
63:14 71:9
writing (1) 71:24
written (1) 5:14
wrong (3) 25:18 75:7
78:10

 	1 	 1 (1) 27:20
10 (1) 1:19
1012 (4) 11:15 12:4
12:16 55:22
1112 (3) 30:24 58:13
58:13
11th (1) 49:11
1213 (1) 58:14
12l3 (1) 59:9
1314 (3) 58:14,19
59:9
136 (1) 1:12
1415 (2) 58:10 59:8
15 (3) 49:18 70:25
80:2
18 (1) 79:5
1st (1) 23:3

tomorrow (1) 4:7	48:24 49:1,22 50:9	50:11,22 52:11	 	

tonight (1) 50:16	52:2,8,9,15 53:6,23	53:4 54:16,23

 	X 	

 	2 	

top (3) 14:8 44:10	55:9,12,18 57:17	57:14,24 59:18	x (3) 29:20 61:5 62:3	2 (2) 37:24 71:25
46:9	58:1,2 65:2,2,25	60:16 62:19 65:7			20 (3) 53:15,18 75:9

traditional (1) 66:19

66:6,10,13,15 67:24	65:20,20 66:17,18

 	Y 	

2000 (1) 75:20

trained (1) 63:14	68:5	76:11	y (2) 61:5 62:3	2011 (3) 27:21 28:17
training (7) 18:10	useful (1) 39:4	ways (2) 47:23 54:22	year (65) 10:4,6,7,9	78:5
76:1,2,5,5,8,16	users (1) 44:3	weapon (1) 79:15	10:10 17:23 18:8	20112012 (1) 78:5
transcript (5) 4:2,16	uses (1) 26:10	web (1) 5:21	19:21 25:10 27:9	2012 (2) 67:24 68:1
76:9 80:7 82:8	usual (1) 53:17	webcast (1) 4:17	30:3,3,3,24 31:13	2013 (2) 1:18 82:15
transitions (1) 39:18			webinar (3) 4:3 7:12	32:1 33:12,13,17	201314 (1) 10:7
trend (1) 59:11	V	9:1	34:10 35:8 38:7	2014 (4) 37:23 38:1,8

trickier (1) 43:20
tried (2) 55:9 56:24
true (1) 82:8
try (5) 6:23 20:23
22:18 62:23 65:11
trying (9) 5:12 6:1,20
28:1 42:17 43:16
43:19 44:15 73:18
turn (3) 35:21 41:20
62:21
turned (2) 35:10
66:10
turns (2) 48:19 49:23
two (36) 3:18 4:3 6:2
12:3,7 18:8 20:13
20:14,18 25:21
26:7 30:5 31:10
34:5,9 44:10,10,25

valid (3) 62:20 75:10
78:5
valleado (5) 2:5 3:2,3
12:7,12
value (9) 28:10,13,20
34:11 40:14,20
43:24 53:14 77:11
variability (2) 50:17
51:7
variable (2) 73:17
76:24
variables (3) 40:8
73:14 74:7
variance (1) 50:12
variety (1) 67:4
verification (2) 24:25
66:14
vicky (2) 2:6 71:15

website (7) 5:22 7:24
20:17 38:21 65:22
71:9,12
week (2) 4:4 9:1
weeks (1) 3:23
welcome (3) 3:3,12
11:21
went (6) 21:4 29:9
35:9 65:1 68:8,9
weve (2) 47:23 61:10
whats (7) 19:24 20:3
21:11,17 63:24,25
65:20
widely (1) 74:19
williams (1) 79:17
wnat (1) 60:1
wonder (1) 10:18
wondering (2) 67:7

43:6,7,22 44:4,7,8,9	38:9
44:9 48:1,3,5,16	2015 (1) 57:13
50:8,8,8,8,23 53:10	20l3 (1) 25:10
53:11,14 55:12	20l314 (1) 43:7
58:7,8,10,15,19,22	22 (1) 1:12
59:1,10,11,15 60:9	25 (2) 1:18 75:7
60:10,17 66:12	 	
68:4 69:5,6,6,7		3 	
72:6,22 75:9	30 (1) 73:8
years (29) 3:23 16:13	304 (1) 72:25
20:18 22:21 23:5	308 (1) 72:24
31:10 49:15,18	309 (1) 72:24
55:23 56:13,19	33rd (1) 76:20
57:18,18,19 58:3,4	34 (4) 11:15 12:4,16
58:10,11,20 59:16	55:22
59:17 68:9,15 75:6	35 (1) 51:24
76:10,12 79:16,22	3501 (1) 1:11
79:24	38 (1) 73:11

Page 92

 	4 	 4 (1) 1:19
40 (2) 16:13 73:12
411 (2) 23:23 24:7
47 (1) 42:1

 	5 	 50th (1) 77:3
52 (2) 42:1 71:19
57 (1) 75:9
5th (4) 67:25 68:8,9
68:13

 	6 	 6 (1) 1:19
60 (2) 51:3,5
64 (1) 80:1
67 (1) 76:11
68 (7) 45:3,3,20,21
46:7 47:17 48:19
6a5 (3) 11:7 23:23
24:7
6th (4) 68:3,8,11,12

 	7 	 7 (3) 62:11 71:5,7
71 (1) 71:23
75 (2) 71:22,22
7th (3) 60:12 68:9,18

 	8 	 84 (1) 71:24

 	9 	 9 (1) 23:9
90 (1) 71:24
92 (1) 72:4
95 (11) 45:21 46:8,15
46:19,19 47:1,6
48:7,10 54:5 70:14
99 (2) 51:21 72:3