1			

2			

3			

4

5	
RULE DEVELOPMENT WORKSHOP PUBLIC INPUT ON DRAFT RULE TEXT		

6

7

8	
Rule 6A-5.030:	District Instructional Personnel and School Administrator Evaluation Systems

Rule 6A-5.0411:	Calculations of Student Learning Growth for Use in School Personnel Evaluations		

9	

10			

11			

12	
DATE:	THURSDAY, FEBRUARY 28, 2013		

13	
TIME:	4:05 p.m. - 7:05 p.m.		

14

15	
LOCATION:	VALENCIA COMMUNITY COLLEGE
CRIMINAL JUSTICE BUILDING, ROOM
8600 VALENCIA COLLEGE LANE	

152	
ORLANDO, FLORIDA	32825
16
HOSTED BY:	KATHRYN S. HEBDA
17	DEPUTY CHANCELLOR, EDUCATOR QUALITY
FLORIDA DEPARTMENT OF EDUCATION
18	325 WEST GAINES STREET, SUITE 1502
TALLAHASSEE, FLORIDA	32399-0400
19	(850) 245-0509

20	REPORTED BY:	SANDRA DIANE EVANS, REGISTERED
PROFESSIONAL REPORTER AND NOTARY
21	PUBLIC, STATE OF FLORIDA AT LARGE

22

23

24

25

1	Transcript of the Rule Development

2	Workshop, Public Input on Draft Rule Text, Rule

3	6A-5.030 and Rule 6A-5.0411, held at the Criminal

4	Justice Building, Valencia Community College, 8600

5	Valencia College Lane, Room 152, Orlando, Florida, on

6	THURSDAY, FEBRUARY 28, 2013 beginning at 4:05 p.m.,

7	before SANDRA DIANE EVANS, Registered Professional

8	Reporter and Notary Public, State of Florida at Large.

9	- - - - -

10	MS. HEBDA:	Good evening everybody, or

11	afternoon.

12	AUDIENCE MEMBERS:	Hello.

13	MS. HEBDA:	Thanks for being here.

14	My name is Kathy Hebda.	I'm with the

15	Department of Education, and I'm in the Division

16	of Educator Quality, and I'm here to welcome you

17	and work with you through the rule workshops for

18	our two administrative rules that we have out now

19	for input.

20	It's about five after four.	So -- and

21	I know some people are probably stuck in traffic,

22	but we'll catch them up as they get here and make

23	sure we have time for everybody's comments.

24	I have to say I'm so happy to see this

25	many of you here.	And that's really great, and
10	

1	I appreciate you taking the time and whatever

2	else it took for you to make it all the way here

3	to the workshop today.

4	Everybody should have three handouts.

5	There should be a copy of the PowerPoint, and

6	you should have two separate rule documents.

7	They look identical from far away, but one is

8	labeled 6A-5.030 and one is labeled 6A-5.0411.

9	If you're just coming in, there is a

10	sign-in sheet here.	And if you think you might

11	want to make a comment later, we have speaker

12	cards, too, we ask you to fill out.	If you

13	haven't already filled one out, you can later.

14	We'll just get it from you at the time you speak.

15	So does everybody have three handouts?

16	Can everybody hear me if I stand here?

17	There's not a clicker for the PowerPoint, so I

18	need to change and use the keyboard, so I'll try

19	to project.

20	If you're just coming in, we have a

21	sign-in sheet and we have handouts right up here

22	in the front.

23	Okay.	We're going to cover a couple

24	things today.	We have two rules for your

25	consideration.	And our agenda and how we'll walk
1	

1	through this over the next couple hours, as you

2	can see in PowerPoint, we have three parts.

3	First, we're going to do an overview

4	presentation of what the law says we're supposed

5	to make rules on, do you know why we are making

6	these rules.	And you always have to have

7	specific rule-making authority for a state agency

8	to make an administrative rule, so we want to

9	orient you to that so you can know what the

10	overall content of the rule is supposed to cover.

11	And, then, also included in the

12	overview presentation is what we're -- what we're

13	intending to say in the actual text.

14	And during the presentation and after

15	the presentation, we're going to have a section

16	that's just on clarifying questions.	You don't

17	have to make an official comment.	You don't have

18	to give your name or anything.	You can just ask

19	questions about what it is that you're seeing,

20	what does the text mean, what did you mean when

21	you said that; all those kinds of things.	And

22	then we'll finish that and then we'll go into

23	public comment.	And when folks make public

24	comments, we'll ask that you fill out a speaker

25	card with the spelling of your name.
1	

1	We do have a court reporter here so

2	that she can make a transcript of the whole

3	meeting.	And then we post those transcripts

4	on-line, just like we posted the two webinar

5	recordings on-line from two weeks ago when we

6	had those rule workshops, so everybody can hear

7	what everybody is asking and see if their

8	questions got asked and what the answers were

9	that couldn't attend; all of those kind of

10	things.	Plus, it also helps people think about

11	what other people might be suggesting for

12	improvements to the rule, and then it may spur

13	your thinking on it as well to give us other

14	suggestions.

15	And when we make the public comments,

16	at that point we won't be doing Q&A.	No one is

17	going to debate your comments.	Your comments

18	will just stand as they are.	And depending on

19	the number of people we have who want to make

20	comments and how much time we have left at that

21	point, we may have to time people just to make

22	sure everybody gets to say something.

23	You can send us -- you can write more

24	comments down on your sheet.	You can send us

25	more comments on-line.	I'll show you the web
1	

1	address where we're going to keep that up for

2	another month.	There's plenty of opportunities

3	for you to make comments even if we don't get

4	through everything you want to say today in the

5	interest of time.

6	Those are our three parts.

7	So, I would ask, even when you're

8	doing just the question-and-answer part, if you

9	could speak loudly enough so the court reporter

10	can hear what you're saying so that we can

11	capture those questions and answers.

12	And it may be you ask a question that

13	I can't answer today, but at least we'll have it

14	so we can take it back and get you a response.

15	So, a couple more things about input

16	and participation.	Like I just said, if you

17	would please give us your name.	And on the

18	speaker card if you'll write it out, or print it

19	out, or clearly write it out so the court

20	reporter gets that accurately, that would be

21	great.	And we've already been through the time

22	part.

23	There is -- on your PowerPoint

24	handout, you'll see the website where you can go

25	on-line and give more comments and go back and
1	

1	encourage your friends to do the same thing, and

2	all of that on this very same website.

3	Come on in, folks.	We have a sign-in

4	sheet here and three handouts.

5	What about the kind of input we're

6	hoping for?	We're hoping for a couple of

7	different things.	First, the kind of input you

8	would expect.

9	As we go through the presentation and

10	as you actually look at the texts in these two

11	documents, this is proposed text.	This is not

12	what's going to the State Board of Education

13	yet.

14	So what we're hoping you'll give us

15	is editing -- take this out, don't like this,

16	add something about this subject that you didn't

17	cover, I see how you've worded this here, but I

18	think you ought to word this this way.

19	We've already gotten some suggestions

20	like that, and they're extremely helpful because

21	one of the reasons why we give you an overview

22	of what we intend to say is so that you can also

23	give us feedback on whether or not we did that

24	with these words.

25	You may look at the presentation and
1	

1	say, well, you said you wanted to say that, but

2	when I read this paragraph, that's not what it

3	means to me.	I think you ought to say it this

4	way.	And those kinds of comments are very, very

5	helpful to us because we want it to be clear.

6	So, those are the kind of things we're seeking

7	input on.

8	And let me show you the timeline that

9	we're working on right now.	In the month of

10	February, for a few more days -- and that's when

11	we're doing the workshops.	We did two webinar

12	workshops the week before last.	This is the

13	fourth -- well, the third day of workshops this

14	week.	We've done Panama City, Jacksonville,

15	Broward County, Lee County, and today.	And then

16	we're going to take a little break from doing

17	workshops for a little bit and kind of see all

18	the input we're getting both from the workshops,

19	posted transcripts and the things we're getting

20	on-line.

21	So, in March and April we're going to

22	do a few things.

23	As you know, legislative session also

24	happens in March and April.	And anything could

25	happen.	So something could happen -- a Bill
1	

1	could pass and get signed into law that affects

2	the contents of these rules.	So, we don't plan

3	on taking these rules to the State Board until at

4	least June for their consideration.

5	So, during March and April we'll

6	monitor legislative session.	We'll also look at

7	the public comment and feedback, provide some

8	responses, provide some new draft text, and also

9	run some data because some of the things that are

10	in these rules have to do with setting standards

11	for use of student growth results and

12	evaluations.	And, so, we'll need to do some

13	statewide simulations.	We're working on the

14	ones now that are in the proposed rule, but we've

15	already gotten some new suggestions from people

16	on how to do those a little differently.

17	And, so, what we would like to do in

18	the month of March is run some simulations on

19	those suggestions and see how they come out

20	statewide.

21	And some of you already know there

22	are some school districts that have already

23	taken their local data and tried to apply the

24	standards that are proposed here and see how they

25	come out.	And all that is, just not important,
1	

1	but it's vital for us to know how the rules will

2	actually impact individual districts because

3	we're making these rules to cover the whole

4	state.	So knowing how they impact individual

5	districts is also an important part of the

6	process.

7	So, then, if the timeline stays as it

8	is and the State Board looks at the rules in

9	June, and maybe even July, they would not go into

10	effect until the following school year.	So even

11	though these are about educator evaluations and

12	they might even get passed by the State Board of

13	Education in June, which is during the current

14	school year, they wouldn't affect this current

15	school year's evaluation.	It would only be next

16	year at the earliest, okay?

17	Any questions about the timeline, or

18	inputs for today's agenda?

19	Yes, ma'am.

20	AUDIENCE MEMBER:	What is the deadline for

21	people to submit comments to you on-line?

22	MS. HEBDA:	We're going to leave that open

23	through at least the end of March, but probably

24	through April, too, because of the fact that

25	we're looking at legislative sessions at that
1	

1	time.	The earliest it would close would be the

2	end of March, but I anticipate it will stay

3	probably through April.

4	Any other questions?

5	All right, let's go into -- we'll do

6	one rule at a time.

7	The first one we're going to

8	look at is 6A-5.030, District Instructional

9	Personnel and School Administrator Evaluation

10	Systems.

11	And you notice that we're going to

12	do two things.

13	Remember I said we're going to look

14	at what the law says we have to make rules on.

15	And, then, we're actually going to look at the

16	content of that rule and what we intend to put in

17	the rule so when you examine the text, you can

18	tell us whether we did that or not.

19	The first thing we have to know is

20	that rule authority.	What is it we're supposed

21	to have in this rule?

22	Both of these rules are based on

23	requirements in Section 1012.34 Florida Statutes,

24	which is the section on district personnel

25	evaluation systems.
1	

1	The content of this particular rule

2	deals with actually how districts submit to the

3	Department and how the Department reviews and

4	approves district personnel evaluation systems.

5	It also has to include how districts annually

6	report data, what's contained in the

7	Commissioner's annual report that's due every

8	December 1st, and then a monitoring process for

9	how the Department will monitor district

10	implementation evaluation systems.

11	Now, prior to the changes in the

12	Student Success Act, the Department has been

13	required to review and approve instructional

14	personnel evaluation systems for school districts

15	for over a decade, but what was new in that law

16	was that we also had to approve the

17	administrator evaluation system, the school

18	administrator evaluation system and we're now

19	required to monitor.

20	So whenever a state agency makes a

21	rule for something like this -- for example,

22	this is a rule that affects between school

23	districts and the Department of Education, the

24	Department has to set forth what are we going

25	to do, what is the process and what are we going
1	

1	to be looking for to monitor, what are we going

2	to be looking for to review and approve an

3	evaluation system.

4	So we have to go back to the content

5	of what the law says has to be included in that

6	evaluation system.

7	So, if you will, look at your handout

8	for the actual rule, 6A-5.030 and let me just

9	orient you to how it's set up based on this

10	content.	And then we'll look at some of the

11	content itself.

12	Now, one thing we tried to do for

13	consistency is actually use the version of the

14	rule that's printed in the Florida Administrative

15	Weeklies so we don't have different versions

16	flying around.	The drawback to that, though, is

17	they don't have page numbers, so try to keep up

18	with me as we go through the pages.	You may even

19	want to get a pencil and you may want to number

20	them as we go through.

21	The text of the rule starts on the

22	second page.	It will be Page 2.	And the way

23	we've constructed the rule -- as you can see,

24	there's a definition section first because

25	there's some things that we needed to define just
1	

1	for the purposes of this rule.	These things can

2	have other meanings in other places, but for

3	purposes of this process, that's what they're

4	going to mean here.	And that's what the

5	definition section is for.	And that goes all the

6	way through what will be Page 3.	And then on

7	Page 4, you see somewhere near the bottom, you

8	see a second section, section two, which is

9	the submission process.	That's just how would

10	a district send us their evaluation system so

11	that we can review it and approve it.

12	That's pretty short.

13	And then you see on the next page

14	about a third of the way down, section three.

15	We're going to come back to section three because

16	section three is the meat of the rule, meaning

17	that's where all the content of the evaluation

18	system lies, in that section.	And it has to be

19	our description of the evidence that we're going

20	to look for so that a district knows what to

21	submit and we'll know what to review and approve.

22	So, section three is the longest section.

23	AUDIENCE MEMBER:	It starts with content

24	of approved evaluation systems?

25	MS. HEBDA:	That's correct.	Starts with
1	

1	content of approved evaluation systems.

2	And for those of you that were

3	watching this process last year, this is the rule

4	that was challenged and went to the

5	administrative law judge.	And they said we

6	needed to do our technical process of

7	rulemaking differently.

8	And one of the things that the

9	administrative law judge took exception to is

10	that we had the content of section three in a

11	form, in a checklist.	And we modeled it in the

12	same checklist that districts had already used

13	for Race to the Top to submit their evaluation

14	systems, and he didn't think we should have

15	content in a form; that we needed content in the

16	narrative of the rule.	So that content is not in

17	a form.	It's in the narrative of the rule.	If

18	you're watching the old process, that's the

19	change for this version, but that makes it go on

20	for a little while.

21	So, it goes on for a few pages

22	because it's double-spaced and all those sorts of

23	things.

24	And, then, if you get to what would be

25	numbered as Page 12.	Actually -- sorry, I lied
1	

1	-- Page 11.	Page 11.	Page 11 at the very top

2	says -- there's a little sentence -- the middle

3	sentence says, district documentation shall

4	include the peer assistance process.	At the very

5	top of what would be Page 11.

6	AUDIENCE MEMBER:	We're on Page 11 now?

7	MS. HEBDA:	Yes, sir.	Page 11.	There's

8	section four and section five.

9	Section four just is the initial

10	review process, which is, what are we going to do

11	when we get a district system to review, how long

12	is it going to take us to review, what timeline

13	do we have to follow?	That's what section four

14	is.

15	Section five is the approval process.

16	So in section five we define what does it mean

17	to be fully approved, what does it mean to be

18	conditionally approved, what does it mean to be

19	denied, what happens, and how long does a

20	district have, then, if their district is denied,

21	to get it back to full approval status.	All

22	those things are explained in section five.

23	And section six is something we had

24	to address, which is district modifications to

25	an evaluation system.	But the law says the
1	

1	district can modify its system at any time.

2	Yes, sir.

3	AUDIENCE MEMBER:	A question, then, on

4	this approval process.

5	MS. HEBDA:	Yes, sir.

6	AUDIENCE MEMBER:	If a district's process

7	is denied --

8	MS. HEBDA:	Yes.

9	AUDIENCE MEMBER:	-- it indicates that

10	they are given a time frame to correct

11	whatever --

12	MS. HEBDA:	That's correct.

13	AUDIENCE MEMBER:	-- what is the ultimate

14	final impact on the district if they are unable

15	to comply, or unwilling, perhaps, to comply?

16	MS. HEBDA:	In the event the district is

17	noncompliant for whatever reason, even after

18	this process has taken place, then this --

19	implementation of this law is no different than

20	implementation of any other law.

21	The Commissioner has an obligation,

22	the State Board has an obligation to ensure

23	compliance with all the school laws.

24	And, so, there is a section of law,

25	the number which escapes me, but it's in the
1	

1	one thousands, that talks about the

2	Commissioner's responsibilities.	And one of his

3	responsibilities then is to go through a series

4	of iterative processes to help the district get

5	into compliance, starting with making a report

6	to the State Board of Education and those kinds

7	of things.

8	AUDIENCE MEMBER:	There's no withholding

9	of funds?

10	MS. HEBDA:	That's the last thing you can

11	do.	That's the very last thing you can do in

12	that list.	There's a list of things that are

13	options for the Commissioner, starting with just

14	making a report to the State Board of Education.

15	AUDIENCE MEMBER:	Thank you.

16	MS. HEBDA:	That's where that ends up.	So,

17	in that case there's no special process for this

18	law.	It's just like any other law.

19	Good question.

20	Then section six, which is the bottom

21	of Page -- what will be Page 12 is district

22	modifications, as I mentioned, because the

23	district can modify its own evaluation system

24	any time it chooses, but the law says that

25	substantive or substantial revisions come back
1	

1	to the Department for review and approval.

2	So we had to define what does it mean

3	to have a substantive or a substantial revision.

4	So, that's what we did in section six.

5	An example of something that would be

6	nonsubstantive is -- well, for example, the

7	school board members names change because you

8	have new school board members so you reprint the

9	front cover.	That's nonsubstantial.

10	Yes, sir.

11	AUDIENCE MEMBER:	Would it be substantial

12	if, for example, in our county we said in order

13	to be highly effective 60 percent of the

14	observed elements must be at the innovating

15	level and --

16	MS. HEBDA:	If you had formerly set --

17	formerly --

18	AUDIENCE MEMBER:	-- we change that to 40

19	percent or 50 percent because we realized 60

20	percent was unattainable because we pulled it out

21	of the air the first time and now the data

22	suggests it be lower?

23	MS. HEBDA:	We have a list of things in the

24	rule that we have proposed for your consideration

25	that would be considered substantive changes.
1	

1	And number two on the top of the next page is the

2	scoring and weighting method.	So that, to me,

3	seems that would --

4	AUDIENCE MEMBER:	That would be a

5	substantial change?

6	MS. HEBDA:	That would be a substantial

7	change.

8	AUDIENCE MEMBER:	Thank you.

9	MS. HEBDA:	Correct.	Very good example.

10	Then, remember, I mention we have to

11	monitor.	So section seven deals with how we're

12	going to monitor.	And we've patterned -- for

13	everybody's consideration, we've patterned our

14	monitoring process for evaluation systems on the

15	same process we already use to look at district

16	professional development systems.

17	If any of you are familiar with that,

18	looking at the protocol changes we've had for

19	professional development -- we do an annual

20	report on professional development, but we

21	actually don't monitor districts on site except

22	for once every five years.	It's a five-year

23	cycle where we see every district once during a

24	five-year period, and then we do a report on that

25	cycle.	And then we look at the standards, see
1	

1	if they need to be revised, and then we start a

2	new cycle.

3	And, so, we're proposing we would do

4	the same thing here because there is annual

5	reporting of results.	The Commissioner makes an

6	annual report on evaluation systems.	So, our

7	proposal is that, in addition to that, we would

8	only come on site for a full review once every

9	five years.

10	Yes, ma'am.

11	AUDIENCE MEMBER:	I know that in the state

12	statute a teacher can be fired in three years.

13	So if the district is out of compliance and you

14	don't come for five years, where does that leave

15	the teacher?

16	MS. HEBDA:	That's a good question.	That's

17	a very good question.	Some of those things we

18	should be able to detect in the annual reporting.

19	But if you don't see where we can do that between

20	when we have an annual reporting and the

21	monitoring every five years, then that would be a

22	very good suggestion to make in the comments.

23	AUDIENCE MEMBER:	Do it every three years?

24	MS. HEBDA:	Yes, sir.

25	AUDIENCE MEMBER:	Under (a) 2, evaluator
1	

1	accuracy and inter-rater reliability where you're

2	supposed to monitor that --

3	MS. HEBDA:	Yes.

4	AUDIENCE MEMBER:	-- can you give me a clue

5	as to how you will possibly evaluate the

6	evaluator accuracy and inter-rater reliability

7	when most teachers are evaluated one time, or

8	maybe a few times by a single administrator?

9	MS. HEBDA:	Thank you.	That's a great

10	question.

11	In section seven we have two

12	subsections, if you will.	Subsection A, which

13	begins right underneath section seven, talks

14	about how the -- remember the district has to

15	do an annual review?

16	AUDIENCE MEMBER:	Uh-huh.

17	MS. HEBDA:	So subsection A deals with

18	the district's annual review of its own

19	evaluation system.

20	So, one of the things that we expect

21	is that the district is monitoring that as well

22	and checking the calibration, checking for

23	inter-rater reliability, checking to see --

24	looking across all of their district data to see

25	if there are places where the data would lead you
1	

1	to think, I need to go and investigate this and

2	see if this is really what should be going on at

3	this school.

4	And, then, when you get to subsection

5	B, one of the things we said we're going to do,

6	when we go on site to monitor, is to look at

7	those district's annual reports.	So how is the

8	district -- the district self-assessment,

9	district monitoring, we would look at that data

10	as well to make sure the district is dealing with

11	those things they find in their annual

12	self-assessment.

13	So, we're not going to do calibration

14	for people in individual districts, but we're

15	going to check to make sure the district is

16	dealing with those things when they find

17	anomalies.

18	AUDIENCE MEMBER:	Would you get that

19	through the report that the district submits

20	to you guys?

21	MS. HEBDA:	They would have to submit that

22	report to us.

23	His question was:	Would they submit

24	that report to us.

25	Yes, sir.
1	

1	AUDIENCE MEMBER:	It would seem to me the

2	district would have the very same problem as was

3	first proposed.	If the typical teacher is only

4	evaluated once or twice by his or her principal

5	during the year, the district is going to be no

6	more able to come up with inter-rater reliability

7	on the instrument that's used than anybody else

8	would.

9	MS. HEBDA:	That's a very good point.	And

10	the literature you read on observations and

11	things like that suggest that once a year is not

12	often enough for that sort of thing.

13	So, we haven't put a requirement for

14	a number of observations.	The only thing we've

15	dealt with is what the law deals with, which is

16	making sure a newly-hired teacher gets

17	evaluated twice in a year, but that may be a

18	suggestion that you think is important, or

19	somebody else thinks is important for us to deal

20	with.

21	AUDIENCE MEMBER:	But, further, for

22	inter-rater reliability on an instrument, doesn't

23	the same teacher have to be observed by two

24	different raters during the year?

25	MS. HEBDA:	There is --
1	

1	AUDIENCE MEMBER:	So, if the principal does

2	two observations of you during the year, wouldn't

3	there have to be a second person brought in to

4	observe you if you're the teacher in order to do

5	inter-rater reliability?

6	MS. HEBDA:	That's what you want to check

7	for.	There are also ways to check for

8	inter-rater reliability among groups of people.

9	There are number of ways people are getting at

10	that.	But, certainly, in the best situation, you

11	would want everybody looking at the same lesson

12	at the same time.	Some people do it with video.

13	They'll video a lesson and then through them

14	watching the video -- they'll have ten or 15

15	people watch the same video and see if they get

16	the same results.

17	AUDIENCE MEMBER:	So the State, then,

18	expects the district to have the technology to be

19	able to do that so we could demonstrate

20	inter-rater reliability.

21	MS. HEBDA:	That way or some other way.

22	Yes, ma'am.

23	AUDIENCE MEMBER:	Under evaluator

24	accuracy --

25	MS. HEBDA:	Yes.
1	

1	AUDIENCE MEMBER:	-- how is that going to

2	be monitored?

3	I represent different districts and

4	they have several different evaluation systems,

5	and in none of them is there any type of

6	objective test that the evaluator has to take to

7	show that they are, in fact, competent to

8	administer the evaluation.

9	Accuracy and inter-rater reliability,

10	as you know, are two different things.

11	MS. HEBDA:	That's right.

12	AUDIENCE MEMBER:	Inter-rater reliability,

13	you and I can agree that we see something --

14	MS. HEBDA:	And we both could be wrong.

15	AUDIENCE MEMBER:	Absolutely.

16	And, so, how is the DOE going to

17	monitor whether there is some type of a

18	substantial objective test for the evaluation

19	system so that we know that the evaluators are

20	competent to administer these instruments?

21	MS. HEBDA:	That's a great question.	We

22	haven't dealt with these words specifically in

23	here.	We've only said these are the things we

24	are going to monitor.	So if what you think is

25	we need to get down more finally grained on
1	

1	exactly what we are going to be looking for when

2	we monitor, that may be a suggestion you want to

3	make for the rule.

4	Yes, ma'am.

5	AUDIENCE MEMBER:	Consistency and fidelity

6	of implementation of assessments used to measure

7	student growth and performance.	If districts are

8	permitted to develop their own end-course

9	assessments, and that sort of thing, how on earth

10	can you ensure there's consistency and fidelity

11	of implementation of assessments for students?

12	MS. HEBDA:	That's a great question.

13	There are a couple things that we can

14	do.	We've been talking about these with school

15	districts.	Some of these go back to the -- like,

16	the item banks and the test platforms that we're

17	providing to districts to use from which the

18	items are all vetted, they all know they're on

19	the line of the common core, they're on the

20	standards for the course.	And the test platform

21	software helps the district seek assessments

22	for the right purposes.	Some assessments are

23	summated, some interim, some formative; those

24	kinds of things.

25	Secondly, we've said that we will
1	

1	help districts with guidelines and technical

2	assistance, but it provides these are the things

3	you should be doing, both as a way to build

4	assessments but also a way to administer

5	assessments that people can use for consistency

6	across districts.	It doesn't mean that

7	necessarily Levy County and Orange County would

8	have to have the exact same assessment for

9	seventh grade social studies.	They may choose

10	to.	If they both get them from the item bank and

11	use the same test properties, then likely they'll

12	be similar enough you can almost think of them

13	as two forms of the same test because they're

14	looking at the same course with the same course

15	description, same standard, same benchmark; all

16	those things.

17	However, at this point in time,

18	there's not a regulatory authority for the State

19	to actually approve the individual assessment,

20	only to monitor this process and provide

21	technical assistance.	So, we're trying to do

22	that by providing a quality tool to build the

23	assessment and by providing guidelines that

24	districts can adopt and say we're going to stick

25	to the State guidelines and on how we actually
1	

1	administer these.	And then you can have some

2	consistency across districts without us making

3	every single test anybody ever uses for any

4	course.

5	Sir.

6	AUDIENCE MEMBER:	On the second to the

7	last page, guidelines for training of evaluators,

8	number nine --

9	MS. HEBDA:	Yes.

10	AUDIENCE MEMBER:	-- are these guidelines

11	for evaluator training required because it only

12	says the districts need to consider them?

13	MS. HEBDA:	That's correct.	That's the

14	language of the law.	The law says the district

15	has to have a training program.	We have to have

16	a training program for your evaluators.	And when

17	the district develops its training program, they

18	have to consider the guidelines provided by the

19	Department.

20	AUDIENCE MEMBER:	So it's not required?

21	MS. HEBDA:	It's not required, but we put

22	them in the rules, because they pertain to those

23	things, so everybody can find them, they're all

24	in one spot and you aren't trying to hunt around

25	the web to find the rules for the guidelines.
1	

1	Yes, sir.

2	AUDIENCE MEMBER:	Just out of curiosity,

3	if it's not required, then how can you go back

4	to number seven and ensure inter-rater

5	reliability and accuracy if there's not a

6	requirement that a hundred principals are trained

7	to do this, in a sense, the same way; yet, you're

8	supposed to monitor that?	I know you may not

9	have this answer, but it's illogical to presume

10	that they are going to be consistent to

11	have inter-rater reliability when that's not a

12	requirement that they be trained.

13	MS. HEBDA:	Different folks will tell you

14	you can get to inter-rater reliability different

15	ways.

16	Again, if there are suggestions to

17	make this more specific, then we'll take those

18	suggestions.

19	Short of us telling districts exactly

20	how they have to do that, which the law doesn't

21	say we can do, right?

22	And he said he doesn't want us to do.

23	Put that on the record.

24	This is how we've gone about it so

25	far for your consideration, which is exactly
1	

1	why we're having this workshop.	If we need to

2	be more specific somewhere, if there's something

3	missing, that's what we want to know.

4	Yes, sir.

5	AUDIENCE MEMBER:	On Page 11 near the top,

6	paragraph three, a process for amending

7	evaluations.	If data from the current school

8	year are received within 90 days after the close

9	of the school year, the district is not required

10	to finalize.

11	MS. HEBDA:	That's right.

12	AUDIENCE MEMBER:	I'm a little confused by

13	that because, just as a personal note, I sat

14	down with my principal, or my assistant

15	principal, probably about a month ago to finalize

16	my summated evaluation for last year because

17	that's when the data became available at our

18	school for the student growth data to fill in the

19	student growth component.

20	Now, if I only have -- if there's only

21	90 days to amend a summated evaluation and the

22	student growth data is not available until

23	sometime next November, I'm confused as to how

24	you can include that because I would say that

25	student growth data is not available, you can't
1	

1	make that part of my evaluation, so I guess you

2	have to go with my professional practices, but

3	the law says I have to include student growth.

4	So, how do we get around that dilemma?

5	MS. HEBDA:	Well, that's a great question.

6	And the first year of implementation was

7	difficult for everybody, but the rule has to be

8	constructed the way the law is laid out.	So,

9	the law says the district has an option to

10	provide a process to amend evaluations when

11	additional data is provided.	It doesn't say it

12	has to be student growth data.	It can be

13	instructional practice data.	It can be anything

14	that's provided.	And, so, they can choose to do

15	it and they can choose how they do it, but it

16	doesn't say the evaluation has to be finalized

17	90 days, or 30 days, or 60 days, or anything

18	else.

19	The situation you experienced this

20	year is in no way anything anybody would want to

21	happen.	Of course not.

22	AUDIENCE MEMBER:	But, yet, it did.

23	MS. HEBDA:	It did.	Things do happen that

24	we don't want to happen.	Absolutely.

25	Absolutely.	But that's not the way the law
1	

1	contemplated things working.

2	AUDIENCE MEMBER:	But, according to this

3	rule, if I was to find out, say, a week later

4	that the data used to generate my student growth

5	score was wrong, I would have no recourse because

6	we're outside that window --

7	MS. HEBDA:	No, because your evaluation --

8	if your evaluation was finalized a month ago,

9	then you got 90 days.

10	AUDIENCE MEMBER:	So, it's 90 --

11	MS. HEBDA:	You can actually have longer

12	than that.

13	AUDIENCE MEMBER:	This says after the

14	close of the school year.	Then that language

15	needs to be changed.

16	AUDIENCE MEMBER:	It does say from the

17	close of the school year.

18	AUDIENCE MEMBER:	How long after 90 days

19	would be too long?

20	MS. HEBDA:	That's a great question.

21	That's a great question.

22	AUDIENCE MEMBER:	If we're not able to see

23	any of the testing or any of the results or even

24	which kids were being evaluated on, how are we

25	able to sign off on anything when there's so
1	

1	many unknowns?	Is there going to be a way that

2	we are going to see every child that was tied to

3	us and be able to see the results of their test?

4	MS. HEBDA:	Actually, there is.	Some of

5	the data that we provide -- I don't know if

6	your districts participate in roster

7	verifications.	The Department's tool (ph) had

8	its own process for roster verification.	There

9	is a process that every district goes through,

10	whether they use our tool based on the web or

11	not, for verification of rosters in survey

12	two, which means the first semester, and

13	survey three the second semester.

14	So, survey two roster verification

15	is closed.	In a few weeks survey three roster

16	verification will start.	And that's when you

17	look at your class rosters and say these are the

18	kids that are assigned to me and these are not,

19	or this one is not, or there's a student missing,

20	we need to get that corrected.	And those are --

21	I know we do roster verification for lots of

22	different reasons, but that specifically for

23	purposes of which ones will be included in your

24	student growth score if you're teaching reading,

25	language arts, mathematics for VAM or whatever
1	

1	other purpose.

2	AUDIENCE MEMBER:	But I was told I was

3	tied to 300 and something kids.	I do not have

4	300 and something kids.	So, I signed my little

5	roster sheets this year during our FTE (ph),

6	and I verified these students.	But if this is

7	going for three years' worth of students, I want

8	to see who those students were; which ones were

9	actually tied to me if that is where my VAM score

10	is coming from.

11	Are we going to have a database that

12	has all of those kids' names, then -- the ones

13	that have excessive absences, the ones that have

14	whatever other issues they have?	If this comes

15	down to our job, we need to be able to see

16	everything that is being tied to us, and right

17	now we're not.

18	MS. HEBDA:	I agree with you.

19	AUDIENCE MEMBER:	So, is there going to

20	be something like that?

21	MS. HEBDA:	We have one year so far.	Most

22	districts use '11-'12 as the first year.	This

23	is year one.

24	AUDIENCE MEMBER:	Is that something that

25	is going to be looked into?
1	

1	MS. HEBDA:	Absolutely.	Yes, we can look

2	into that.	I don't exactly know how we would

3	capture the roster verification and hold it for

4	three years, but, hearing that suggestion, we

5	can work on that.	We know we're doing it one

6	year at a time.	And once that year is verified,

7	that year is done and we go to the next school

8	year and you verify semester by semester in the

9	second school year.

10	So, I'm not sure how we would be able

11	to go back and let you see the previous year's

12	data.	I know it's still stored somewhere, but I

13	don't know that we have a process for you to go

14	back and see the year before.

15	AUDIENCE MEMBER:	Yes, because right now

16	we're just signing off on a number that was

17	given to us that nobody can explain to us how

18	that number came about.

19	MS. HEBDA:	I understand what you're

20	saying.

21	AUDIENCE MEMBER:	We're not even sure if

22	the correct kids are on that list.	Some are

23	missing, some are there.	I don't know.	The

24	total did not match up to what I had.	So, I

25	don't know what is going on.	I don't know why
1	

1	I should sign off on any of it if I don't know.

2	A lot of people are very concerned about signing

3	something unknown.

4	MS. HEBDA:	Right.	I can see why they

5	would be.

6	Yes, ma'am.

7	AUDIENCE MEMBER:	Along that same line, I

8	know we do the roster verification.	Our county

9	does that.	Although I may have verified my

10	roster in February, I have no way of knowing that

11	those are the students that were included in my

12	VAM unless you show it to me.	Just because I

13	verified in February doesn't mean there was an

14	error somewhere.

15	So, the district can verify -- or

16	maybe they can't, I don't know -- but how does

17	the teacher get to verify?	And we always don't

18	have any way to know what was the student

19	expected growth and what was the student growth

20	for each individual student.

21	MS. HEBDA:	Right, right.

22	AUDIENCE MEMBER:	Actually, those are very

23	important pieces of information.	And along with

24	what you said, when you talk about all the

25	variables in the formula here, which I carry with
1	

1	me, in that formula we don't even know what the

2	effect is.

3	I mean, if I have gifted students,

4	so the effect of being a gifted student in the

5	variable, does that mean they have a greater

6	growth they're expected to have, or a less of a

7	growth?	It doesn't even let you know that.

8	MS. HEBDA:	Can I address just one thing?

9	All of those issues are in the second rule.

10	AUDIENCE MEMBER:	I knew you were going to

11	say that.

12	MS. HEBDA:	We're going to do that.	Even

13	before we have the -- before we get to that rule,

14	one of the things that we're going to be piloting

15	soon is going to be a web tool where a teacher

16	can see his or her own students and see what

17	their expected score was and whether they met or

18	exceeded that expected score because we have that

19	data.

20	Some districts have developed their

21	own tools to provide that already to their

22	teachers, but what they can't do is see statewide

23	data.	So one of the things that our tool will be

24	able to do is let you see how that compares with

25	people who taught your same subjects statewide.
1	

1	So you can see that information, too.

2	AUDIENCE MEMBER:	The discrepancies amongst

3	counties -- I come from a small county.	We can't

4	afford to do something like that.	We don't have

5	a statistician employed to us.	There's nobody

6	that can do that.	Yet, you go to a large county,

7	they do have that kind of technology.	So, it's

8	very unfair.

9	AUDIENCE MEMBER:	To go along with that, as

10	a matter of -- the whole notion that every

11	district is functioning under its own unique set

12	of circumstances, the fact that on Page 4

13	performance level has certainly been given

14	descriptive language so that teachers must fall

15	into one of these four categories; highly

16	effective, effective, needs improvement or

17	unsatisfactory, it means different things in

18	different districts.

19	So how do I as Joe Public, looking in,

20	trying to decide where I buy my house -- look at

21	Hillsborough County that has 40 percent better

22	highly effective teachers than Pasco who has

23	than three percent highly effective, how are we

24	not disadvantaging our communities with this?

25	MS. HEBDA:	That's exactly what the second
1	

1	rule has in it, too.

2	AUDIENCE MEMBER:	Oh, the second rule.

3	MS. HEBDA:	Yes, ma'am.

4	AUDIENCE MEMBER:	Sorry.

5	MS. HEBDA:	Don't be sorry.	That's exactly

6	why we're doing the second rule.

7	Yes, sir.

8	AUDIENCE MEMBER:	I'm in this rule.

9	MS. HEBDA:	Okay, good.

10	AUDIENCE MEMBER:	Under 1 (h), it talks

11	about newly hired by the district means the first

12	year in which instructional personnel are

13	employed.	Such personnel are newly hired for

14	their first year of employment in a district

15	regardless of their prior work experience

16	elsewhere.

17	So does that mean that their prior

18	evals in the other districts, the VAM for those

19	scores elsewhere don't count?

20	MS. HEBDA:	No.	All that means is to

21	define newly hired so that you know who has to

22	have two evaluations in the first year.

23	That's what that's for.

24	AUDIENCE MEMBER:	Why couldn't they go off

25	of their previous year's evaluation to
1	

1	determine, then, that they didn't need to two

2	evaluations if they are a teacher with experience

3	and have scores from previous -- their previous

4	job?

5	MS. HEBDA:	That's a good question.	It's

6	just that the law says anybody newly hired by the

7	district has to have two evaluations in the first

8	year.	So we've defined that so everybody is on

9	notice as to who that is.

10	Now, the makeup of those two

11	evaluations could certainly vary for somebody

12	who has experience and somebody who doesn't.

13	But, you do need to take two -- you do need to

14	give someone a midpoint to look at how they're

15	doing who is newly hired by your district so

16	they have an idea of how they're faring that

17	year.

18	Yes, ma'am.

19	AUDIENCE MEMBER:	I'm not sure if this is

20	the right time, but as far as the VAM models are

21	concerned, our score -- I have 13 of my students

22	that were attached to me -- I had them for maybe

23	six weeks, two months at the most, and then I

24	had a different class.	So, would that really --

25	MS. HEBDA:	At this point -- and you're
1	

1	right.	You're right.

2	The VAM is in the second rule, but

3	let's talk about that just for a second.	Well,

4	can I save it when I get to the screen with all

5	the variables on it?	We'll come back.	Don't

6	let me miss that.

7	Is this about this rule?

8	AUDIENCE MEMBER:	Yes.	About the newly

9	hired.	How does that apply to teachers that might

10	move, say, from a district school to a charter

11	school, a charter school to a district school,

12	or from charter to charter since they are being

13	switched -- (inaudible)

14	MS. HEBDA:	That's a great question.	We

15	also had a similar question of someone who

16	moves from administrator to instructional

17	personnel, or instructional personnel to

18	paraprofessional and back to instructional

19	personnel.	And some of those things we don't

20	know the answer to yet.	But, knowing that people

21	need us to define them is a good thing.	And, so,

22	if there's something you need us to be more

23	explicit about -- and maybe you don't want us to

24	be more explicit.	Maybe you want us to let

25	districts decide that on their own.	But, for
1	

1	your consideration, I ask you to think about

2	those things.	And, so, if there's some way you

3	feel that should be more consistent and we do

4	define it in the rule so that everyone does it

5	the same way, that's something you can make a

6	suggestion on.	If that's something that you

7	still want to be left to a school district's

8	discretion, then don't make that suggestion, but

9	that is something to think about.

10	AUDIENCE MEMBER:	I just want to clarify

11	the giving of suggestions.	There is a court

12	reporter taking down what we say.	All the

13	suggestions that everybody has made, do they need

14	to resubmit those?

15	MS. HEBDA:	No.

16	AUDIENCE MEMBER:	You're going to use that?

17	MS. HEBDA:	It will be in the record, that's

18	right.

19	Again, if you want to make a comment

20	at the end of it, the end of the Q&A, you can

21	absolutely make those suggestions again.	If you

22	want to type them up and send them on-line, you

23	can do all that; but she is capturing all the

24	Q&A.

25	Yes, ma'am.
1	

1	AUDIENCE MEMBER:	I have a question.

2	Here this says the Department is going to monitor

3	school districts every five years.	Is the

4	Department going to treat charters as a school

5	district and monitor them every five years, or

6	do they have their own plan?

7	MS. HEBDA:	That's a great question.	She

8	asked about charter schools.

9	The Department doesn't actually

10	approve the charter school evaluations.	The

11	districts monitor their charter schools for

12	personal evaluations the same way they monitor

13	charter schools with defined laws and all other

14	laws right now.

15	AUDIENCE MEMBER:	So the districts should

16	be expected to monitor them every five years or

17	just however they --

18	MS. HEBDA:	However the district chooses

19	to do it.

20	AUDIENCE MEMBER:	So I'm trying to imagine

21	this.	I'm a fifth grade teacher, and you're my

22	fifth grade student.	And then two years from now

23	when you're in seventh grade and I'm wanting to

24	verify my VAM things.	I would like to be able to

25	see what your expected score was back when
1	

1	you were a fifth grader and how you did back when

2	you were a fifth grader.	I'm wondering whether

3	that's in line with FERPA laws because I

4	might not have an educational interest in your

5	scores --

6	MS. HEBDA:	That's a great question.

7	AUDIENCE MEMBER:	-- given then you're now

8	a seventh grader in maybe another state.

9	MS. HEBDA:	That's a great question.

10	We know that the web tool that we're

11	looking at piloting now has the current year in

12	it, or the past year, the ones we just had.	I

13	don't know that we've contemplated having the

14	three years in the web tool, necessarily.	So,

15	we'll have to work on that.

16	AUDIENCE MEMBER:	Referring to her

17	question:	How does that apply to the Florida

18	Virtual School?

19	MS. HEBDA:	Florida Virtual School?

20	There's a portion of Florida Virtual School

21	that is like for all public schools that receive

22	a school grade, just like a laboratory school, or

23	any other entity that functions as its own school

24	district.

25	THE COURT REPORTER:	I couldn't hear what
1	

1	you said.

2	MS. HEBDA:	I'm sorry.

3	There's a portion of Florida Virtual School

4	that functions as its own school district, or as

5	its own school.	They get a school grade now,

6	which they hadn't in the past, but they do now.

7	The teachers that teach under that heading, for

8	lack of a better word, are the ones that the

9	Florida Virtual School evaluates under their

10	approved evaluation system.	They have one just

11	like any other district.

12	But we were discussing this earlier

13	today, too.	There are other teachers that may

14	just teach a course here and there for the

15	Florida Virtual School, not part of their regular

16	public school and those teachers may be employed

17	by some other district on a full-time basis, and

18	they're certainly being evaluated on a full-time

19	basis from that other district.

20	At this point in time, we have not

21	done anything with the other teachers that teach

22	occasionally for the Florida Virtual School.	If

23	they have a regular contract someplace else

24	and we've only -- Florida Virtual School only

25	includes in its evaluation system for this law
1	

1	the ones that are teaching in their regular K-12

2	public school.

3	I don't know if that answers your

4	question, but...

5	AUDIENCE MEMBER:	It helps a little.

6	MS. HEBDA:	Okay.	That's probably about

7	as much as I can help you with Florida Virtual

8	School.

9	AUDIENCE MEMBER:	Under section six where

10	you're talking about substantial modifications --

11	MS. HEBDA:	Yes, sir.

12	AUDIENCE MEMBER:	-- number seven is the

13	one I need some clarification on.	It says:

14	There are changes in the personnel who may

15	contribute evidence to be used in evaluations."

16	MS. HEBDA:	Yes.

17	AUDIENCE MEMBER:	What does that exactly

18	mean?

19	MS. HEBDA:	The law says that we are --

20	you're required to be evaluated by your

21	supervisor, which for most teachers is the

22	principal or the assistant principal.	But then

23	you have an option of including a peer evaluation

24	process or peer review process.	Only a few

25	districts do that right now.
1	

1	Let's say your district does not do

2	that, but then next year they decide they want

3	to.	If they decide they want to now include peer

4	evaluations in their evaluation system, that's a

5	new person who is going to be contributing

6	the evaluations, that would be a substantial

7	or a substantive change.

8	AUDIENCE MEMBER:	Thanks.

9	MS. HEBDA:	Sure.

10	Yes, ma'am.

11	AUDIENCE MEMBER:	Just a clarification.

12	Subsection H, input mechanisms, number two.	"For

13	instructional personnel and school

14	administrators, a mechanism for parents to

15	provide input into employee evaluations where

16	appropriate, and a description of the district's

17	criteria for use of such parental input", can

18	you provide clarification as to what that means?

19	MS. HEBDA:	I can.	Would you mind if I

20	did it on the next slide?

21	I'm a little concerned that we're not

22	going to get to the next rule.

23	Can I go ahead with this one?	And I

24	know you have a lot of VAM questions and things

25	like that.	I want to make sure we get to those.
1	

1	One more question before we go on.

2	AUDIENCE MEMBER:	Under section two,

3	submission process, (a), the second sentence

4	reads:	"Documentation includes handbooks,

5	manuals, forms, and such district policies and

6	regulations that the district uses to inform

7	employees and evaluators of the procedures and

8	content of the district's evaluation system."

9	Would the DOE expect a collective

10	bargaining agreement to be included within these

11	documents as many collective bargaining

12	agreements have details about the evaluation

13	procedures and process in them?	And, if so, does

14	the DOE believe that it has the authority to

15	reject or modify language that's been duly

16	negotiated and ratified?

17	MS. HEBDA:	That's a great question.

18	The only time we would want a

19	document of any kind is if it's the one that

20	actually has your evaluation system procedures

21	and content in it.

22	And the reason why that's said that

23	way is one of the things we were asked way back

24	when we started Race to the Top was, are you

25	going to give us a template and we can just fill
1	

1	it out and that will be our evaluation system.

2	And we said no way.	You're going to do your

3	evaluation system and create a document that

4	you're going to use to inform your teachers or

5	inform your principals, and those things, and

6	that's the document we want to see.	We don't

7	want you answering questions for us and having

8	the real system someplace else.

9	So the only time we need to see

10	anything in a collective bargaining agreement

11	would be if it actually had content or anything

12	of the evaluation system in it.

13	AUDIENCE MEMBER:	So, then, that goes to

14	the second question.

15	MS. HEBDA:	I'm going to answer your second

16	question.	We don't regulate collective

17	bargaining agreements.	That's not part of our

18	job.	However, if, for example, a district has

19	in its evaluation system manual everything that

20	complies with the law but then may have signed a

21	collective bargaining agreement that says, yeah,

22	but we're not going to use that, we're going to

23	use that something else that's not in that book

24	and that something else doesn't comply with the

25	law, we have to point that out.
1	

1	We can't negotiate your contract for

2	you.	We can't do any of those kinds of things.

3	But if we find that you're not complying with the

4	law, we have to say so.

5	AUDIENCE MEMBER:	And if you said so, what

6	would that entail?	What would the consequences

7	be?

8	MS. HEBDA:	Then we would have to go back

9	to the section that says how do we review and

10	approve.	And it depends on what it was.

11	There's a section that defines what

12	you need to do for conditional approval, for full

13	approval, what would happen if you were denied.

14	And if you're denied, you have so many days to

15	correct the deficiency.	So, it depends on what

16	category of things would fall into that

17	collective bargaining agreement.	We try to

18	separate in the conditional and full approval

19	things that were -- like you mentioned parent

20	input.	If a district has got an evaluation

21	system that meets everything else in the law,

22	but they forgot to do parent input or they didn't

23	include parent input, we're not going to deny a

24	whole system just because they didn't do parent

25	input.
1	

1	That would fall into the category of

2	saying, okay, we're going to conditionally

3	approve.	Go ahead implement.	Everything is

4	fine.	But, on this time frame, you're going to

5	have to figure out how you're going to do parent

6	input.

7	So, we tried to separate the big

8	important things that you really need to conduct

9	an evaluation -- like, do you have observation

10	instruments and those kind of things -- from

11	those other laundry lists that I'm about to get

12	to that we have to watch for.

13	So the first thing we depend on is

14	what is it.	And, then, we would go from there,

15	based on this rule, to determine what the status

16	of the system was.	Is it fully approved,

17	conditionally approved or denied?

18	Now, what happens -- and you can see

19	this in the initial review process -- there's a

20	back and forth with school districts.	A lot of

21	times -- because their rule really implements

22	the stuff we've already done for the last two

23	years, which is, the district sends something to

24	us.	We look at it.	Everything we can approve,

25	we approve.	And something we don't understand,
1	

1	we write back and say we don't understand this

2	part.	It looks like it doesn't make sense to

3	us, or it looks like it doesn't comply with the

4	law.	Can you give me more information or can you

5	clarify what this means.

6	So the same thing would happen.	It

7	doesn't matter that it's in a collective

8	bargaining document, or a manual, or anything

9	else.	If the district submits it to us as part

10	of documentation for their evaluation, then we're

11	going to review the part they tell us to review.

12	We won't review the whole collective bargaining

13	agreement.	We don't review the whole manual

14	unless they tell us, you got to look at the whole

15	manual to find these things.	We're only going to

16	look at the part that pertains to what's in the

17	rule.

18	And if we find something that is out

19	of compliance, whatever it is, we'll have to

20	point it out.

21	AUDIENCE MEMBER:	So the answer is

22	basically yes, if your collective bargaining

23	agreement specifically outlines in detail how

24	this process is going to happen, that's something

25	you guys will look at every year?
1	

1	MS. HEBDA:	If it's not already in your

2	manual.	I mean, I would think you would want

3	that in your employee manual, too, the parts

4	that were pertinent.	And if they're in your

5	employee manual and that's where you put it, then

6	we'll look at it there.	We don't need four

7	copies of it wherever it shows up.

8	Does that make sense?

9	AUDIENCE MEMBER:	Yes.

10	MS. HEBDA:	Yes, sir.

11	AUDIENCE MEMBER:	Question on Page 6,

12	paragraph six.	It says:	"For classroom teachers

13	newly hired by the district" --

14	MS. HEBDA:	Yes.

15	AUDIENCE MEMBER:	-- "the student

16	performance assessment data, calculation methods,

17	and performance standards will be applied in the

18	first evaluation."

19	You indicated earlier that newly-hired

20	people are evaluated twice; once mid-year, once

21	at the end.

22	What kind of student performance

23	assessment data would be available for a

24	newly-hired teacher who was observed and

25	evaluated in, I believe the time frame is
1	

1	October?

2	MS. HEBDA:	Could be.	And some districts

3	do it by November or January.	It doesn't say

4	it has to be mid-year.	Typically, districts

5	call it mid-year kind of colloquially.

6	AUDIENCE MEMBER:	If the idea is to give

7	the person a way point -- you're doing okay,

8	you need to fix this -- it kind of needs to be

9	early on.

10	In our district, we actually use

11	three.

12	MS. HEBDA:	Good.

13	AUDIENCE MEMBER:	But it says that first

14	evaluation must include -- and I want to make

15	sure I have the words -- student performance

16	assessment data.	And that student performance

17	assessment data would be the VAM associated --

18	MS. HEBDA:	Wouldn't be the VAM because

19	the VAM is --

20	AUDIENCE MEMBER:	If it's associated with

21	some kind of EOC, or something like that, how

22	has that happened?	Where is that data to provide

23	that component?

24	MS. HEBDA:	Right.	The early year

25	evaluation, and whenever it occurs, can't have
1	

1	end-of-the-year summated data in it because it's

2	not the end of the year.

3	So what districts have used so far --

4	and the reason why we have asked them to tell us

5	what they're going to use -- so a district will

6	show us they've thought about that and they've

7	said what that is.	If they tell us whether

8	they're using progressive monitoring data, if

9	they're using individual teacher assessments,

10	whatever those things are, to give the teacher a

11	look at how their kids are doing so far during

12	the year.

13	The law says it needs to be an

14	evaluation, not just an observation.

15	So how does the teacher get an idea

16	from the principal, or whoever the evaluator is

17	-- how are my kids doing so far?	That way

18	you're just not waiting to the end of the year

19	to see if you met --

20	AUDIENCE MEMBER:	Like how many kids, what

21	grades on their first report card?

22	MS. HEBDA:	Could be.	Could be.	That's

23	up to the local school district.	It's going to

24	make a difference depending on what the teacher

25	is teaching and anything else.	It's just
1	

1	supposed to be progress at that point.	So

2	whatever student progress has happened, that's

3	what you would want to have a discussion about.

4	It's really supposed to make sure the teacher is

5	getting information and not just -- nothing is

6	happening until all of a sudden the end of the

7	year hits and here is the final and, oh, I wish

8	I had known early on; that somebody had told me

9	that I was out of whack or I was ahead of the

10	game.	So, that's what that is for.

11	I'm going to go on.	I promised and

12	I said that section three --

13	AUDIENCE MEMBER:	You said you would take

14	questions on this at the end?

15	MS. HEBDA:	Absolutely.

16	The next thing is -- so here is all

17	the content that needs to be in section three of

18	this rule.	I'm going to go through it kind of

19	quickly because we've already talked about a lot

20	of it already.	But, just so you know, if

21	you're having to look for it, section three, you

22	need to look for the purpose of the evaluation

23	system, make sure it's tied to professional

24	development, make sure it includes student

25	learning growth, and all those kinds of things;
1	

1	and that the district is using that data for

2	informing professional development in school

3	and district improvement plans.

4	We need to make sure the district is

5	using the ratings that are required in the law.

6	We know what those are.	We need to make sure

7	that we have criteria for these three areas

8	because that's what the law says the evaluation

9	is made up of -- performance of students,

10	instructional practice for teacher evaluations,

11	school leadership for leadership evaluations,

12	and professional and job responsibilities.

13	But many districts, not all, many

14	districts combine these two together.	And that's

15	okay.	You don't have to have them in separate

16	sections as long as you're covering criteria for

17	both.

18	This is what the law says about

19	instructional practice.	Many of you are very

20	familiar with that.	I won't spend too much time

21	on it.

22	This is what the law says about

23	instructional leadership.	For teachers and

24	instructional practice, they're tied to the

25	Educator's Office practices (ph).	For
1	

1	principals, they're tied to the Florida

2	leadership standards, plus the law provides

3	some more detailed information that has to be

4	included in the principal's evaluation; that it's

5	related to faculty development and support and

6	effective instruction.

7	Performance of students, you're

8	familiar with this part.	This is what we're

9	checking for in the district evaluation system.

10	This is not the VAM rule yet.	This is what

11	we're looking for in the district's evaluation:

12	At least 50 percent of the evaluation is based

13	upon data in assessment of student learning,

14	either by statewide assessments or local

15	assessments, the option for districts to reduce

16	the evaluation percentages on performance of

17	students to 40 percent when there's an absence of

18	three years' worth of data, whether or not the

19	district is going to split noninstructional

20	evaluation, personnel evaluations between state

21	assessments and other student outcome data, and

22	what are they including, and then whether the

23	district is exercising the options for choosing

24	to combine state and local assessments, and

25	whether or not they're choosing achievement
1	

1	measures, the performance measures, instead of

2	measuring learning growth for a particular class.

3	And the reason why that's an option is because

4	there are some, for example, industry

5	certifications, certification exams, that are

6	just pass/fail.	You can't really measure growth

7	on a pass/fail test.	So a district can have an

8	option to set an achievement measure for those

9	courses, as long as they explain what that is in

10	their evaluation system so the teachers know what

11	that is.

12	Finally, we're familiar with these,

13	too.	This is the section that says what are we

14	going to do until July 1, 2015 if there are

15	classroom teachers whose student assessments we

16	use now are not going to be used in the

17	evaluation system.	And it provides three things.

18	It provides:	If the teachers' students take the

19	statewide assessment -- you're not going to use

20	the local assessment right now -- then they use

21	how those students assigned to the teacher do on

22	the statewide assessments.	If the teachers'

23	students don't take statewide assessments, then

24	learning targets, individual learning targets

25	are established for those students and that
1	

1	teacher based on the goals of the school

2	improvement plan and then approved by the

3	school principal.

4	Some of you do this already in your

5	individual professional development plan.	It's

6	very similar to that.

7	And, then, there's a third option,

8	which is, the school district superintendent can

9	assign instructional personnel team -- or

10	instructional team data to an individual for the

11	purposes of their evaluation.	That's where many

12	of us are familiar with receiving schoolwide

13	data, or eighth grade levelwide data.	That comes

14	from this sentence right here.

15	We look to see if the district is

16	doing that.	And, if they are, how they explain

17	to whom it applies and how it applies.

18	And this is that laundry list I

19	mentioned.	This is everything else in the

20	evaluation system in the law that we have to

21	check for.	And that's why section three gets

22	pretty long.

23	We have to look for, do you have

24	observation instruments with indicators on the

25	(Inaudible) or the leadership standards, are
1	

1	we doing two evaluations a year for newly-hired

2	personnel, have you developed your evaluator

3	training system, and have you considered the

4	guidelines provided by the Department, what's

5	your process of informing everybody about your

6	evaluation system, how have you done that and

7	what's the district doing about that, has the

8	district used multiple data sources in the

9	evaluation system yet, are you using evaluation

10	system data for professional development and

11	school improvement the way the law says you're

12	supposed to do, have you provided for parental

13	input.

14	That was your question.

15	The law just says the district has to

16	provide a way for parents to provide input into

17	the evaluation, teachers and administrators, as

18	appropriate.	That is what it says, as

19	appropriate.

20	So, we've not said what appropriate

21	means.	We leave that to the school district to

22	define.

23	And right now everybody has got an

24	approved system with some kind of parental input

25	in it.
1	

1	So if you like for us to define that

2	further, we can; but, if you're happy the way it

3	is, we'll leave it alone.

4	Next thing is teaching fields needing

5	special procedures.	Those happen all the time.

6	There's a special class, there's a special

7	school, there are teachers that go on maternity

8	leave.	There are all kinds of things that

9	happen.	And, so, how do you provide for those

10	circumstances in your evaluation system?

11	We talked about the annual review by

12	the district for the self-evaluation and

13	self-monitoring process, what are they doing for

14	that.

15	And, then, here are the options.	We

16	mentioned some of these earlier, too.	Options to

17	include peer evaluators, options to include

18	assistance for folks that need it, input from

19	additional personnel into the evaluation system,

20	evaluation amendments -- you brought up earlier

21	-- and additional professional responsibilities

22	beyond the State Board rules.

23	The State Board rule for professional

24	responsibilities is the Code of Ethics and the

25	Principles of Professional Conduct.	That's what
1	

1	we consider professional responsibilities on our

2	end.	The district may have other employment

3	requirements and other things they require as

4	part of the evaluation system, and they're free

5	to add those.

6	Yes, ma'am.

7	AUDIENCE MEMBER:	Have districts been

8	required to submit an annual review of their

9	evaluation systems for '11-'12?

10	MS. HEBDA:	Not yet because we don't have

11	a rule yet.	We're still proposing one.

12	Yes, sir.

13	AUDIENCE MEMBER:	Going back two slides --

14	I think it was two slides -- where you were

15	talking -- well, I don't need you to go back to

16	it -- where it talked about the option to reduce

17	student growth to 40 percent if less than three

18	years of data is available.

19	MS. HEBDA:	For that person, yes.

20	AUDIENCE MEMBER:	For that person?

21	MS. HEBDA:	Right.

22	AUDIENCE MEMBER:	Is the DOE conducive --

23	well, that's not the word I want.	Would they be

24	agreeable to a situation where a district set up

25	a system where it was on a case-by-case basis?
1	

1	In other words, for some people going back three

2	years and using old data from like 2009-'10 might

3	be good.

4	In our district we were really afraid

5	of what last year was going to bring with the

6	FCAT, so we went with three years.	And we're

7	going to stick with that.

8	But, what if you had somebody who,

9	like, one of those years, like, sucked?	I'm

10	sorry for the language.	But that year just

11	sucked.	And I don't want to use it, so I want to

12	use just the two and, therefore, maybe drop it

13	to 40 percent.	Or maybe if -- if we're using

14	two years, somebody had a great two years,

15	somebody had a not so great two years, is it

16	possible that -- you know, for some people it

17	would be 50 percent and for some people it will

18	drop to 40, or is that now something that you

19	would all say no?

20	MS. HEBDA:	Well, I'm going to start with

21	no, but I'll tell you what the yes would be.

22	No, there's no provision for cherry

23	picking which years you like the best to use.	It

24	is all your three years' worth of data if you

25	have it.
1	

1	Now, we did give the districts the

2	option to start with '11-'12 as year one and not

3	use that data we did for historical modeling.	So

4	many districts consider '11-'12 year one and

5	everybody is on 40 percent to start with.

6	Now, you may have a situation, though,

7	where you have one year -- let's say, the teacher

8	for example, out of three years, the middle year

9	they agreed to take a different teaching

10	assignment, and for that year they were going to

11	do hospital homebound -- and you don't really

12	have any data on your hospital homebound students

13	that you know how to calculate or you know how to

14	work with -- so that teacher ends up with two

15	years' data; the one three years ago and that

16	current year and that middle year there just

17	isn't any data for.	So, there could be -- that's

18	where that special teaching fields thing comes

19	in.	But there are issues like that.	Just like

20	if someone goes on maternity leave and there's

21	no evaluation that year.	All kinds of things

22	like that happen.	And a district can always

23	provide for those things.

24	If we have, God forbid, a set of

25	hurricanes or something, people are going to
1	

1	provide for that and that will be different.

2	Yes, sir.

3	AUDIENCE MEMBER:	The multiple data

4	sources, what would be -- what would multiple

5	data sources include?	What would those sources

6	include?

7	MS. HEBDA:	It can include several things.

8	There are a number of things districts have

9	already talked about and some have sort of

10	implemented.

11	The peer review process is one of

12	them where you have observations by two different

13	people.	So, you have multiple data sources

14	there.

15	It also can happen in classrooms.

16	Like, for some of the younger students in

17	kindergarten and first grade, the district may

18	use multiple assessments to assess those students

19	throughout the year, and all of that assessment

20	data is included in the evaluation system.

21	AUDIENCE MEMBER:	I know that the statute

22	says multiple data sources of performance data --

23	multiple data or sources of performance data must

24	be examined, but it doesn't require their

25	employment.	And then -- you got multiple data
1	

1	sources here, but then later on -- meaning, the

2	result is based on more than two sources of

3	data.	Why can't it be just -- why can't two

4	sources qualify as multiple sources?

5	MS. HEBDA:	The literature says you really

6	need to have more -- as much data as you can in

7	the evaluation system.	So we drew the line after

8	two.

9	I mean, there really are two sections

10	of the evaluation system when people combine

11	professional and job responsibilities with their

12	instructional practice, and it's 50/50 or 60/40,

13	however you're starting out.

14	To the extent that we can encourage

15	people to use more data, we felt that it would be

16	important to draw the line in the sand to have at

17	least three.

18	You're going to have -- if you use

19	parental input, you're going to have a third

20	data source anyway, depending on how you use it,

21	if you actually --

22	AUDIENCE MEMBER:	That's a decision by the

23	DOE?

24	MS. HEBDA:	It is.

25	Multiple means -- we felt multiple
1	

1	meant three, not two.

2	Now, we are -- we did -- because we

3	started this way under Race to the Top as well

4	because Race to the Top before we started with

5	the Student Success Act expected multiple

6	measures as well, or multi-metric evaluation

7	system.

8	And, so, we asked districts for a

9	timeline on when they would start to include that

10	third metric or fourth metric, or whatever they

11	wanted to add.	So, we're expecting that to be

12	done by -- when all the other exceptions expire.

13	So, not everybody has more than two

14	right now.	Some people just have two.

15	Yes, ma'am.

16	AUDIENCE MEMBER:	I know we talked a

17	little bit about the evaluator training.	One of

18	the concerns that I have is we've been evaluated

19	last year and this year.	Well, mine is coming

20	up in a couple days, but it's already opened up

21	the window and started.	One of my administrators

22	already expressed that they were trained wrong

23	last year.	And they're going to be going through

24	another training.	But, in the meantime, they're

25	not going through the other training, so we're
1	

1	still being evaluated on whatever they were doing

2	wrong last year.

3	AUDIENCE MEMBER:	Wow.

4	AUDIENCE MEMBER:	Yeah, wow.

5	So there needs to be something to

6	where -- a test or something that these people

7	have to take to make sure that what they're

8	doing is accurate because now we're having two

9	years of evaluations that -- apparently, the

10	district tried to bring in a consulting company

11	that was less expensive and they didn't train

12	them right.

13	MS. HEBDA:	If our language in the rule

14	is not enough, then suggest stronger language

15	for the evaluator training process.

16	All right.	Next rule, .0411.	Here

17	is the rule authority.	These are the things we

18	have to have in this rule.

19	We have to have each formula, the

20	actual math itself, for measuring student

21	learning growth that is approved by the

22	Commissioner.

23	At this point in time we only have

24	one, which was the one that was developed in

25	June of 2011 for use with FCAT data.	So, that's
1	

1	the formula that you see in this rule now.

2	The Student Growth Implementation

3	Committee met yesterday and they looked at a

4	process to review Algebra I.	And they

5	recommended one model for looking at ninth grade

6	-- students take algebra in ninth grade.	It

7	seemed to work really.	And, so, we may have

8	an Algebra I model very soon.	If we do, then,

9	this year would just be optional for districts

10	to use that information, but next year it will be

11	required.	You would have to go with this rule

12	as well.	Right now it's not because it was

13	just yesterday they talked about it.

14	Yes, ma'am.

15	AUDIENCE MEMBER:	Now, in the middle

16	school we have Algebra I as eighth grade

17	advanced.

18	MS. HEBDA:	That's right.

19	AUDIENCE MEMBER:	So, how is that teacher

20	evaluated if they're teaching some algebra

21	classes and some basic eighth grade classes?

22	Where does their student growth come from, the

23	algebra and the course or the FCAT?

24	MS. HEBDA:	Actually, it should come from

25	both if they teach both kinds of classes.
1	

1	Like, a district that teaches sixth

2	grade reading or seventh grade reading or a mix

3	of kinds of classes, it should come from all of

4	those things.

5	I think, most likely, a point in time

6	when districts are first starting in '11-'12

7	if a teacher's -- the majority of her day was a

8	course associated with FCAT and there was one

9	other course, probably the one other course may

10	not have been included because they weren't sure

11	what to do with that one course yet.	So,

12	districts are working into some of these other

13	growth measures.

14	AUDIENCE MEMBER:	So, for this year, how

15	would that have been calculated for the teacher

16	that was doing the algebra?

17	MS. HEBDA:	It depends how the district

18	did it.	They can still use the Algebra I

19	results.	They can still use the end of course

20	results, but they probably would have set a

21	performance or an achievement measure using those

22	results.

23	AUDIENCE MEMBER:	But that is a different

24	formula?

25	MS. HEBDA:	That's a different formula.
1	

1	AUDIENCE MEMBER:	So that means you have

2	to merge the two formulas.	That ought to be

3	interesting.

4	MS. HEBDA:	That's right.

5	Yes, ma'am.

6	AUDIENCE MEMBER:	When you said that the

7	ninth grade algebra results worked really well

8	and that may be a model, does that mean that

9	algebra taught in any other grade would not get

10	a VAM, just ninth grade?

11	MS. HEBDA:	They may at some point, but the

12	first one they approved was the one that worked

13	with the ninth grade class.	So, they kind of

14	split the model up between because if you look

15	across, students take algebra anywhere from

16	sixth grade to twelfth grade.	And the pretest

17	for algebra, of course, is going to be right

18	now, their most recent score on math FCAT, which

19	has some of algebra on it, but it's not an

20	algebra pretest a hundred percent.

21	So, if you can think about that and

22	think about the kind of students that take

23	algebra in the earlier grades, probably your

24	more advanced students who get tracked into

25	those kind of classes and think about the student
1	

1	that is taking it in twelfth grade, then we don't

2	have a prior score for them for five years.

3	It's more difficult to measure growth

4	when a student is taking algebra in twelfth

5	grade than it is if they're taking it in ninth

6	grade when they typically do.

7	So, it was pretty easy for the student

8	growth committee to approve the model for the

9	ninth grade, and they're still working on the

10	ones for the others.	In the meantime, the

11	district can use its achievement measures that

12	they choose for using Algebra I data.

13	Yes and yes.

14	AUDIENCE MEMBER:	You said the models for

15	ninth grade Algebra I students -- you said

16	something about a twelfth grader takes algebra.

17	Every student takes Algebra I in ninth grade,

18	right?

19	MS. HEBDA:	No.

20	AUDIENCE MEMBER:	You have to take Algebra

21	II to graduate, right, so...

22	MS. HEBDA:	Some students have been in

23	cohorts where they haven't had that requirement.

24	That requirement wasn't in place when they

25	started ninth grade.	And there could still be
1	

1	students who are taking it in tenth grade,

2	eleventh grade and twelfth grade.	And you have

3	some students who are taking it as early as

4	sixth grade.

5	AUDIENCE MEMBER:	I understand your early

6	ones, but, like, students who are in ninth grade

7	now are required to have four years of math in

8	high school and are required to have, at least,

9	Algebra II to graduate.	So, I mean, maybe it's

10	different in different districts, but I can't

11	imagine a situation -- to meet those criteria,

12	you almost have to start every kid in ninth

13	grade.	You certainly couldn't start them later

14	than ninth grade.

15	MS. HEBDA:	The majority of students take

16	it in ninth grade.	The second biggest is going

17	to be eighth grade, and then tenth grade; but

18	there are students who, as twelfth graders -- we

19	have test results for them -- that are in

20	Algebra I classes, so it does happen.

21	Yes, sir.

22	Yes, ma'am.	I'm sorry, I missed you.

23	AUDIENCE MEMBER:	Who, me?

24	MS. HEBDA:	Yes.

25	AUDIENCE MEMBER:	I'm one of those teachers
1	

1	who do teach eighth grade, and I do teach

2	Algebra I.

3	MS. HEBDA:	Right.

4	AUDIENCE MEMBER:	And, so, this past year,

5	'11-'12, my scores were based on the FCAT.	And

6	I didn't prepare my students for FCAT because my

7	eighth graders -- they weren't honors.	They were

8	regular eighth grade students.	So, taking

9	algebra was a big challenge for them.	So, I

10	spent my year teaching the algebra because they

11	had to pass the end-of-course exam in order to

12	continue in their career.	Passing the FCAT

13	eighth grade math wouldn't matter a hoot to them.

14	So, I did my job.	87 percent of my

15	students passed the first time, which was

16	phenomenal.	It was one of the highest ones in

17	the district.

18	I have unsatisfactory VAM because it

19	was based on their FCAT math.

20	So, how are we going to -- so, now,

21	we have three years of data.	My first year is

22	going to be unsatisfactory.	I have to live with

23	this for two more years.	And I did my job.

24	MS. HEBDA:	You shouldn't have to live with

25	it for two more years.	Your district is free to
1	

1	use your Algebra I results and they are free to

2	use your Algebra I results last year.	And I

3	think that probably is a conversation that you

4	should have with somebody in your district office

5	to see why that was.

6	AUDIENCE MEMBER:	Because we didn't

7	negotiate it that way because at the time we had

8	to have -- it had to be the VAM.	It had to be

9	this formula for last year.

10	I don't understand how a growth model

11	can be equated to an achievement test.	End of

12	course is an achievement test.

13	MS. HEBDA:	We're going to talk about that

14	in a few minutes, actually.

15	Yes, yes, yes.

16	AUDIENCE MEMBER:	My question is a

17	practical one.	And I'm looking out for your best

18	interests, trust me.

19	You're saying that after all of this

20	time, rolling out the Algebra I EOC with all of

21	the field testing and the first time when it

22	didn't count for the kids except for this way,

23	we are now finally getting to the point where we

24	have a VAM model that's ready to go to the

25	Commissioner for approval --
1	

1	MS. HEBDA:	Okay.

2	AUDIENCE MEMBER:	Okay -- for the ninth

3	grade only because we haven't been able to figure

4	out what those parameters and what those

5	coefficients would be for the other grades.

6	MS. HEBDA:	Correct.

7	AUDIENCE MEMBER:	I'm looking at a

8	situation where come 2014-'15, we have to have

9	these formulas for everything.

10	How on earth are you going to have a

11	VAM model for my calculus class or an anatomy

12	class where they're rolling out the end-of-course

13	exam in 2014-'15?	There's no historical data,

14	there's no logical precursor on which to base the

15	statistical basis for how do you come up with the

16	numbers in the VAM.	How on earth are you going

17	to have a formula that is going to be holding

18	people's jobs and lives accountable at that

19	point?

20	MS. HEBDA:	We're not going to have a VAM

21	for everything because everything doesn't need

22	a VAM.	That's what I want to talk to you about

23	later.	Because it's got to be practical and it's

24	got to make sense.

25	There are assessments that will
1	

1	support a covariate adjustment model like FCAT

2	or SAT-10, and there are assessments that don't.

3	And we can't try to cram a covariate adjustment

4	model into every single assessment.	It's got to

5	make sense instructionally for what you want the

6	students to get out of the class, what the goals

7	are for the student.

8	And, then, we're going to talk about

9	some ways the districts can use the information

10	from the State VAM to actually do something

11	equivalent for those courses that meets the

12	instructional needs for those courses.

13	And, so, if you give me one

14	opportunity to get there, I'll try to work with

15	you on that.

16	AUDIENCE MEMBER:	Okay.

17	MS. HEBDA:	Yes, ma'am, and then yes,

18	ma'am.

19	AUDIENCE MEMBER:	I just wanted to clarify

20	really quickly on behalf of a friend of mine who

21	teaches eighth grade and also has two classes of

22	algebra students.	This year our district would

23	have the option to use algebra EOC, but her

24	algebra students are still taking the FCAT, so

25	they are calculated into her VAM.
1	

1	MS. HEBDA:	Right.

2	There are two things that you'll --

3	the district will have different kinds of data

4	for teachers, but they use the data they use

5	according to their adopted plan.	So, for a

6	teacher who has two algebra classes and the rest

7	is regular eighth grade math -- is that her

8	situation?

9	AUDIENCE MEMBER:	Yes.

10	MS. HEBDA:	Okay.	So, she'll have a VAM

11	for the eighth grade math.	And then you'll have

12	algebra results for her that are just the student

13	performance.	It's not a VAM.	It's just the

14	student performance for those courses.

15	So, this will be an example of what

16	we get to in just a minute of how the district

17	can combine those two things.

18	But she'll have a value-added result

19	for her eighth grade FCAT result based on the kids

20	who took it that year, and also have a seventh

21	grade score.	And then the district can use the

22	Algebra I results instead of performance

23	expectation for those courses.

24	AUDIENCE MEMBER:	So, then, the algebra

25	kids would be extracted from her VAM score?
1	

1	MS. HEBDA:	Right now we don't run a -- we

2	don't run the algebra courses with the other

3	math courses.	We run those separately.

4	Yes, ma'am.

5	AUDIENCE MEMBER:	So if I'm a high school

6	Algebra I teacher and I've got ninth, tenth,

7	eleventh and twelfth graders in the class, my

8	algebra VAM will only be calculated based on the

9	ninth grade students that are in my class?

10	MS. HEBDA:	So far based on the approved

11	model, that's correct.	That doesn't mean they

12	can't use your other data.	It just means your

13	VAM is only going to cover the ninth graders.

14	AUDIENCE MEMBER:	And, then, if I'm

15	an eighth grade Algebra I teacher -- and that's

16	all I teach.	My kids don't have to take eighth

17	grade FCAT math --

18	MS. HEBDA:	Right.	Some districts don't

19	double test.

20	AUDIENCE MEMBER:	-- so, I wouldn't have

21	a math VAM at all, so would the district have

22	to come up with alternate --

23	MS. HEBDA:	The district would set

24	achievement standards if they wanted to for

25	Algebra I in eighth grade.
1	

1	Now, what's in this rule?

2	I'm worried about time.

3	We're going to see the formula.	We're

4	going to see the implementation procedures that

5	go with the formula, and, then, we're going to

6	talk about -- roster verification has to be in

7	this rule as well.	Then, this is the part that

8	very often we need a lot of feedback on, and we

9	don't the rest of these.

10	So, these are the standards.

11	Somebody asked about consistency

12	across districts earlier, how we're going to have

13	that; everybody on the same metric.

14	We have to establish standards for

15	student growth for two reasons.	And the two

16	reasons are going to be important when we get to

17	that part of the presentation.

18	The first reason is just what you said

19	so that we have consistency across districts, so

20	that when everybody is calculating performance of

21	students results right now using FCAT results

22	from the VAM, the value-added model, that those

23	results are then -- the resulting numbers, the

24	value-added scores, which the aggregate score

25	is converted to a proportion of a year's growth.
1	

1	Because we know the average year's growth across

2	the state by grade and subject.	So, everybody

3	would use that result the same way.

4	Then, there's a second reason for

5	those standards.	That's these two right here.

6	The law says that we have to establish

7	the standards so that if the student growth is

8	low enough -- it is so low that an employee would

9	receive an unsatisfactory overall evaluation,

10	okay -- is everybody clear on that?	There's two

11	reasons.	There's consistency in how we do

12	performance of students, but then the law

13	requires us to set a standard so that if that

14	standard is not met, it would result in an

15	overall unsatisfactory evaluation.

16	AUDIENCE MEMBER:	Regardless of how you do

17	on the --

18	MS. HEBDA:	That's correct.	That's

19	correct.

20	AUDIENCE MEMBER:	I got it.

21	MS. HEBDA:	And a floor for the district to

22	consider a person effective and a floor for

23	highly effective.

24	So, keep those two reasons in mind

25	because there is a goal for achieving
1	

1	consistency, but then there's the second reason

2	for the standards in the rules.	So, we're

3	going to talk about both of those so everybody is

4	clear on what it says and you can make your

5	comments accordingly.

6	AUDIENCE MEMBER:	So the law does say,

7	then, that, basically, 50 percent -- a district

8	could say student growth is 50 percent of your

9	evaluation, but it overrides the evaluation

10	process?

11	MS. HEBDA:	It would for the second reason.

12	What we've tried to do in the rule is

13	separate those two uses so that we can still

14	have standards that are used for consistency,

15	but, then, we tried to provide -- and you'll see

16	those later -- what are the conditions under

17	which this second reason would actually apply --

18	AUDIENCE MEMBER:	Oh, I'm familiar with

19	those.	Trust me.

20	MS. HEBDA:	So, now -- this is just a

21	reminder -- we have two separate growth formulas,

22	the ones for State assessments that everybody has

23	to use once they're adopted.	Right now we only

24	have one for FCAT.	We're hoping Algebra I very

25	soon.	And, then, we have the growth formulas
1	

1	that district use, or performance or achievement

2	standards that districts use.	And the law says

3	that the district has to establish equally

4	appropriate learning growth formulas, ways to

5	look at learning growth that's equally

6	appropriate for whatever the State has set for

7	State assessments.	So, we're going to talk about

8	how we get to that.

9	This is a reminder because I told

10	you we have to have the math in the rule.	So,

11	this is a reminder of what's in the model.	Some

12	of you already know this, but it was developed by

13	the Student Growth Implementation Committee and

14	recommended to the Commissioner, and the

15	Commissioner adopted what the SGIC recommended

16	without any changes.

17	You can go on-line still to that

18	website that's in the PowerPoint to see who is

19	on the committee, where they teach; all those

20	things.

21	What's explained in the rule, in

22	addition to the formula itself, is what kind of

23	formula is it.	What does it mean to be a

24	covariate adjustment model?	It means that you

25	start with prior year score, prior achievement,
1	

1	and then you adjust by student

2	characteristics, which are called covariants.

3	You adjust how you expect that student to do

4	based to that prior score and the other things

5	that you know about the student that are included

6	in the model.	Then, what you do is you look to

7	see if -- if this is the predicted score for the

8	student using the developmental scale, last

9	year's score, current year score, you look to see

10	what the difference is in where you expected the

11	student to be based on his last year's score and

12	everything else you know about him and how he

13	actually did.	This is one student.

14	So, all of these differences in what

15	was expected to happen and what did happen, those

16	are rolled up together for all students assigned

17	to a teacher, and that's what produces a

18	value-added score.

19	All of this is all the other things

20	that we know can account for student learning

21	that are included in the prior score -- gifted

22	status.	I'll run down the list for you on the

23	next slide.

24	What's not accounted for, then, is

25	what's attributed to the teacher.	It's what
1	

1	the teacher brings to the student growth

2	learning.

3	The reason why the committee will

4	tell you they selected this model out of the

5	eight they looked at is -- there were some that

6	were in percentiles, there were ranks every year,

7	and things like that -- because this is going to

8	be used for personnel evaluations, for evaluation

9	of an individual person, not school

10	accountability.	So they wanted to get down to --

11	all this other stuff weeded out -- what's the

12	teacher contributing.	So, that's why they

13	selected that model.

14	Yes, ma'am.

15	AUDIENCE MEMBER:	So they can't really

16	tell how that student is supposed to be

17	performing?	They don't know everything.	They

18	don't know whether that child's family is going

19	through a divorce or that child just -- I just

20	had a student, her dad just died the other day.

21	I don't expect her to be right for a while.

22	MS. HEBDA:	That's right.

23	AUDIENCE MEMBER:	So it's not taking into

24	affect a lot of those.	And those of us who teach

25	in more challenging schools have more challenging
1	

1	issues that we have to deal with.	It's not

2	measured anyhow.	It just doesn't make sense.

3	MS. HEBDA:	Right.	There could be things like

4	that that --

5	AUDIENCE MEMBER:	Could be?	There are.

6	AUDIENCE MEMBER:	No could be.	There are.

7	There's a lot of it.

8	MS. HEBDA:	There are a million things that

9	we'll never have in a database, you're right.

10	AUDIENCE MEMBER:	When we talk about the

11	attendance -- we got a program at our school that

12	is pushing these kids to come to school.	We'll

13	reward them, hoopla, we have parties and

14	everything.	But that child sitting in my class

15	with his head down doing absolutely nothing but

16	in school, if it is going to account for

17	my evaluation, I don't want that kid in my class

18	if he's just going to sit there and do nothing.

19	MS. HEBDA:	Thank you for your comment.

20	I understand.

21	AUDIENCE MEMBER:	It's putting us in a

22	difficult situation because we want to help

23	these children, but then we get penalized

24	because things aren't being measured correctly.

25	MS. HEBDA:	Okay.	These are the things
1	

1	that are measured.

2	And what is not included in this

3	presentation, because it's not in the rule, is

4	the impact data that we ran on '11-'12 to see if

5	there were advantages or disadvantages to

6	teachers across the state, meaning in reading,

7	language arts and mathematics.

8	Now, I know you said you're measured

9	on 300 students.	So, it sounds like you're on

10	schoolwide data.	So, I'm not exactly sure how

11	that happened.	You have 300 students --

12	AUDIENCE MEMBER:	I'm a math teacher, 300

13	students for three years.

14	MS. HEBDA:	For three years, okay.	So not

15	300 this year?

16	AUDIENCE MEMBER:	No.

17	MS. HEBDA:	Okay.	I was a little worried

18	about that.

19	All right.	These are the student

20	characteristics and the classroom

21	characteristics and the school characteristics

22	that are included in the model:	Two years of

23	prior achievement scores per student if the

24	student has two years of achievement scores.

25	By far, not just in the research
1	

1	literature but also in our own models for FCAT,

2	the strongest predictor in what makes the --

3	what accounts for almost all the variants that

4	you see in a student score is in their prior

5	score.

6	Of course, that would certainly be

7	an exception if a student has performed a certain

8	way for years and all of a sudden a parent dies

9	or some other tragic thing happens.	So, there

10	would certainly need to be some kind of

11	circumstance that you can't put in a value-added

12	model because we don't have that data on every

13	student, but there certainly could be things

14	that your district could take into account when

15	those things happen.

16	AUDIENCE MEMBER:	Well, that's why we kind

17	of need to have that database of our students.

18	MS. HEBDA:	Absolutely.

19	AUDIENCE MEMBER:	So we can see everything.

20	I guarantee you they're not going to come to us

21	and ask us about every student we have.

22	MS. HEBDA:	I don't know how to help you

23	with that relationship with your district.

24	The number of subject-relevant

25	courses means that some people can be in a
1	

1	reading and a language arts course in the same

2	year, and that seems to matter, or two math

3	courses in the same year.	That may seem to help.

4	Students with disability status, the

5	way the committee determined that was not

6	whether they're disabled, yes or no, but by

7	actually individual disability.	It matters

8	what the kind of disability it is and how well

9	those students across the state do.	And that

10	makes the formula a lot longer, and it makes it

11	a lot more complicated.	But, again the

12	committee went back to -- even though it may not

13	matter and some of these are not statistically

14	significant to the whole state, it could matter

15	to an individual teacher who has multiple of

16	those students in his or her class.

17	Yes, ma'am.

18	AUDIENCE MEMBER:	So are you saying there's

19	a separate -- we teach students with severe

20	autism.	Are you saying there's a separate

21	formula?

22	MS. HEBDA:	There's not a separate formula,

23	but how students with severe autism -- if they

24	take the FCAT because some students don't.	If

25	they don't take the FCAT, they're not in this
1	

1	formula.	So, this is to do with FCAT data.

2	So, your most severe profound

3	students who take the Florida Alternate

4	Assessment, they're not in this formula.	This is

5	only to do with students who take FCAT.

6	And, again, just like any other thing

7	that you would do for school accountability or

8	anything else, if they don't have a prior score

9	-- they came from Georgia -- and this is their

10	first year in the system, this is the first year

11	they took the FCAT, then they're not included

12	because you can't measure growth if there's not

13	a prior score.

14	But there may be several reasons why

15	students might not be included, but that would

16	be one of them.

17	The English language learner status,

18	are they receiving services in their first two

19	years.

20	Attendance, and that is only daily

21	attendance.	That's all we collect at the State

22	level right now.	So, it's up to a district to

23	determine if there are policies --

24	AUDIENCE MEMBER:	Because some kids miss

25	first period every day or --
1	

1	MS. HEBDA:	Or sixth period every day.

2	AUDIENCE MEMBER:	Yeah.	So they're missing

3	your class, but they're there most of the day,

4	so that doesn't show up in that formula.

5	MS. HEBDA:	It doesn't here, but a district

6	can make that provision in their evaluation

7	system.

8	Gifted status is a yes or no

9	variable.

10	Mobility is the number of transitions

11	in and out of school.	That can be to a different

12	school.	It also can be in and out of the same

13	school throughout the year because all that is

14	reported to the statewide, and that matters.

15	That lowers -- in most cases, sometimes

16	not statistically significantly, but it does

17	lower the expectation when they're in and out

18	and in and out.

19	Difference from modal age as an

20	indicator of retention or promotion.	Rather than

21	just having a retention flag yes or no, Dade

22	County has done some research that actually works

23	really well to say that if a student is different

24	in age from what they are expected to be for that

25	class from what the other kids are in that class,
1	

1	then you can see that in some grades and

2	subjects, it will make a difference in how they

3	score across the state.

4	And I should probably say that in all

5	of these instances, all these covariates, are

6	measured not by just thinking of different

7	weights for things -- and this matters more than

8	that, or ten times this, or add two points for

9	something because we think it's important -- it's

10	actually based on how these students with these

11	particular characteristics perform in the state

12	of Florida.	And it's all based on the

13	developmental scale.

14	One of the questions we get a lot is,

15	when the State was setting the performance

16	standards for the five levels for FCAT 2.0,

17	what's going to be level one, what's the cut

18	score for level two, three, four and five.

19	People were concerned that those level cut scores

20	were going to affect their value-added score,

21	but the value-added score is not related to

22	levels one, two, three, four and five.	It only

23	uses the developmental scale.

24	So, wherever your student starts --

25	that individual student starts on the
1	

1	developmental scale, that's the prior score.

2	You don't have to get them to level two to make

3	a gain.	You don't have to get them to level

4	three to make a gain.	You take them from

5	wherever they start to wherever they end up.

6	And wherever they end up then, the prediction --

7	the expectation of where they end up is based on

8	that prior score, however they started, plus

9	those things we know about them.	That green

10	bar is adjusted based on how other students with

11	that same characteristic do across the state.

12	Yes, yes, yes.

13	AUDIENCE MEMBER:	What was the rationale

14	to purposely exclude socio-economic status from

15	the student characteristics?

16	MS. HEBDA:	I can't tell you the rationale,

17	I can tell you two things.	The first thing is,

18	there are four things that the law said we could

19	not include when we were determining what's going

20	to be in that green bar, in how do we expect a

21	student to do.	And that's one of them, they

22	chose socio-economic status, race and ethnicity

23	and gender.	So, we not allowed to choose -- to

24	look at those four things.

25	AUDIENCE MEMBER:	Okay.
1	

1	MS. HEBDA:	Now, we are required to use the

2	prior score, which is good, because that's the

3	biggest predictor anyway, but then we were also

4	required to examine disability status, English

5	language learner status and attendance.	And you

6	can see the committee recommended all these

7	in here.	Plus, the committee recommended all

8	these other things to be added as well.	So,

9	they didn't stop with the things that they were

10	required to look at.	They added lots of things

11	into the model.

12	AUDIENCE MEMBER:	Up to two years' prior

13	growth.

14	MS. HEBDA:	That's right.

15	AUDIENCE MEMBER:	Prior achievement scores.

16	That doesn't mean there have to be before you can

17	establish a growth target.

18	MS. HEBDA:	That's right.

19	AUDIENCE MEMBER:	So, for example, the

20	child who is taking his first ever FCAT, like my

21	grandson did last year in the third grade, there

22	is no data on prior achievement.

23	MS. HEBDA:	That's right.

24	AUDIENCE MEMBER:	Therefore, his third

25	grade teacher, I think, was held -- perhaps
1	

1	disadvantaged compared to, say, his eventual

2	eighth grade math teacher who will be --

3	they'll be able to look at two years of prior

4	math achievement to have a much more reliable

5	target for him to hit.	The last year was a crap

6	shoot in terms of where you think he's going to

7	hit, in terms of we don't have prior achievement

8	scores.	That's a problem.

9	The second issue is -- I need some

10	clarity on attendance.	I know that it's the days

11	of attendance.

12	It talks about percent of attendance.

13	Is that percent of the entire school year, or a

14	percent of the days they could have been -- the

15	days the child could have been there?

16	Is it out of 180, or if the kid showed

17	up in January, is it out of 90, for example?

18	MS. HEBDA:	Let me answer the attendance

19	question real quick.	The districts historically

20	reported daily attendance to the State in survey

21	five, which is the last survey period, which is

22	the end of the school year.	And it would be the

23	summer when they report that data to us.	But,

24	because we want to do this calculation sooner,

25	then we're looking at attendance all the way up
1	

1	through the day that the FCAT is given since

2	this is the FCAT model.	It's the beginning of

3	school year until the testing day.	And it's the

4	number of days present.

5	AUDIENCE MEMBER:	My understanding of the

6	way you arrived at the number to use in the

7	formula was you group students and you look at

8	this group of students had X number of days of

9	attendance or X percentage of days of attendance,

10	they typically perform this, this other quadray

11	(Ph) had this, how fine greened was that?	Was

12	it by five, by tens, by ones?

13	MS. HEBDA:	That's a great question.	I

14	don't remember off the top of my head.	It's in

15	the technical report, and I can look that up for

16	you and provide it to you.

17	AUDIENCE MEMBER:	But there is no --

18	MS. HEBDA:	But it is a rolling variable.

19	So, it's not every single data --

20	AUDIENCE MEMBER:	You mention that

21	districts have an ability to account for the

22	scenario where -- for example, what's the -- even

23	with elementary school kids.	A child has reading

24	early in the day, but the child has orthodontists

25	appointments, you know, which take him out one
1	

1	day a week.	So, he misses 20 percent of his

2	reading.	And the kid comes back after reading

3	class is over, but he's missed reading class, but

4	he is in attendance during that day.	Your model

5	does not take that into account; is that correct?

6	MS. HEBDA:	That's correct.

7	AUDIENCE MEMBER:	You said, though, that

8	the district could somehow account for that.	My

9	question is:	How?

10	MS. HEBDA:	Through roster verification

11	because one of the things that we did last year

12	-- and we're going to do the same thing this

13	year is, even though -- as I've explained

14	earlier, even though we have roster verification

15	for the first semester and the second semester,

16	there's one more chance for the district to

17	true-up the file that it's going to use, matching

18	the teachers and the students before we actually

19	run the VAM calculation after even survey three

20	data is verified.

21	So, at that point if there are

22	students that need to come out of the roster

23	because for all those reasons, that can be done

24	at that time.

25	AUDIENCE MEMBER:	Does the district have
1	

1	to -- does the district have to -- what's the

2	word I'm looking for -- justify to the State that

3	this child should be pulled out?

4	MS. HEBDA:	At this point we're not asking

5	that the district justify those things now.

6	AUDIENCE MEMBER:	So, really, districts

7	could gain the system by pulling out -- by saying

8	we want to extract these kids who we believe are

9	performing poorly?

10	MS. HEBDA:	I suppose anything could happen.

11	AUDIENCE MEMBER:	Would that be up to the

12	individual teacher to take that to their county

13	and say, look, I have little Johnny here who has

14	20 minutes out of my class period two days a week

15	for the last 20 days, I would like for him to be

16	removed from my value-added model?	That's up to

17	us as an individual.

18	MS. HEBDA:	That process of roster

19	verification between the teacher and the school

20	and the school and the district is set up by each

21	district.	So it may be that it's left to the

22	teacher to report that to the principal.	It may

23	be the teacher reports it to the district

24	directly.	I don't know how it works in anybody's

25	district.
1	

1	But, there is daily course attendance

2	at the school, kept at the district level, but

3	each district's communication process is going

4	to be different.	So, that would be a question to

5	ask.

6	Yes, ma'am.

7	AUDIENCE MEMBER:	Leveling the playing

8	field.	We've identified that the strongest

9	predictor is student growth.	But, theoretically

10	-- I have an eighth grade student who is mobile

11	because the parents are migrant workers.	So,

12	they're in and out of school.	And they are an

13	English learner, language learner, and they've

14	been identified with a learning disability,

15	SLD.

16	Okay, so, what percent of those

17	factors is going to be considered in leveling the

18	playing field for me?

19	MS. HEBDA:	Actually, they will all be

20	considered because we have data on all of those

21	things.

22	AUDIENCE MEMBER:	At what percent?

23	MS. HEBDA:	That's the question we were

24	asked earlier.

25	So, it's not a set of random weights,
1	

1	and we don't multiply those by ten or 20.	It's

2	all based on development skills for points of

3	how, on average, kids with those same

4	characteristics, or that same characteristic,

5	did across the state.

6	So, one of the things that we have

7	on the website right now -- and I can send it to

8	whoever I need to so you can find it, but on the

9	same page that's in front of the PowerPoint where

10	the evaluation system information is, there's

11	something called a technical report, the VAM

12	technical report.	Go to the back of it.	If you

13	go to the very back of the technical report, by

14	grade and by subject, they show you what are

15	called the coefficients, which means how much was

16	that adjustment, up or down, in the development

17	scale score points by each one of these things.

18	So, you can see for section -- and

19	they do vary by grade and subject.	So, you'll

20	want to look to see -- for a particular

21	disability, you can see -- or for gifted status

22	-- how much is that coefficient?	How much was

23	that green bar adjusted for those students up

24	or down based on being gifted, for example.	You

25	can see that by grade and by subject.
1	

1	AUDIENCE MEMBER:	Where would that -- where

2	would those kind of coefficients actually be

3	found?

4	MS. HEBDA:	It's in the value-added model

5	technical report, okay, which is on that same web

6	page under student growth.	Go under student

7	growth and you'll see that, among other things --

8	PowerPoints and everything else -- but look for

9	the technical report and go straight to the back.

10	Go straight to the back and you can see what they

11	were for '11-'12.

12	So, we need to publish those each

13	year, because kids take the test again every

14	year, so you can know what they were.

15	AUDIENCE MEMBER:	Do you use the prior

16	year's data to come up with those --

17	MS. HEBDA:	Current year.

18	AUDIENCE MEMBER:	The ones in the actual

19	year?

20	MS. HEBDA:	That's right.

21	And that's why we get to the issue of

22	-- teachers want to know at the very beginning

23	of the year -- what's that predicted score for

24	that kid?	So, we can't know what that is at

25	the beginning of the year because we don't
1	

1	know how many days they're going to be present

2	and how much mobility they're going to have.	We

3	don't know those things.	And, so, you want to

4	know what it is.

5	Now, what you can do, though, is we

6	can provide -- by looking at that information,

7	you can see -- because you do know the kid's

8	prior FCAT score.	That's something you do get as

9	a teacher.

10	So, if you look at the coefficients,

11	you can see from the prior year how do kids

12	with that same score tend to do?

13	AUDIENCE MEMBER:	Would there be a formula

14	for each student?

15	MS. HEBDA:	There's not a formula for each

16	student, but you can see how those kids with that

17	same score tended to score the next year, what

18	was the second score?

19	AUDIENCE MEMBER:	For our students would

20	we get the formula?

21	MS. HEBDA:	Well, for all of the --

22	AUDIENCE MEMBER:	Each student has to have

23	their data to put into the formula and some of

24	them have more coefficients than others because

25	they have more of these characteristics.
1	

1	MS. HEBDA:	We can give you the predicted

2	score and you can see the actual score.

3	AUDIENCE MEMBER:	Will we be given a list

4	of what student characteristics were included

5	in their formula?

6	MS. HEBDA:	That's a good question.	I

7	don't know if that's included in the current

8	web page, but that's a good question.

9	Yes, ma'am.

10	AUDIENCE MEMBER:	Come to the student

11	disability category there since that's what I

12	deal with most.

13	I'm curious, because this has come

14	up, when you were talking about the ones that

15	are more severe, they're not counted because

16	they don't take standardized tests.	However,

17	there are some that are, quote, mildly disabled,

18	that are required to take the FCAT, but cannot

19	perform well on the FCAT because, (a), they don't

20	get the accommodations they get in the classroom

21	on the FCAT; and/or (b), their disability in some

22	way, shape or form, either other health impaired

23	or traumatic brain injury, won't allow them to

24	retain that information for such a comprehensive

25	exam.	But, yet, I'll still be calculated on that
1	

1	because the State also says we can't exempt but

2	a certain percentage -- I think it's like three

3	percent -- from taking the FCAT.

4	So, how is this formula really going

5	to calculate my score in a fair manner when all

6	those things come into play?

7	MS. HEBDA:	I think you'd want to look at

8	what your scores were, what your score was and

9	what your students scores were, and then,

10	again, look at the coefficients again because

11	traumatic brain injury is a cognitive disability,

12	it was included in the list.

13	And you can look and see the

14	coefficient.	You're going to see a traumatic

15	brain injury has a huge coefficient, and you're

16	going to see that it's very technically

17	significant.	So, those students would not be

18	compared to a student who didn't have

19	traumatic brain injury.

20	You're also going to need to compare

21	it to the student based on their entering score.

22	So, they wouldn't have started at level three or

23	four necessarily unless they just had the brain

24	injury, I imagine.	But if they had that for

25	many number of years, then their score is going
1	

1	to be at a certain level.	And, then, they're

2	only going to be expected to do so well, based

3	on how all other students who have the exact same

4	injury and the same disability across the state

5	did.

6	AUDIENCE MEMBER:	The one that kind of

7	levels that out is the specific learning

8	disabled student because that incorporates a lot

9	of different things.	There could be mild and

10	supposed to be higher-performing kids, but I have

11	a number of them that are not that high performing.

12	MS. HEBDA:	And their entering test score

13	is the basis for their whole prediction, not the

14	high level of some other kid.	Their score and

15	every other kid that started at that same spot

16	and then also has that same disability.

17	AUDIENCE MEMBER:	May I ask two questions?

18	MS. HEBDA:	You can ask anything you want.

19	AUDIENCE MEMBER:	First of all, getting

20	back to what she was saying -- I'm Janette.

21	But I have autistic, SLD, language; all of them.

22	So, the students with disabilities -- is the

23	autistic percentage the same as the language

24	percentage?

25	MS. HEBDA:	Not necessarily.	Each
1	

1	disability is separate.

2	AUDIENCE MEMBER:	And that's by diagnosis

3	of the counselor -- I mean, the psychologist at

4	the school?

5	MS. HEBDA:	However the district has

6	reported that information to us.

7	AUDIENCE MEMBER:	Secondly, talking about

8	absences.

9	MS. HEBDA:	Yes.

10	AUDIENCE MEMBER:	In our county, they take

11	the FTE in October and then again in -- I

12	think it's February.	And if those students were

13	not at both of those FTEs, they do not count.

14	I want to make a comment, so that it's

15	on the record, to let you know what's happening

16	out there.

17	If there is students that don't count,

18	which may be your child, or your grandchild, or

19	whatever, they are verbally being told, sorry,

20	you don't count, you don't get to have tutoring,

21	I'm going to take all of my students in this

22	grade level and put them in a class by themselves

23	with a brand-new teacher, or whatever -- even if

24	their IEP says one thing, you're not going to

25	get it because the only students -- the only
1	

1	thing that is going to happen -- I mean, the only

2	ones who are going to get their IEP, that is going

3	to get everything that they need are the students

4	that count.

5	This is verbally being told to these

6	children -- sorry, you don't count.	You have to

7	go over here.

8	This is real life that is happening

9	out there because of this.

10	I want that in the record because,

11	God forbid, that it was ever my grandchild,

12	or yours, or whoever, to be told from second

13	grade, sorry, you don't count -- or third grade,

14	I mean.	Sorry you don't count.	I know you can't

15	read, I know you can't, you need this help; but,

16	sorry, you cannot have it because you don't

17	count.

18	MS. HEBDA:	I will add one thing to that.

19	For purposes of value-added calculation, which is

20	different than school grades, because school

21	grades does require the two/three match -- for

22	value-added purposes the student only has to be

23	in either survey two or survey three because we

24	have block scheduled courses that only last a

25	semester.	We would eliminate all the block
1	

1	scheduled courses if we didn't do that and

2	because we account for attendance.

3	Now, there are some districts in

4	their evaluation system plan that require a

5	survey two/three match, and there's nothing

6	preventing them from doing that.	But if they

7	leave students in a course, teacher course

8	record -- they're there for one semester

9	because it's a semester long course -- then

10	we'll calculate the value added for that one

11	course, that one survey.	We will do that.

12	AUDIENCE MEMBER:	I'm talking elementary.

13	AUDIENCE MEMBER:	You have to be there --

14	MS. HEBDA:	You don't have to be there for

15	VAM unless the district has that policy and

16	that's the file they give to us when roster

17	verification is over with.

18	AUDIENCE MEMBER:	It depends on the

19	district?

20	MS. HEBDA:	That's right.	They're the

21	ones -- we calculate -- as long as they're in

22	one survey period or the other, we'll calculate

23	that because we have so many block courses.

24	AUDIENCE MEMBER:	Please understand that

25	is real life and that is happening.
1	

1	MS. HEBDA:	I'm glad you reported that.

2	I'm not glad to hear it, but I appreciate you

3	reporting it.

4	One, two, three.

5	AUDIENCE MEMBER:	This whole VAM, the way

6	you're saying the characteristics in

7	determining whether or not they're really -- by

8	looking at the data you're determining whether

9	or not it's a negative or a positive impact on

10	the growth?

11	MS. HEBDA:	That's right.

12	AUDIENCE MEMBER:	So, it requires a lot of

13	trust that we are trusting that the State is

14	doing this correctly.	So that's a great concern

15	to me because we've had issues in the past with

16	scores not being accurate, and so on and so

17	forth.	So, I mean, our livelihoods, our careers

18	depend upon this.	I mean, the State needs to

19	come up with a way to prove what they're doing

20	from the beginning.

21	I mean, I know they need to show us

22	what students are on our roster, what their

23	projected growth was; but, I mean, it goes much

24	further than that.	It goes all the way down to

25	whether you came up with a correct number, or
1	

1	whatever it is, the statistical number on the

2	ELL students, if they did that correctly.

3	So, I don't understand how, when

4	your career depends on it, your livelihood does,

5	that we can just trust that this is the way it's

6	going to happen.

7	MS. HEBDA:	No, you're right.

8	There are three things that we can do

9	-- we do now to try to have a second set of eyes,

10	if you will, to look at the information.

11	Part of it comes from the relationship with the

12	school districts who send us the data, and they

13	also verify the rosters and everything else.

14	We also have a technical advisory

15	group, who is made up of national researchers

16	who examine the model and look at the results

17	every year.	They have that second set of eyes

18	that aren't Florida people.	They're other

19	national experts that do this same kind of work

20	and they make commentary on the model and how

21	it's run.

22	Those are the two things I know now,

23	plus the teacher looking at the roster.	And when

24	the web tool is able, looking at the things that

25	are reported about each student and if you agree
1	

1	with those or disagree and how that might happen.

2	The only other thing I can tell you

3	is that we could certainly have an option not to

4	do something that's as complicated as a covariate

5	adjustment model that has things like Greek

6	letters in it and all that other kinds of stuff

7	that has to be calculated on a statewide basis

8	by somebody who has the capability and the

9	capacity to do that, like the Department, and do

10	a simple growth model.	We can go back to that

11	chart and we can get rid of that green bar and

12	just say last year, this year, subtract.	There's

13	the answer.	Everybody can figure it out with

14	a pencil and paper; but, then, we wouldn't have

15	any way to account for all these other things

16	that could influence learning.

17	And some states don't.	Some states

18	don't do all this.	They just say last year, this

19	year, like we do for school grades.	The learning

20	gain -- did you move up a level, or did you stay

21	in level three, or did you move up this much on

22	the developmental scale on level one?

23	We'll have to decide as a state

24	whether it's worth doing this or not, whether we

25	just need to make it more simple and not worry
1	

1	about all that, but that's a decision we have to

2	work through.	But, in the meantime, we will

3	continue to do as much as we can so that people

4	-- and one of the reasons why we also didn't

5	buy a value-added model from a vendor, because

6	there are people out there looking, we contracted

7	with American Institute for Research to work

8	with the student growth committee so that our

9	model is in that technical paper, and it's in a

10	rule.	It's not in a license agreement hidden

11	somewhere so that nobody can replicate it.	It's

12	out there.	Anybody can replicate it, who does

13	that kind of work, they can.	So, we're trying

14	to be as transparent as possible.

15	Although, we still have more to do to meet your

16	expectations, I know.

17	Yes, sir.

18	AUDIENCE MEMBER:	The only thing I would

19	add to what you just said is that anybody

20	that has ever seen that formula, unless you are

21	an absolute world-renowned statistician, nobody

22	can understand it.	Let's be candid.

23	You got people's careers on the line

24	for a subject formula that it cannot understand,

25	no one can explain it to them.
1	

1

2	it is.

3

This is not your fault.	It is what

Granted, all what you said makes

4	sense.	These things are there to be variables

5	to contribute to the formula.

6	But the word trust that Beth uses

7	is absolutely correct.	And I don't believe that

8	trust exists currently throughout the state,

9	let's be honest.	You got fear factors and

10	everything else going into this.

11	MS. HEBDA:	Right.

12	AUDIENCE MEMBER:	So, there are some issues

13	that have to be reevaluated, whether it's through

14	our legislature or whoever because this --

15	MS. HEBDA:	That's a decision to make.

16	That's a decision to make.

17	AUDIENCE MEMBER:	Thank you.

18	MS. HEBDA:	If it just is too complicated

19	and we just need to go back and not look at what

20	else is called (inaudible) learning and just

21	look at plain old learning gains and look at a

22	level one -- did you go from level two to level

23	three -- someone is going to raise their hand

24	and say, I failed it, my kids can't do it.	So,

25	we need to decide that.	We need to have that
4	

1	debate.

2

If this is too hard, then we need

3	suggestions for what else we can do, absolutely.

4	Yes, sir.

5	AUDIENCE MEMBER:	I just had some friends

6	call me yesterday and tell me about a change

7	that took place in their Algebra I classes.	They

8	had a teacher who resigned.	This is like the

9	second time a teacher for this one particular

10	class has resigned.	And they've had subs

11	throughout the year.	And, so, they took that

12	class and they split it up between the remaining

13	teachers.	Actually, the whole day.	So, five

14	classes were split up, three went to teachers

15	for an extra class, and they took the other two

16	classes and separated them in the other classes.

17	How will VAM account for those

18	students' test scores?

19	MS. HEBDA:	I don't know that you can

20	account for that.

21	AUDIENCE MEMBER:	For the teachers who

22	are currently teaching them?

23	MS. HEBDA:	I don't know that you can

24	account for that with the model.

25	AUDIENCE MEMBER:	That doesn't fall under
3	

1	mobility?

2	MS. HEBDA:	It may.	It may.

3	One thing I would say, because I got

4	a little bit lost in who went here and how many

5	teachers --

6	AUDIENCE MEMBER:	You had five teachers

7	-- five new teachers take on this one teacher's

8	students.

9	MS. HEBDA:	So they got split up to five

10	teachers.

11	AUDIENCE MEMBER:	Between five different

12	teachers.	And these students that got split up,

13	throughout the course of the school year, have

14	had at least four different teachers in their

15	class.	And one just resigned this last week --

16	MS. HEBDA:	Right.

17	AUDIENCE MEMBER:	-- who started, I guess,

18	at the beginning of the semester.	And then last

19	semester --

20	MS. HEBDA:	Okay.

21	AUDIENCE MEMBER:	-- they had a couple

22	long-term subs and another teacher who resigned.

23	So, how does the VAM account for that?

24	MS. HEBDA:	I don't think you can account

25	for that in a VAM.	I think those are the kind
1	

1	of situations that are not going to happen

2	routinely in every single Algebra I class across

3	the state.	Those are anomalies that happen.

4	The district is going to have to say,

5	you know what, we had five teachers for these

6	students, I cannot possibly tell you which

7	student is assigned to which teacher for this

8	year, so we're not going to use that one.

9	I mean, those things are going to

10	happen.	The school is going to burn down or

11	something and things are going to have to

12	happen --

13	AUDIENCE MEMBER:	Those teachers would

14	have to challenge those students as being a

15	part of their roster.

16	MS. HEBDA:	I don't know what the process

17	is again.	But if the teacher tells the principal

18	or the school tells the -- that's between the

19	-- for the district to explain to the folks how

20	to get that done, but that certainly sounds like

21	a situation to me that somebody would want to

22	take a look at in the overall evaluation system,

23	but we're not going to have variables where we

24	have five teachers, or more, necessarily in a

25	model.
1	

1	So, some things can be taken care of

2	in a growth model, some things can be taken care

3	of in an observation instrument, or procedure,

4	and some things just have to be taken care of in that

5	category of special procedures because they're

6	just different.

7	I'm a little concerned that we're not

8	to going to get to the standards because we have

9	only five minutes left in the scheduled meeting.

10	I'll stay as long as you want me to.	I know you

11	have to go places.

12	AUDIENCE MEMBER:	I have a language

13	question.	It has to do with the level playing

14	field.	I notice it takes into account classroom

15	-- class size.

16	MS. HEBDA:	Yes.

17	AUDIENCE MEMBER:	So the language says

18	that:	"This variable is a count of the number

19	of students assigned to the teacher."

20	If I have a class that's eight deep

21	versus a class that's 22, is it a different

22	number factored differently for the kids in the

23	room of 22 than it is for the kids of eight?

24	MS. HEBDA:	Right.	When you look at the

25	coefficients in the back, you'll see that -- it
1	

1	generally -- I'm probably going to remember this

2	wrong, but, if I remember correctly, it

3	generally takes a difference in ten kids to

4	actually move a developmental scale score point.

5	So, if you have a difference of eight

6	and 22, that could make a difference.	But if

7	you have a difference of 22 and 21, probably not.

8	AUDIENCE MEMBER:	Okay.	So this is not

9	what would be considered unclear language?

10	MS. HEBDA:	It could be.

11	AUDIENCE MEMBER:	Variables that count.

12	MS. HEBDA:	It could be.	It very well

13	could be.

14	Yes.

15	AUDIENCE MEMBER:	This whole thing about

16	challenging a particular student because of

17	unusual circumstances presumes that we know

18	which kids were attached to which teachers by

19	name.

20	MS. HEBDA:	Yes.

21	AUDIENCE MEMBER:	If you just have a raw

22	number without the kids -- if we don't know who

23	we have to challenge -- and I do notice on

24	Page 5, paragraph D it says, at least 30 days

25	a school -- a district will have at least 30
1	

1	days to review the data.	I'm glad to see at

2	least 30 because this data is likely to be

3	released perhaps in the summer when teachers

4	aren't on board.

5	There is no -- is there perceived

6	to be a cut-off date not beyond 120 days, or is

7	that open-ended?

8	MS. HEBDA:	Not necessarily.	I think that

9	one of the things we are trying to put parameters

10	around, because if the roster verification

11	process happens the way it should, even with

12	time between when survey three ends and the end

13	of the school year, there should be that time in

14	there when the districts, just like they're

15	verifying for FCAT for school accountability

16	purposes and getting the right students matched

17	with the right test score and making sure

18	that's all correct and accurate --

19	AUDIENCE MEMBER:	You mean, presumably --

20	MS. HEBDA:	-- that should happen before

21	the end of the school year.	And, then, after

22	school is done we build a file to run the

23	calculation.	So we should get the data back to

24	the districts by July, so that they have that

25	information by July.	That's the goal.
1	

1

2	could.

3

Let's go to the standards, if we

A couple things you need to know

4	before we do the standards so that you understand

5	how we constructed them is -- the first thing is,

6	in one -- of the things that you're looking at

7	when you're looking at value added -- remember,

8	we saw the predicted school for each student and

9	what the teacher's value-added score is -- zero

10	is actually a good thing because that's how

11	students typically performed across the whole

12	state in that grade and subject.

13	So, then, that means that a number

14	above zero -- a positive number means the

15	students assigned to that teacher, on average,

16	outperformed what they were expected to do, again

17	based on developmental scale score points and

18	all those things in the model.	And then a

19	negative number would mean that, on average,

20	students assigned to that teacher performed

21	less well than expected.	It doesn't mean that

22	every single student did.

23	So, then, when you combine all those

24	things together for all the students assigned

25	to the teacher, then you're looking at, on
4	

1	average, how does that work?

2	So, we're looking at zero being

3	typical, and then things above zero and things

4	below zero.

5	One of the other things you get --

6	and this is going to get to the question you had

7	earlier, how would a district set comparable

8	standards?

9	One of the things we also get from

10	the value-added modeling is we get that percent,

11	that number percent, of students that met or

12	exceeded their expected score.

13	So, one of things that districts look

14	at -- in fact, there are eight districts in the

15	state for '11-'12 use this information from the

16	value-added model rather than the score because

17	to them it was similar to learning gains and it

18	was easier to understand on how many of those

19	teacher's students that are assigned to that

20	teacher met or exceeded their expected score.

21	And that's what they used.	And, even if you

22	don't use that information for -- or even if

23	your district uses a score, they still get this

24	information.	So you could get this information,

25	and it's important to note.	So, that's one piece
1	

1	of data we have.

2	The other piece of data we have is

3	called the standard error, the standard error

4	of measurement.

5	And we always use the example to

6	explain the standard error measurement of the

7	polling that's done all the time, is the

8	President doing a good job.	And a polling

9	company will poll five states and poll registered

10	voters in five states and ask whether or not the

11	President is doing a good job.	And let's say,

12	for the sake of argument, that's voters responded

13	to the poll and 50 percent said yes, 50 percent

14	said no.

15	So, what you see reported is 50

16	percent say the President is doing a good job.

17	And down at the bottom you'll see plus or minus

18	three points, or plus or minus five, plus or

19	minus one.	That means -- that does not mean that

20	50 percent of the voters that they just polled

21	could have said something different.	That's what

22	they said.	50 percent of them gave that answer.

23	That was their answer.	That's the truth.

24	What they're saying is, if I had

25	sampled a different set of registered voters in
1	

1	those same five states and met all whatever

2	those characteristics were of the people I

3	surveyed, I'm confident that they would have

4	given me the same result within three points on

5	either side; that that result would have been

6	somewhere between 47 and 53.

7	Now, how does that relate to what

8	we're talking about here?	Because you just

9	earned, some of you did, earned a value-added

10	score if you taught reading, language arts,

11	mathematics.	That value-added score represents

12	what happened that year.	It is the 50 percent

13	that we saw in that poll; but your district

14	also received with that score this, the standard

15	error that goes with that score, which means that

16	if you had been assigned a different group of

17	students that year, because students aren't

18	randomly assigned to schools, students aren't

19	randomly assigned to teachers.	They're assigned

20	to teachers for lots of different reasons.	So,

21	if you have been given a different group of

22	students, your district is confident that

23	within standard error, that confidence interval,

24	that plus standard error, minus standard error

25	number, that your score would have been in that
1	

1	range given a different group of students.	It

2	doesn't change what your score was.	Your score

3	was a representation of what actually happened,

4	but that confidence interval is used to say how

5	confident am I, given a different set of

6	circumstances, that things would have turned out

7	the same way.	That's what it's there for.

8	And the reason why we use that

9	information is not to say, again, the value-added

10	score doesn't matter or didn't happen, it did

11	happen, it is the score, it's just to say how

12	confident are we in making a judgment about that

13	teacher's performance on that one criterion in

14	the evaluation system using that score.

15	So, it's really sound statistics and

16	we want people to use that sound statistics.

17	We don't want you making snap judgments.

18	So, for each value-added score, we

19	provide a standard error.	And from that, the

20	district needs to construct its confidence

21	interval.

22	Now, before I show you what our

23	standards are, there are a couple of things I

24	want you to know.	These are things you have to

25	consider before we get the feedback on the
1	

1	students.

2

You also know, or you're familiar

3	with probably, that when you do value-added

4	modeling, you end up with a distribution from the

5	lowest to the highest score.	And every year you

6	get a new distribution because students retested

7	and you get a new calculation of scores on things

8	that happen.

9	We could -- just say every year we're

10	going to use that new distribution and somebody

11	is going to be at the top and somebody is going

12	to be at the bottom and let it be a rank.	Some

13	states do percentile rankings that way, but we

14	don't want to do that.	We know we're going to

15	get a distribution every year, but what we're

16	trying to do is use -- we proposed '11-'12.	Some

17	people have said maybe you want to do '12-'13,

18	but for right now what's for your consideration

19	is '11-'12.

20	Remember I said that a value-added

21	score is a proportion of a year's growth.

22	That we would use the growth that

23	happened in '11-'12 as our anchor.	So that could

24	become our criterion reference.	So that every

25	year when we do a distribution, people can get
20	

1	better and better and better and that anchor

2	stays where it was.

3	So, if the average year's growth was

4	ten points, for example, in seventh grade math

5	the first year but we start doing better than

6	that, your performance evaluation is then based

7	on how much better you are every year than that

8	original seven points.	And if we don't do that,

9	we'll still get the distribution, but then you're

10	not sure how you're going to do until you find

11	how the other 168,000 teachers did.	You know

12	what I mean?	That's good information to know.

13	It's good for you to know just professionally

14	how you might have compared to other people.

15	So, we're trying to make this --

16	because we don't want to limit the number of

17	people who are going to be highly effective or

18	establish a percentage of people that have to be

19	unsatisfactory or needs improvement.	We want to

20	establish a criterion standard based on that

21	year's -- that annual growth that people can try

22	to beat every single year.	And that way you'll

23	know what you're shooting for.

24	Yes, sir.

25	AUDIENCE MEMBER:	This sounds real good
1	

1	except they ran a simulation in Seminole County.

2	The numbers that I saw were that over 16 percent

3	of the teachers last year would have been rated

4	unsatisfactory or needs improvement.	And I

5	understand you don't want to rank all the

6	teachers in using that method, but the population

7	of teachers in a roughly normal sample --

8	MS. HEBDA:	It is.

9	AUDIENCE MEMBER:	-- should fall within a

10	normal bell curve.	And, so, in a normal bell

11	curve 95 percent of the people should be average,

12	should be in the mean, right two standard

13	deviations --

14	MS. HEBDA:	Uh-huh.

15	AUDIENCE MEMBER:	So, by that logic, which

16	is pure statistical fact two-and-a-half percent

17	of the people -- well, as much fact as it can be

18	-- but, two-and-a-half percent of the people

19	should be in that lowest that unsatisfactory

20	group where the model that has been proposed was

21	way higher than that.

22	So, is there some consideration taken

23	that this model may not be good if 16 percent

24	of the people are filling up in a category that

25	should really only include two-and-a-half
1	

1	percent of the population?

2	MS. HEBDA:	In fact, that is exactly why

3	we are doing workshops on this because those

4	standards are set for the entire state.	And

5	Seminole may have run some data.	And other

6	districts may run their own data and say, well,

7	the population that I have that lives in my

8	district may not necessarily be a normal bell

9	curve of the whole state, but here is a slice

10	of the bell curve that we have.

11	So, these proposed standards -- our

12	suggestion in Seminole County is that you adjust

13	them thus and so, okay, and that's important to

14	do.	And one of our -- even though I haven't

15	seen the data that's been run in Seminole

16	County -- I haven't laid eyes on it yet, I did

17	read the news article.	In fact, I contributed

18	to the news articles yesterday because we got a

19	call --

20	AUDIENCE MEMBER:	I got a quote from

21	here.

22	MS. HEBDA:	And they said, what do you

23	think about their data?

24	I haven't seen their data.

25	But I can tell you that I wasn't
1	

1	kidding when I said I was grateful they were

2	running data.

3	And, frankly, if we hadn't started

4	this process last year and allowed districts

5	to establish their own cut points last year,

6	they wouldn't have been able to run data because

7	they wouldn't know what we were talking about.

8	So, the fact that people can do that

9	and they can give us good feedback and say, this

10	is how we think you ought to adjust these cut

11	points, or you want to add one here or add one,

12	that's exactly why we're doing these workshops,

13	precisely why we're doing it.

14	So, let me show you what the

15	standards are so you can then tell me how you

16	think we should fix them.

17	AUDIENCE MEMBER:	Did I correctly

18	understand you to say that our district can

19	adjust the cut scores as necessary, based

20	on what this gentleman had said, and the data

21	that's been run.	So the district has that

22	permission?

23	MS. HEBDA:	They currently -- they did

24	that for '11-'12.

25	AUDIENCE MEMBER:	Not currently, next.
1	

1	MS. HEBDA:	The current school year,

2	'11-'12-'13, they can.	If this were to go into

3	effect, they would not be able to adjust them,

4	okay?	So, that's why we're proposing this.

5	There was a question about every

6	district doing something different.	I can't tell

7	if I go from one district to the next what I'm

8	doing, so that's why we're proposing this.	The

9	law says we should, and we're going to do that,

10	but we're providing these for your

11	consideration.

12	Remember that timeline.	This is us

13	putting this out here and you have time to give

14	us some more suggestions.	And I'm going to tell

15	you what some of those were if you let me turn

16	the page.

17	AUDIENCE MEMBER:	Well, while you're

18	turning the page, in our district last year when

19	we were creating our standards for the

20	negotiations, the way we established those

21	numbers were -- I would reach low, but that would

22	be crude -- picking numbers out of the air.	We

23	said this is what we think DOE would accept.

24	We have no reason to believe it's 60 percent,

25	but we think DOE will buy in, so that's what
1	

1	we're going to go with.	It was just made-up

2	stuff.

3	Your 95 percent confidence -- I'm

4	familiar with this.	I was at the webinar.

5	MS. HEBDA:	Great.

6	AUDIENCE MEMBER:	That 95 percent

7	confidence, well, where did that come from?

8	Where did it come from?	Was there any data --

9	I mean, you haven't run the data, so where did it

10	come from?

11	MS. HEBDA:	It comes from --

12	AUDIENCE MEMBER:	You're proposing a rule

13	and asking us to modify it, but the rule has got

14	no statistical background.

15	MS. HEBDA:	Well, I have think that we just

16	heard from the fact person back there on

17	statistics where the 95 percent interval comes

18	from.	Because what you're trying to do in the

19	system is be sure that you're making the right

20	judgment.

21	AUDIENCE MEMBER:	I understand statistical

22	math.	I'm a math teacher.

23	MS. HEBDA:	I know you do, but you asked a

24	question that other people -- that I need to

25	answer because you asked me a question.	So, I'm
1	

1	going to answer your question unless you tell me

2	it was just rhetorical, then I'll keep going.

3	We can do either one.	I can handle either one.

4	And, so, given your question:	This

5	is just a graphic to get you ready for what I'm

6	going to show you next.

7	And we're past six, so if you to have

8	to go, please don't feel bad.	I won't be at all

9	offended.

10	This is the score.	Remember, the

11	score is what happened.	That's what the diamond

12	represents.	Then using our standard error of

13	measurement that comes with each score, a lot of

14	which is based on the number of students that you

15	have in the store, the number of data points.

16	Then you can construct a 68 percent confidence

17	interval using one standard error plus, one

18	standard error minus -- remember, that plus or

19	minus three in the opinion poll -- and you can

20	be 68 percent confident that this would have

21	happened given any other set of circumstances,

22	or you can use two standard errors and you can

23	be 95 percent confident that this is where it

24	would be.	This is where your store would have

25	lied -- lain, whatever the word is.	Somebody
1	

1	in English help me.

2	AUDIENCE MEMBER:	Lied.

3	MS. HEBDA:	-- given a different set of

4	circumstances.

5	AUDIENCE MEMBER:	Fallen.

6	MS. HEBDA:	Fallen, yes.	Arrived.

7	If you had been given a different set

8	of students, a different set of circumstances.

9	So, given that information -- let's

10	talk about what the initial proposal is and the

11	additional things that we've already heard so

12	that you can get your juices flowing as you go

13	back and think about this.

14	Yes.

15	AUDIENCE MEMBER:	Using the '11-'12 growth

16	as a starting point --

17	MS. HEBDA:	Right.

18	AUDIENCE MEMBER:	-- is that dependent on

19	the State's maintaining the difficulty level of

20	the FCAT from year to year?	If, for example,

21	the seventh grade FCAT would become suddenly

22	considerably easier, to reach a higher scale

23	score on the seventh grade FCAT this year than

24	last year, would that follow things up?

25	MS. HEBDA:	Well, that's one of the
1	

1	questions we received.	Because in trying to set

2	the anchor, one of the questions one of our

3	attendees had -- I want to say it was in Lee

4	County, but it could have been Broward.	Don't

5	hold me to that.	You'll see the transcript at

6	some point.	There are different forms of the

7	FCAT given.	It's not identical.	There are

8	three or four, or whatever they are, for each

9	of those tests for all those psychometric reasons

10	why you do that, and test security reasons.

11	And there was a question in the mind

12	of this person, who is an assessment person,

13	whether or not they were equal enough and the

14	forms were the same enough so that that anchor

15	would work.	And, so, they said they would go

16	back and think about whether it would have to

17	have maybe '11 and '12 -- '11-'12 and '12-'13

18	data combined to be the anchor for the criterion

19	because you'd have two years of assessment data

20	to look at.	Maybe that would be a better idea.

21	So, again, we started with '11-'12

22	because we were trying to get this concept of the

23	anchor so that everybody knows what they're

24	shooting for and how to get better.	So, there

25	could be lots of reasons why you would like us
18	

1	to use a different anchor or add data to the mix

2	to make that a different anchor.	So, that's one

3	consideration that we've heard already so far.

4	Yes, ma'am.

5	AUDIENCE MEMBER:	I looked at the

6	value-added model and I applied it to our

7	district.	And with about 2,000 teachers that

8	would mean 50 would be highly effective and 50

9	would be unsatisfactory, and that would be the

10	sum of the evaluation regardless of what the

11	principal or any of the other staff --

12	MS. HEBDA:	Not necessarily, but go

13	ahead.

14	AUDIENCE MEMBER:	That's just my

15	understanding right now.	Here is my concern.

16	By 2013-'14 we don't have the ability to have

17	end-of-course exams in every subject.

18	MS. HEBDA:	That's right.

19	AUDIENCE MEMBER:	So when I sit here with

20	members of our school district, I have one

21	teacher who was evaluated based on kids in a

22	different school.	And even though she was the

23	teacher of the year and had a highly effective

24	rating, she dropped to unsatisfactory.	Another

25	teacher got the school grade and that school
1	

1	grade happened to be an A.	And, so, that

2	teacher doesn't teach FCAT kids but ended up

3	getting a highly effective.

4	So, without data that is truly linked

5	to the students that everybody teaches, how can

6	we use this formula because, quite frankly, if

7	I was at her school, I would be looking to get

8	out.	I would want to go to an A school --

9	MS. HEBDA:	Absolutely.

10	AUDIENCE MEMBER:	-- because that's where

11	I'm going to be highly effective, and I might

12	have a chance of being that 50 percent.

13	We have a teacher here in another

14	school that was an A school, however, she taught

15	all tenth grade classes.	And, so, her rating --

16	the principal was highly effective.	She became

17	effective with her rating; but, yet teachers at

18	her school that don't even step a foot into the

19	classroom -- like the guidance counselors and

20	the deans, they got a highly effective

21	evaluation and she actually did the grunt work

22	and was in the trenches teaching these kids and

23	she ended up effective.	So, this is not making

24	any sense to me because --

25	MS. HEBDA:	You're exactly right.
1	

1	AUDIENCE MEMBER:	-- it's so unfair and

2	inequitable.

3	MS. HEBDA:	You're exactly right.	So, if

4	I can just jump in for a second because what

5	you're saying is absolutely true.	And in the

6	rule, we don't apply it to those teachers

7	because they don't have that kind of

8	evaluation yet.

9	So, if I could talk about --

10	remember, we had the two reasons for doing the

11	standards, the one for consistency purposes for

12	just looking at that one criterion for

13	performance of students, and the other reason

14	where they would eventually become somebody's

15	entire evaluation, we've got conditions around

16	that second part.

17	So, if it's okay with you, let me

18	talk about what the standards are so people can

19	know what they're going to be giving feedback on

20	that, and then let me go to that second part,

21	when would that even happen for somebody,

22	okay?

23	AUDIENCE MEMBER:	Okay.

24	MS. HEBDA:	Because you're absolutely

25	right.
1	

1	First thing to know about the

2	diagram, remember zero meaning typical

3	performance.	Exactly what you'd expect students

4	and what students did across the state by grade

5	and subject, that's what that black line

6	represents.	So, let's start with the two ends

7	because they're easier to look at, and we'll

8	talk about what's in the middle.

9	So our standard for highly effective

10	would then be the value-added score itself --

11	remember the diamond in the middle, the score

12	itself, is above zero.	Meaning, on average,

13	that person's students performed above where we

14	expected and above the rest of the State.

15	And, in fact, if I construct that

16	95 percent confidence interval, I am 95 percent

17	confident that, given a different group of

18	students, that's exactly where that score would

19	lie still with even the bottom -- the bottom

20	part of where I thought that score could be is

21	still above what the State average was, okay?

22	That's highly effective.

23	Likewise, for unsatisfactory, the

24	score itself is below zero.	And, in fact, I am

25	95 percent confident that, given a different
1	

1	set of circumstances, even the highest possible

2	rating that teacher could have gotten for

3	performance of students would still be below

4	the State average, below zero.

5	Those are the two extremes.

6	We've also provided two definitions

7	for effective.	This one is not needs

8	improvement.	Both of these are effective.

9	The first definition would be the

10	score is above zero.	The actual value-added

11	score you earned that year is above zero.	So, on

12	average, this teacher's students performed

13	better that the State in that grade subject,

14	but the entire 95 percent confidence band doesn't

15	quite get above zero.

16	AUDIENCE MEMBER:	So, it's one standard

17	error?

18	MS. HEBDA:	In this category, it's two

19	standard errors.	It's still two standard errors

20	because this is two standard errors that's

21	totally above.	In this case, I construct my

22	two standard errors, and it's not highly

23	effective, but it's still effective.	Not quite

24	as confident as I am for highly effective.

25	The other possibility is the
1	

1	value-added score itself is really below zero.

2	When I add everything up together, I get a score

3	that's below zero.	However, when I construct

4	even a one standard error confidence level, even

5	that one standard error still goes above zero.

6	So, there's a good chance that this score, if I

7	had been given a different group of kids, that

8	the performance would have been possibly above

9	the State average.	So, any of these could be

10	effective.	So your effective group is always

11	going to be your biggest group.

12	Now, the way we've done this --

13	because the law says we have to do

14	unsatisfactory, effective and highly effective --

15	because you're not provided an initial definition

16	for needs improvement or developing, that would

17	be what doesn't meet these standards.

18	So, one of the things we've received

19	a suggestion on is go ahead and provide it.	We

20	know it's the remainder, but go ahead and define

21	what the remainder is.	So, go ahead and provide

22	a definition for needs improvement.

23	The other thing we received a

24	suggestion on is -- because there's a district --

25	Lee County uses this exact same scheme for their
1	

1	performance of students.	But somebody else

2	suggested that we create a second bar and create

3	a second definition for highly effective, just

4	like we have two for effective.	That we keep

5	that one with the 95 percent confidence interval,

6	but then also have another bar that even if I'm

7	not -- even if I'm this teacher here where I'm

8	not 95 percent confident my score might have

9	been above zero, if my score is so high to have

10	exceeded this second bar, then give me credit

11	for that value-added score for that year.	Don't

12	worry about how confident it would have been

13	if somebody else and whatever else had happened

14	because if the score is that high, make it

15	highly effective.

16	So, we have the State average, but

17	then we would have a separate cut point that

18	you could exceed for highly effective regardless

19	of what your standard error of measurement was.

20	Yes, sir.

21	AUDIENCE MEMBER:	Is that to account for,

22	like, the elementary school teacher with a

23	very limited sample size --

24	MS. HEBDA:	That's right.

25	AUDIENCE MEMBER:	-- who has a huge
1	

1	standard error, so he would have to score like

2	a 3.7 --

3	MS. HEBDA:	That's right.

4	AUDIENCE MEMBER:	-- to keep his two

5	standard errors above --

6	MS. HEBDA:	That's exactly right because

7	you don't -- again, you're trying to be confident

8	that you're making the right judgment.	You want

9	to use the statistics wisely to make a good

10	judgment.	You don't want to ignore actually

11	what happened.	If that's what happened, you're

12	getting credit for it.

13	AUDIENCE MEMBER:	To make sure I

14	understand this picture, though, the way you're

15	describing the trigger -- I'm going to call it

16	trigger -- for unsatisfactory, you must -- if

17	you fall under this, you must.	That bar there

18	actually could be way higher as long as that

19	yellow tip --

20	MS. HEBDA:	Doesn't hit this.

21	AUDIENCE MEMBER:	Does it hit zero?

22	MS. HEBDA:	No.

23	AUDIENCE MEMBER:	Go above zero?

24	MS. HEBDA:	It doesn't hit zero.	So, I

25	think we said effective was zero or above, and
3	

1	everything else -- this would be below zero.

2	AUDIENCE MEMBER:	So, if I had a VAM score

3	of minus .5.

4	MS. HEBDA:	Okay.

5	AUDIENCE MEMBER:	No.	Minus .2 --

6	MS. HEBDA:	Okay.	Which means 20 percent

7	below State average.

8	AUDIENCE MEMBER:	-- and my standard

9	error was .1, adding two standard errors of that

10	gets me to zero.

11	MS. HEBDA:	That's right.

12	AUDIENCE MEMBER:	Do I avoid the trigger

13	or --

14	MS. HEBDA:	Yes.

15	AUDIENCE MEMBER:	I avoid the trigger.

16	MS. HEBDA:	That's right.

17	AUDIENCE MEMBER:	But if it's .9, I

18	wouldn't.

19	MS. HEBDA:	Now, let me talk about the

20	trigger for a second.	I'm sorry to use that

21	word, but you already did, so --

22	AUDIENCE MEMBER:	It is.	It is.

23	MS. HEBDA:	-- let's talk about --

24	Can I do that first, do you mind?

25	AUDIENCE MEMBER:	Yes.
1	

1	MS. HEBDA:	I appreciate that.	I'm sorry.

2	It's already 20 minutes after six.	I know

3	y'all want to go home.

4	These are the standards we propose.

5	You heard we've got other suggestions of how we

6	can do a second for highly effective, maybe

7	even a second one for unsatisfactory, you know,

8	for those situations you just described.	Maybe

9	there's another lower bar you want to set for

10	unsatisfactory, not just the State average.

11	Those are the kinds of suggestions we want.

12	They're very, very important.

13	Now, let's talk about when they

14	become triggers because we're proposing these to

15	be the standards you would use for just

16	performance of students, just how everybody would

17	calculate that information, the same across

18	districts.

19	But what would have to happen for a

20	person to actually -- for this to be their

21	whole evaluation?	First thing is exactly what

22	you were describing before.

23	AUDIENCE MEMBER:	I moved over.

24	MS. HEBDA:	I thought where did she go?

25	I finally got to her part and she left.
1	

1	The first thing that would have to

2	happen is we would need to have three years of

3	data for that person.	It couldn't be based on

4	one year of data or two years of data.	You have

5	to have three years of data.	And that three

6	years of data could not be on a temporary

7	measure.	Like you're talking about being

8	measured on schoolwide, kids you don't teach.

9	If you have a year of data in your

10	three years that's one of those, it couldn't be

11	a trigger for you.	It would just be the data

12	that you have that you're using in the

13	evaluation to calculate performance of students.

14	Okay, that's the first concept because we have

15	to have the standards.	According to the law,

16	we got to make them.

17	So, we tried to put conditions under

18	which they would even count for anybody.	And

19	we also are suggesting that they would not

20	count for nonclassroom instructional personnel

21	because only part of their evaluation is even

22	based on statewide student data and they don't

23	actually teach those classes.	So, our

24	suggestion is they would not count for

25	nonclassroom instructional personnel.
1	

1	AUDIENCE MEMBER:	Only a certain

2	percentage of teachers would even be eligible

3	to be highly effective.

4	MS. HEBDA:	No.	You could still be

5	highly effective based on your whole evaluation.

6	AUDIENCE MEMBER:	Without the test

7	student part?

8	MS. HEBDA:	They would have the student

9	test part, but that would be combined	with

10	instructional practice, however your district

11	does it, and then you would have your subject

12	grade, just like it was in '11-'12.	It just

13	wouldn't be automatic.

14	AUDIENCE MEMBER:	Wouldn't the trigger

15	only apply if you meet all those conditions?

16	MS. HEBDA:	That's correct.	Only if you

17	meet all these conditions would it be

18	automatic.

19	AUDIENCE MEMBER:	A guidance counselor

20	doesn't worry about this trigger ever?

21	MS. HEBDA:	That's correct.

22	AUDIENCE MEMBER:	And somebody who is only

23	in their second year doesn't worry about the

24	trigger or (inaudible).

25	MS. HEBDA:	And somebody who is being
1	

1	evaluated on temporary measures, like team data

2	and things like that, doesn't worry about that.

3	Now, it doesn't mean that the district

4	shouldn't set comparable standards for those

5	other things.

6	So, if, for example -- we talked

7	about this and now I'm going to point you to the

8	rule to look at this because I want you to see

9	the language that we used for the district

10	standards.	If you look at .0411 --

11	THE COURT REPORTER:	I'm going to need

12	to change my paper.

13	MS. HEBDA:	Okay.

14	(Off the record while reporter changes

15	her paper.)

16	THE COURT REPORTER:	Okay.	I'm ready.

17	AUDIENCE MEMBER:	Are these standards

18	being proposed literally to become adopted by

19	all districts?	I mean, we are one of those

20	eight you referred that instead of getting into

21	standard deviations, we said if a certain

22	percentage of our students meet or exceed our

23	learning gains, they get an effectively (sic),

24	highly effective, et cetera.	That's out the

25	window if we go to this; am I correct?
1	

1	MS. HEBDA:	That's right.	Every district

2	will be on the same scale.	If we go to this, the

3	same scale, the same cut scores.

4	AUDIENCE MEMBER:	And that will be

5	mandatory?

6	MS. HEBDA:	If the rule goes into effect,

7	it will be mandatory.	The law requires us to set

8	them.	We haven't set them yet.	We've been

9	letting people do their thing.

10	AUDIENCE MEMBER:	This is '13-'14 we're

11	talking about?

12	MS. HEBDA:	'13-'14, yeah.	It will change

13	nothing for this year.	Even if it were to be

14	adopted before the year's over, doesn't change

15	anything for '12-'13.

16	Yes, sir.

17	AUDIENCE MEMBER:	What prohibits the DOE

18	from allowing districts to do their own thing

19	forever?	Is there something in this law that

20	says --

21	MS. HEBDA:	Yes.	That's the page earlier

22	where I said that's the requirement for the law

23	that we do this.	The law says we have to do it.

24	There are people who ask us every

25	day, how come you haven't already done it?
1	

1

2	well?

3

AUDIENCE MEMBER:	Is it charter schools as

MS. HEBDA:	It will be charter schools

4	as well, correct.

5	And you had a question before I go

6	to this?

7	AUDIENCE MEMBER:	Yes.	Just a

8	clarification.	If you look at your graphic, if

9	you look at the very bottom, the distance from

10	the diamond to the top of the red, whatever

11	length that is, say, that far --

12	MS. HEBDA:	Yes.

13	AUDIENCE MEMBER:	-- the yellow part above

14	that should be exactly the same length?

15	MS. HEBDA:	It probably should, yeah.

16	Sorry.	Bad graphics.

17	AUDIENCE MEMBER:	The yellow one on the

18	bottom is a little bit shorter than the yellow

19	at the top.	But each is supposed to represent

20	one standard error above that and then two

21	standard errors.

22	MS. HEBDA:	That's correct.

23	AUDIENCE MEMBER:	There's nothing magic

24	about one standard error and two standard

25	errors --
4	

1	MS. HEBDA:	Except for the degree of

2	confidence.

3	AUDIENCE MEMBER:	We commonly talk about

4	that.	And, commonly, we talk about 95 percent

5	confidence interval and 68 percent interval.

6	MS. HEBDA:	That's right.

7	AUDIENCE MEMBER:	But there's no reason

8	why we couldn't use a 50 percent confidence

9	interval or an 80 percent confidence interval.

10	Instead of using one standard error above the

11	red and one standard error below, you could use

12	.75 standard error above and .75 below.

13	MS. HEBDA:	Yes.

14	AUDIENCE MEMBER:	And then .80 above and

15	.80 below.

16	MS. HEBDA:	Correct.

17	AUDIENCE MEMBER:	Doing so might get that

18	third bar from the right to where it would be

19	entirely above the line and that might increase

20	the number of highly effective.

21	So, if it were determined that too

22	few teachers, for example, were highly effective

23	based on the rule you're doing, one way you

24	could get more highly effective would be simply

25	to change from one and two standard errors to
1	

1	something like --

2	MS. HEBDA:	One-and-a-half.

3	AUDIENCE MEMBER:	-- one-and-a-half or

4	one seventy-five.

5	MS. HEBDA:	Absolutely.

6	AUDIENCE MEMBER:	You might want to keep

7	the 95 confidence interval or the unsatisfactory

8	to be absolutely certainly before you categorize

9	somebody that way that is 95 percent confident --

10	(Inaudible).

11	MS. HEBDA:	Very good statement.

12	Yes.

13	AUDIENCE MEMBER:	Can you help me

14	understand why you have used two standard errors

15	for identifying highly effective or

16	unsatisfactory and, yet, used one standard

17	error, which is 68 percent degree of confidence,

18	which I don't believe any research would accept

19	that as any specific conclusion?	And why use

20	one standard error for effective where you could

21	use two and increase the probability of

22	including more people in the effective band

23	because of the severe negative career

24	consequences associated with being identified

25	as a needs improvement teacher?
1	

1	MS. HEBDA:	You could make --

2	AUDIENCE MEMBER:	What's the logic?

3	MS. HEBDA:	You could make that

4	suggestion.

5	We did this based on the fact that

6	this score was actually below zero.	But, if you

7	think that being below zero, further below zero

8	is okay for effective, then make that suggestion.

9	Make that suggestion.

10	We're going to run data on simulations

11	and lots of different things, not just these.

12	So, we'll take the suggestions and run those and

13	see how it turns out and we'll go from there.

14	AUDIENCE MEMBER:	Can you give us an idea

15	-- because we're talking about standards in the

16	abstract and scores in the abstract.	What might

17	be a typical standard error for, say, a

18	secondary teacher with 125 students, all of whom

19	are accounted for in this versus, say, an

20	elementary schoolteacher with 18 for three years?

21	So, that's 54.

22	MS. HEBDA:	That's a great question.

23	AUDIENCE MEMBER:	What is the size of it?

24	MS. HEBDA:	I don't know the numbers off

25	the top of my head.
1	

1	AUDIENCE MEMBER:	Now, are these numbers,

2	like, .2, .3 and the standard is .1 or .5, or

3	do we have any clue?

4	MS. HEBDA:	We do have a clue.	I just

5	don't have it off the top of my head.	You'll

6	find that in the technical report.

7	Now, what I will tell you is:

8	There are some standard errors that are so large

9	that you don't even want to use that score.

10	But, then, at the same time, then you feel like

11	you're stuck with the confidence band and you

12	don't want to acknowledge what actually might

13	have happened with that teacher even though it

14	was just a few students.	So, that's why the

15	different definitions, or maybe a second

16	definition for some of these are going to be

17	important because you want to be confident, you

18	want to be sure, as we said.	But, you know,

19	there is some research that says you want to be

20	95 percent confident about everything.	Other

21	research says depending on what you're doing, 68

22	is pretty darn good.	It depends on what you're

23	looking at and who you're reading.

24	But, this is for a person's

25	evaluation.	So, one of the things that we've
1	

1	discussed is we know most people are going to be

2	in the middle somewhere.	That's what you would

3	expect.	Absolutely you want that.	You expect

4	that in the bell curve, and it's everything

5	else.

6	So you want to be most sure when

7	you're talking about the top category and the

8	bottom category.	And it may be that you want to

9	be extra, extra sure for needs improvement, and

10	extra, extra, extra sure for unsatisfactory.

11	So, what does that extra, extra,

12	extra mean?	95 percent confident, or could it

13	be 95 percent confident, or could it be a really

14	low bar below the State average?	Maybe we could

15	have multiple definitions for that -- I would

16	have to meet both of those to be unsatisfactory,

17	for example.	Those are things that we can

18	always consider.	We need to know what you all

19	think about that and what your data looks like

20	in the districts.

21	Yes, ma'am.

22	AUDIENCE MEMBER:	One thing for possible

23	consideration when you talk about the extra

24	sure and the extra, extra sure.	Given what

25	would happen if someone gets unsatisfactory and
1	

1	that trigger for what occurs overall in the

2	number of either two or three years or two years.

3	I would like to suggest you possibly consider

4	for unsatisfactory and needs improvement three

5	standard errors going.	From two to three only

6	takes you to 95 to 99.7.	And while that's only

7	a little bit, it's not a little bit to that

8	person who may end up losing their job and have

9	a career adjustment as a result of data that had

10	some variability that was not accounted for;

11	such as, people coming into your classroom on

12	February 15th and entering in your classroom

13	and taking the FCAT February 19th.	And you can't

14	account for that, and we know that.

15	MS. HEBDA:	Right.

16	AUDIENCE MEMBER:	But going out to three

17	standard errors could certainly help to go a

18	long way to make sure that we're not putting

19	people in categories in which they don't belong.

20	MS. HEBDA:	Thank you.

21	Yes, ma'am.

22	AUDIENCE MEMBER:	I'm a little concerned

23	on using the standard error on using FCAT, FCAT

24	math has a lot lower standard error than FCAT

25	reading.	And you run the data -- if any
1	

1	district runs their data -- like, when we got

2	our data -- we're in a very large district -- it

3	only had, like 2,000 students -- or 2,000

4	teachers that had three years of data.	And 17

5	percent of our math teachers, only 500 teachers

6	was that approximately, were either highly

7	effective or unsatisfactory.	A much smaller

8	group of reading teachers could be classified

9	the same way because just the nature of the

10	error in the test.

11	MS. HEBDA:	That's right.	And that is

12	true on almost everybody's reading test,

13	national reading tests, and in other states as

14	well.

15	AUDIENCE MEMBER:	If you're a reading

16	teacher, you could remain hidden, and that's a

17	concern.

18	MS. HEBDA:	That's right.	So, I think

19	that's important when you're looking at the

20	suggestions to make for us when you're looking

21	at your data.

22	I would also -- because of the fact

23	that they were trying to set performance

24	standards using -- and I'm going to get to the

25	district ones in just a second.	They were
1	

1	trying to use the performance standards that

2	would work for this model.	We have Algebra I.

3	We'll have to set separate performance

4	standards for Algebra I because that's a

5	different test.	So, we're going to need

6	different ones for Algebra I, too, because

7	that's not the same as this.

8	AUDIENCE MEMBER:	When you say that, why

9	would you not set different standards for FCAT

10	reading and FCAT math?

11	MS. HEBDA:	We could.	We absolutely

12	could.

13	AUDIENCE MEMBER:	The tests are performed

14	very differently.

15	MS. HEBDA:	We could do that.

16	Yes, ma'am.

17	AUDIENCE MEMBER:	I still need your help

18	in you explaining how a nonFCAT teacher can

19	be highly effective.	I still don't understand

20	that.

21	AUDIENCE MEMBER:	Like an art teacher.

22	MS. HEBDA:	Let's go to the district

23	standard piece in the rule, and then we'll go

24	from there.	Because, in truth, the art teacher

25	should -- I know that there are people who are
1	

1	getting schoolwide data and stuff because that's

2	professional teaching, but there are teachers

3	that are supposed to be looked at for art

4	purposes for teaching art.

5	I used to teach music.	I love

6	music.	I want to know what is happening in my

7	music class.

8	So, if the district has assessments

9	for seventh grade social studies, for example,

10	which is not FCAT, then the performance of

11	studies, if I taught social studies all day long

12	-- just to make the example simple, although not

13	everybody does that -- then my performance of

14	student criteria would be based on how my

15	students did in seventh grade social studies,

16	which means the district's local assessment they

17	used, whether they use the adjustment method,

18	whether they use a consortium of districts that

19	use the same assessment, whatever that is, then

20	the district will set standards for performance

21	on seventh grade social studies.

22	Now, how do those become comparable

23	to what we've done in the rule for the FCAT,

24	that gets to what I was trying to show you in

25	the rule and what those possibilities could be.
1	

1	AUDIENCE MEMBER:	Since you're asking for

2	comments --

3	MS. HEBDA:	Yes.

4	AUDIENCE MEMBER:	-- my comment is that

5	we need to extend the timeline on this until

6	districts have evaluations for everybody in this

7	subject and in the area that they teach.

8	I'm really fearful that we're jumping

9	into this thing very early.	You know, we've set

10	a deadline of 2013, which is not that far away.

11	I don't know how our district is going to have

12	the capability of being able to have

13	end-of-course assessments for every single --

14	we're working on them.	You know, we've looked

15	at the data bank.	We're trying to do the best

16	we can, but I don't think it's a reality that all

17	of the districts in the state of Florida are

18	really going to be there at that year.

19	And, so, my suggestion is that we may

20	have to move this forward a couple of years and

21	give the districts time to really have a valid

22	assessment of each individual teacher, what they

23	teach, what they do.	If you're a guidance

24	counselor, use the Florida School Guidance

25	Counseling Standards.	We have technology
1	

1	programs in place.	Our teachers in those areas

2	already have certification tests, but there are

3	so many courses that we're just not going to be

4	there.

5	And I just think it's just so unfair

6	for somebody like Kim Cook, who teaches first

7	grade, to have her VAM portion determined by kids

8	that she has never even seen.

9	MS. HEBDA:	I agree with that a hundred

10	percent.

11	AUDIENCE MEMBER:	So that's my suggestion.

12	MS. HEBDA:	Would it help if, in the

13	meantime -- and you can tell me yes or no, or

14	think about it and let us know over the web or

15	whatever.	But the reason why that happens is

16	there's that sentence in there that says

17	instructional team data can be assigned.	And,

18	you know, does that need to go away?

19	AUDIENCE MEMBER:	Yes.

20	MS. HEBDA:	Does it need to become not an

21	option?

22	AUDIENCE MEMBER:	What do you think?

23	AUDIENCE MEMBER:	My school is primary.

24	It's just K, 1 and 2.	We don't have any

25	children that take the FCAT.	What data is there
1	

1	for my kids?

2	MS. HEBDA:	Your district should at some

3	point -- I can't speak for every single district

4	individually because I don't work in the

5	district.	So, I'm not going to say your district

6	better do this, or better do that, or anything

7	else.	But the whole thing is constructed so that

8	you be evaluated on what it is that you're

9	responsible for teaching.

10	And every day right now in seventh

11	grade social studies and in first grade

12	classrooms and in kindergarten classes and pre-K

13	and everywhere else, kids are taking assessments,

14	they're doing projects, they're doing all kinds

15	of things, and they're earning grades, they're

16	earning GPAs, they're getting promoted, they're

17	getting retained based on people's progression

18	plans.	All those things are going on.

19	So, some of those things are not the

20	things that a district wants to use, also, for

21	evaluation purposes.

22	So, when we talk about creating

23	assessments to evaluate teachers, assessments

24	should be created and adopted and identified

25	so that you're assessing mastery of the content
1	

1	of the course with the student.	And that's what

2	the law says you should do.

3	And, then, the law says you have a

4	step two when it comes to evaluating teachers

5	and principals and other non -- or other

6	classroom -- or other instructional personnel

7	to say, from all those things that you do to

8	assess mastery for students, whether it's

9	a lot of interim assessments, school kids, or

10	it's one final exam, or whatever it is, that

11	you should choose from those things what you're

12	also going to use for evaluation purposes.

13	That's where you choose it from.	You don't make

14	new stuff.	Maybe you have to make something new

15	because you're not satisfied with, but there

16	shouldn't be a separate thing that's done just

17	for evaluating purposes.	It should be what it

18	is that works for that course -- the goals for

19	the course, the learning goals for the course,

20	the State-adopted course description; all those

21	things.

22	The struggle, I think, that people

23	are having right now is saying, what I do over

24	the normal course of work for the teacher to

25	assign grades in these classes, I don't feel
1	

1	comfortable for whatever reason.	There's a

2	list of reasons.	And using that same

3	information for evaluation purposes.

4	Now, sometimes districts do find a

5	way to use information, like third grade results,

6	but there's no -- not in the VAM formula because

7	there's no pretest for that, but they use third

8	grade results based on the number of students

9	that went to grade level, or number of students

10	proficient, based on its district averages and

11	things like that.

12	Some districts have found a way to

13	use more data than less, other districts have

14	not.	They've used the team sentence.

15	So, one of the things we've talked a

16	lot about with the new Commissioner -- this is

17	not in the rule, but is -- and districts have

18	done this on their own as well, formed groups

19	of districts to share resources, share ways to,

20	not just look at assessments themselves, but how

21	are they going to capture assessment data for

22	evaluation purposes -- that's another issue --

23	to make sure all the data is accurate.

24	So, you got all kinds of data you

25	have for your kids and it's in your kid's
1	

1	portfolio and it's in your grade book, which

2	could even mean an electronic grade book, but

3	that's not necessarily the same as the districts

4	having that data to then use some of it for

5	evaluation.

6	So, all of those are big

7	considerations that districts are wrestling with

8	right now.

9	AUDIENCE MEMBER:	Am I wrong, though, to

10	be under the premise that it's to be a statewide

11	assessment?

12	MS. HEBDA:	It doesn't have to be a

13	statewide assessment.

14	AUDIENCE MEMBER:	If I spend my whole day

15	teaching AP, I should be graded according to

16	my AP scores --

17	MS. HEBDA:	That's correct.

18	AUDIENCE MEMBER:	-- and not according to

19	what my school is.

20	MS. HEBDA:	That's correct.

21	AUDIENCE MEMBER:	I have kids that the

22	only time I see them is if they pass by my room

23	to go to the bathroom.

24	MS. HEBDA:	That's right.	You should be

25	evaluated on AP data.
1	

1	AUDIENCE MEMBER:	Which, then, doesn't

2	come out until July.

3	MS. HEBDA:	That's right.

4	AUDIENCE MEMBER:	And AP data can't really

5	be used for growth because there is no scale

6	score.	It's a one, two, three, four or five,

7	and it's going to be an achievement score

8	basis.

9	MS. HEBDA:	That's right.

10	AUDIENCE MEMBER:	So, some of us are

11	going to be held to an achievement basis and

12	some of us are going to be held to a growth

13	basis and the State law says that it's supposed

14	to be based on growth.

15	MS. HEBDA:	The State law says that the

16	district has the option, as I said earlier,

17	to use an achievement measure because in some

18	courses, that's more appropriate.

19	What's the goal of the AP course?

20	To pass the test, right; how many kids can you

21	get to pass the test and get college credit.

22	AUDIENCE MEMBER:	Right.

23	MS. HEBDA:	If that's the goal of your

24	course, why would you not be evaluated on that?

25	AUDIENCE MEMBER:	That's what I'm saying.
1	

1	MS. HEBDA:	Now, a kindergarten teacher --

2	AUDIENCE MEMBER:	I'm one of those

3	mid-level teachers.	You're going to lose me to

4	another state.	You are.	You are going to lose

5	me.	I will leave.

6	MS. HEBDA:	That's an important

7	consideration.

8	Now, a kindergarten teacher would

9	not feel the same way.	A kindergarten teacher

10	would not want to do that.	There are different

11	sets of assessments using kindergarten.

12	And, so, you don't need to make -- by

13	making everybody identical, meaning everybody

14	has got to have a fancy covariate adjustment

15	model, that's not making it more fair.	To make

16	it more fair, you have to go back to what it is

17	you're responsible for teaching, what are the

18	instructional goals for the course and what is

19	your job for teaching those kids.

20	So, one last thing before we go,

21	because it's almost 7:00, if you would please

22	look at Page 7, district level standards.	Paren

23	C on Page 7 -- if you can count over to Page

24	7.	How would a district use information that

25	we get from the model to make comparable
1	

1	standards.	There are a couple of things that

2	we talked about.	We talked about the fact that

3	-- if you look under C, you'll see that there's

4	a one and a two and there's a sentence that's

5	flush left over to the side.	Do you see that?

6	The first thing, here is a way a

7	district can set comparable standards for a

8	course where they have the assessment.	Could

9	be AP.	Even though the AP is a national

10	assessment, it still functions as an

11	end-of-course test the district could use the

12	state assessment.

13	So, the standard must reflect to an

14	equal amount of student learning growth for a

15	similar grade level and subject as established

16	by the State.

17	So, one of the things that we know

18	from here, from the value-added model, is we know

19	that the aggregate score is converted to a

20	proportion of a year's growth.

21	So, for example, some of the data

22	we published recently -- it's still on our

23	website -- is that mathematics' teachers across

24	the State -- and this gets to your point about

25	the reading assessment, I think -- mathematics'
1	

1	teachers across the State, the ones that actually

2	earn a VAM score, not the schoolwide and all that

3	kind of thing, but actually taught those courses

4	associated with FCAT, those that were rated

5	highly effective by their districts, their

6	students, on average, grew about 20 percent more

7	than the State average.	So, the average VAM

8	score for those teachers was .19.	That's about

9	20 percent higher than an average year's growth

10	because we convert that to a proportion of a

11	year's growth.

12	So, one of the things the districts

13	could do with this sentence is say, okay, I know

14	in that year -- they could divide it up by how

15	the district average was, average year's growth,

16	per ninth graders, or math students, or any of

17	those kinds of things and, say, find a standard,

18	whether it's the State standard or the district

19	standard and, say, okay, in the State -- in the

20	State assessment value-added result, highly

21	effective equated to 20 percent above State

22	growth.	So, that's a standard I'm going to set

23	for highly effective in any class where I'm

24	measuring growth for a pre and post test, for

25	example.	That's one way a district can do it.
1	

1	Set whatever the State highly effective turns

2	out to be and say that's where I'm setting my

3	cut point to.	Same thing for effective,

4	average growth, if that is what the State

5	standard is, then average growth for pre and

6	post test courses that I use, that's where I'm

7	going to set my standard for effective.	They

8	could do that all the way through.

9	Now, what about those courses we're

10	not going to measure growth?	We're going have

11	to a performance or an achievement standard.

12	Look at number two.	The standard

13	must result in a comparable number or percent

14	of students who meet or exceed their expected

15	growth.

16	Remember, we said from the

17	value-added model, one of the things we get is

18	the number of percent kids who exceeded their

19	expected scores by classroom and by district.

20	So, a district could say if, on

21	average, all these teachers that were rated

22	highly effective, if, on average, their students

23	in their classes -- if 75 percent, let's say, of

24	their students, on average, met or exceeded

25	their expected growth, then I could set a
1	

1	standard of highly effective of 75 percent for

2	performance standard.

3	So, you could use the results from

4	the State model that covers a large segment

5	of teachers on a State assessment and you could

6	make those standards -- and you can set

7	comparable standards in your district using

8	those.	You can also do the same thing by taking

9	your own district's value-added results in those

10	same things and make it a district comparable

11	level standard for those courses as well.	It

12	doesn't say which one, so you could do either.

13	AUDIENCE MEMBER:	And that could be

14	different from subject to subject?

15	MS. HEBDA:	It could because we have

16	kindergarten and we have AP.

17	AUDIENCE MEMBER:	No, I'm talking about,

18	for example, AP chemistry with a low historical

19	national pass rate versus A plus (ph) that has

20	a high.

21	MS. HEBDA:	That's right.

22	And, actually, Santa Rosa does that.

23	Santa Rosa does that already.	They used that

24	in '11-'12, Santa Rosa County.	They have been

25	giving AP geography for whatever, and they knew
1	

1	what those kids were going to do.	They knew what

2	the average scores were for AP geography.	And,

3	so, they set cut points using that average and

4	then set a higher cut point for highly effective.

5	Then, they were going to be offering AP

6	calculus for the first time that year.	And they

7	knew it wasn't going to turn out the same way as

8	AP geography.

9	So, what they did was, not having

10	taught it before, they looked at the national

11	pass rate but they also took a step back and

12	said how are those kids doing in regular calculus

13	in my school, how do they traditionally do?	And

14	they took a look at that data and the national

15	standards to set different cut points for

16	effective and highly effective growth for AP

17	calculus because it was the first time they were

18	ever offering it.	And that made more sense

19	instructionally and it may more sense for the

20	evaluation, and that's what they did.	So, that's

21	an option.

22	Now, that's a lot to keep up with,

23	though because what you're looking at is the

24	learning goals for those courses, but that is

25	something you could do.
1	

1	AUDIENCE MEMBER:	Now that they have some

2	data, would they adjust it?

3	MS. HEBDA:	I imagine they do because

4	Santa Rosa is all in the weeds with their stuff.

5	Now, let me say one quick thing before

6	we do questions.	There is that third sentence

7	there there's flush left underneath.	Flush

8	left sentence means it modifies everything in

9	that little section.

10	So, if number one and number two

11	don't work, I've got this other course, I've got

12	this special population of kids, I got whatever

13	I got, and I can't use number one and I can't use

14	number two, a district could suggest another way

15	they want to use to set a comparable growth

16	standards and just provide us the instructional

17	reason for doing so.	And we said

18	instructional reason because we're trying to

19	keep hammering back on the fact that this needs

20	to be based on the learning goals for that

21	course and expectations for the kids in that

22	course.

23	Question.

24	AUDIENCE MEMBER:	Just very quickly,

25	before any more people leave, could you possibly
1	

1	give us the link that you told us for the input?

2	MS. HEBDA:	Yes.	It's right on the front

3	of your PowerPoint, actually.	That very first

4	thing.	The slides at the very front under giving

5	input and participation.	Right here at the

6	bottom.	That's where all the rule information

7	is provided.	And also right there you'll see we

8	want your comments in big red letters.	You just

9	click that e-mail address and the e-mail

10	window pops up and you can send us your comments

11	there.

12	Yes, ma'am.

13	AUDIENCE MEMBER:	As far as the rule for

14	the VAM goes, is there some sort of verbiage

15	in here that says that -- for the appeal process,

16	that they have to a supply an appeal process in

17	a certain amount of time?

18	MS. HEBDA:	No, there isn't.	The law sets

19	forth the whole process for how people have to

20	be notified, and all those kind of things, but

21	we haven't elaborated on that in the rule.

22	AUDIENCE MEMBER:	Because I'm wanting to

23	appeal my score.	I had 44 students and only

24	23 showed up and my score was a certain amount.

25	I said I wanted to appeal, and all of a sudden
1	

1	17 more students showed up and my score never

2	changed.	So, I want to appeal it, basically,

3	because I want to understand it.	They're not

4	telling me how to appeal it.	They're just

5	saying, okay, we'll appeal it.

6	I'm one of those teachers that are

7	leaving because of this.	I'm going to North

8	Carolina.	But I would like to have this all

9	settled before I go.

10	Is there some sort of ruling or

11	something that they have to have it settled by

12	a -- one year later?

13	MS. HEBDA:	No.	Well, there is a timeline

14	provided in the law for when a teacher is

15	determined to be unsatisfactory, and the teacher

16	has a timeline to get back for that, but it

17	doesn't say anything about appeal on any kind of

18	result for evaluations.

19	AUDIENCE MEMBER:	My scores were fine.	I'm

20	not appealing my scores.	I'm appealing the

21	process.	I'm appealing on how they come up with

22	my scores.

23	MS. HEBDA:	Then, that should be in the

24	contract, shouldn't it?

25	Yes.
1	

1	AUDIENCE MEMBER:	What contract?

2	MS. HEBDA:	Your collective bargaining

3	contract should set forth what your appeal

4	process is for grieving the actual process.

5	Yes, ma'am.

6	AUDIENCE MEMBER:	So I had three years --

7	I don't understand my whole VAM.	One year was

8	a fifth grade with 27 children, which was over

9	the class size.	And I had a huge amount --

10	because my school is in a high-crime,

11	high-drug, high-poverty area, so I had a huge

12	amount of discipline referrals and students that

13	had no concern or care for learning.	It was like

14	a gang mindset and I felt like a juvenile

15	detention officer more than a teacher, ever.	And,

16	so, the very next year I went to a third grade

17	class and I had 13 students.	They split the

18	class because it was low, and we had to lose

19	another teacher.	So, I had third grade.	Those

20	students were six weeks, maybe two months, and

21	then I went to second grade.	And those are my

22	kids.	And I don't understand how any of that

23	reflects me as a teacher and my ability, or the

24	kids, when a few years back I was at a different

25	school and my FCAT scores were the highest in
1	

1	the county outside of a gifted.	So, I'm just

2	really kind of heart broken and don't understand

3	where -- why, let's say, discipline referrals

4	wouldn't count, how many times the police have

5	been called on some of these children.	I don't

6	know why that doesn't fit into this at all.

7	I can understand socio-economic

8	status because I know that not all are -- are

9	stunted -- I guess, that's not it.	There's got

10	to be some accountability somehow somewhere

11	for a school like that.

12	MS. HEBDA:	Right.	There are a couple

13	of --

14	AUDIENCE MEMBER:	Or else --

15	MS. HEBDA:	I'm sorry?

16	AUDIENCE MEMBER:	Or else if they're

17	getting assessed and that doesn't count, who is

18	going to want to teach there?

19	MS. HEBDA:	That's right.

20	I'll tell you a couple of things.	The

21	first thing, if I understood that whole scenario

22	correctly, it sounded very similar to the five

23	different teachers for the one group of students.

24	It just sounds like there's an exception.

25	But, if there weren't all of that and
1	

1	you stayed in that school that had the really

2	difficult circumstances to teach in, the only

3	thing that the model accounts for currently --

4	because the student growth committee thought

5	about discipline data, and they discussed that in

6	one of their meetings, and you can kind of see

7	that in the white paper that chronicles things

8	that they used.	And they were concerned about

9	the fact that discipline data is not reported

10	consistently across the State.	There was also

11	a concern from -- I want to say it was even

12	Hillsborough, too, because Hillsborough started

13	their evaluative process even before the State

14	did.	They were concerned that they didn't want

15	anything in the model that would really be in a

16	teacher's control to do, and they weren't

17	positive that discipline referrals were always

18	in or out of a teacher's control.	So, that was

19	brought up by the committee.	But the committee

20	did say that one of the things they wanted to do,

21	even though they started with this set of

22	covariates in the beginning, that if there were

23	better ways -- like somebody mentioned RTI data

24	one time, too, and they wanted to collect --

25	you know, districts kind of report some RTI
1	

1	data, but it's not consistent, but if we could

2	get a consistent way to include those things,

3	that we would start including those in the

4	model and see how they worked.

5	The second -- the thing the model

6	does control for right now and does include is

7	the school component.	Most schools are somewhere

8	around in the middle, but there are some schools

9	in the State that are quite, quite, quite

10	different.	And there are schools that are very

11	high-growth schools, and schools that are very

12	low-growth schools.

13	And, so, one of the reasons for

14	removing -- for looking at the school component

15	and considering that as another covariate,

16	essentially, is because of the differences in

17	the school.	And the committee really believed

18	that there were going to be factors in the

19	school that weren't in the list of things that

20	pertained to a kid or a classroom that could

21	overwhelm the teacher's influence.

22	The teachers in the school contribute

23	to the school, but, then, there could be in those

24	cases where the school is just really high

25	performing or really, really low performing, that
1	

1	a teacher could be affected by that, a teacher's

2	kids could be affected by that.	They didn't

3	always agree on what those things were.

4	Sometimes it's the principal, sometimes it's

5	the technology, the instruction materials, the

6	neighborhood or any of those kinds of things.

7	But you can tell that in data, where

8	the schools are different, that that matters,

9	and that's taken into account in the model.	So,

10	there is some mitigation of a low-growth school

11	for teachers who teach there in the model.

12	Now, whether it's enough to cover

13	your circumstances, that I don't know because it

14	sounds like you have other kinds of circumstances

15	about switching around lots of times during the

16	year and --

17	AUDIENCE MEMBER:	How did 13 students that

18	were attached to me for a very short amount of

19	time, how does that -- I don't even understand.

20	MS. HEBDA:	I don't know the answer to

21	that question because I don't know how the roster

22	verification is done.

23	AUDIENCE MEMBER:	Just because they were

24	attached to me for 20 minutes and they're still

25	on my --
1	

1	MS. HEBDA:	I would want to ask how the

2	roster verification was done in your school or

3	in your district to find out what -- if they're

4	-- to find out how that works.

5	AUDIENCE MEMBER:	It would be the first

6	or the second one, according to the State.	You

7	could be on the first two or three.

8	MS. HEBDA:	That's right.	You could be

9	in two or three unless the district has a policy

10	where you got to be in two and three.

11	Yes, sir.

12	AUDIENCE MEMBER:	This whole bit about

13	the way this is rolling out and how it's going

14	to impact different people differently, you know,

15	I understand the threshold and the trigger and

16	all that kind of good stuff.	I'm just telling

17	you a lot.	But, I am a teacher who has not

18	been subjected to student attached scores ever

19	yet.

20	MS. HEBDA:	Okay.

21	AUDIENCE MEMBER:	And probably will not be

22	really ever until I retire, but let's say next

23	year it starts to happen.

24	MS. HEBDA:	Okay.

25	AUDIENCE MEMBER:	So, I will not be
1	

1	subject to having to meet the threshold of having

2	to be highly effective for several years down

3	the road; whereas my -- the Algebra I teacher,

4	who has been teaching Algebra I for several

5	years, as soon as this rule goes into effect

6	must meet that threshold.

7	MS. HEBDA:	Three years after.

8	AUDIENCE MEMBER:	There is three years'

9	worth of data for that person currently

10	existing.

11	MS. HEBDA:	But she may not have been

12	evaluated on that data.

13	AUDIENCE MEMBER:	Well, you're

14	backtracking the data to create those scores.

15	MS. HEBDA:	But it depends on what you're

16	evaluated on.	It's how you were evaluated.

17	AUDIENCE MEMBER:	Well, since this --

18	now, I'm really confused.	Now, I'm really

19	confused.

20	Do we have three years worth of data

21	or not at this point?

22	MS. HEBDA:	There may -- we were just

23	talking about a situation where some districts

24	use their AP data now and some districts don't.

25	AUDIENCE MEMBER:	I'm talking about a
1	

1	third grade -- a fifth grade teacher.

2	MS. HEBDA:	Okay.

3	AUDIENCE MEMBER:	A fifth grade math FCAT.

4	That's what they've been doing, like, forever.

5	MS. HEBDA:	Right.

6	AUDIENCE MEMBER:	Ba, ba, ba, ba.

7	MS. HEBDA:	Right.

8	AUDIENCE MEMBER:	So, they've got the

9	data, like, for instance --

10	MS. HEBDA:	That's correct.

11	AUDIENCE MEMBER:	-- since 2000.	So, we

12	can generate expo facto VAM scores for them for

13	three years.

14	MS. HEBDA:	But we began in '11-'12, so

15	'11-'12 would be considered the first year.	So,

16	'11-'12 is the first year that were evaluated

17	using their real data.

18	AUDIENCE MEMBER:	So, '11-'12.

19	MS. HEBDA:	'12-'13, '13-'14 would be their

20	first year.

21	AUDIENCE MEMBER:	But for me it's not going

22	to happen until 2017.

23	MS. HEBDA:	That's right, because they

24	don't have data for you.

25	AUDIENCE MEMBER:	So, I could be getting
1	

1	these highly effective evaluations, and they

2	can't because they failed to meet the threshold.

3	They could be triggered into an unsatisfactory

4	based on student growth, but I can't.

5	MS. HEBDA:	That's correct.

6	AUDIENCE MEMBER:	That's true.	Okay,

7	now --

8	MS. HEBDA:	Correct.

9	AUDIENCE MEMBER:	-- to see if we can maybe

10	even make this more convoluted and worse, I'm a

11	calculus teacher teaching calculus four periods

12	a day, but I've been teaching one fifth grade

13	math class, for some stupid reason, for the past

14	three years.	So, I have fifth grade attached

15	VAM scores for three years of data, but it only

16	represents one-fifth of my teaching load.	Am I

17	subject to the trigger based on one-fifth of what

18	I did for three years?

19	MS. HEBDA:	Only if your district really

20	has -- if you were evaluated on those things that

21	you were teaching -- it would have to be at least

22	the majority of your day and they would have to

23	have comparable standards.

24	So, if you're being really evaluated

25	on schoolwide data for three years even though
1	

1	you taught that one fifth grade class --

2	AUDIENCE MEMBER:	Even that data does

3	exist and wasn't used against me --

4	MS. HEBDA:	Evaluated.	Evaluated.	It

5	matters on how you were evaluated.

6	AUDIENCE MEMBER:	I think I understand.

7	MS. HEBDA:	Yes, ma'am.

8	AUDIENCE MEMBER:	I've been waiting to go

9	back to this.	It doesn't relate to the current

10	stuff, but I wanted it to be counted for the

11	record.	You had said about the ESE data is

12	based on the kids' disabilities according to

13	the category in SLD -- (inaudible).

14	MS. HEBDA:	That's right.

15	AUDIENCE MEMBER:	I know in our district

16	the TBI and a couple other categories fall under

17	what we call other health impaired.	So, if that

18	is in our system, it doesn't specify TBI, or

19	whatever goes along with other health impaired,

20	is DOE asking for more specific if they get a

21	category like other health impaired?	Likewise,

22	with SLD, you can be dyslexic, auditory

23	processing.	I mean, a number of things just

24	under SLD while this other kid is just SLD

25	under dyslexia.	Does DOE ask for those specific
1	

1	things to go along with it if they get a

2	category for the district or is it just up to the

3	district to figure that out?

4	MS. HEBDA:	The district reports those

5	things to us.	We've looked at the primary

6	disability and the secondary disability that

7	are reported.	So, we haven't broken down, like,

8	a primary disability you're talking about into

9	different kinds of categories.	It's just the

10	primary disability reported to us.	So, SLD is

11	SLD, that's right.

12	AUDIENCE MEMBER:	But OHI could be under

13	asthma and it could be TBI.	So if we report it

14	as OHI, other health impaired, you're just

15	getting it under health impaired, so then it

16	really isn't taking into consideration those

17	other categories because it's not reporting it

18	as the specific thing that it is as opposed to

19	a category.

20	MS. HEBDA:	We do depend on the district

21	to report accurately.

22	AUDIENCE MEMBER:	Right.

23	MS. HEBDA:	Yes, sir.

24	Can I say one more thing?	You may

25	want to get a comment on the record, but I have
1	

1	way abused the court reporter who was hired for

2	two hours.	So, if anybody wants to say anything

3	else on the record, can you write it down and

4	give it to us and I'm going to let her go and end

5	the meeting, but I'll stay as long as anybody

6	wants to talk.

7	AUDIENCE MEMBER:	Well --

8	THE COURT REPORTER:	I need for you to

9	speak up because I had a hard time hearing you

10	the first time you spoke.

11	AUDIENCE MEMBER:	My question to her was

12	-- or, well, I'll ask -- is she ending the

13	meeting now and there won't be a court reporter

14	because I have waited to the end so I didn't

15	interrupt your flow dealing with the

16	technicalities associated with the VAM scores.

17	MS. HEBDA:	You go right ahead and say

18	anything that you want, how is that?

19	AUDIENCE MEMBER:	Okay.	And I am trying

20	to be sensitive and understand that we have worn

21	you.

22	I represent the human resources side

23	of this issue.	My concern is not necessarily

24	with the technicalities of the proposed rule,

25	which you've done an excellent job of explaining.
1	

1	My concern are the impacts that the proposed

2	rule will have on people, on their careers, on

3	their lives.

4	We have run our data in Seminole

5	County.	And one of the troubling aspects is the

6	disconnect between the A plus school

7	accountability grades and the teacher

8	accountability that's coming with this proposed

9	rule.	I know they're two separate things, but

10	there should be some sort of correspondence

11	between them because I've heard -- there's been

12	complaints, how can you have F schools that have

13	98 percent of your teachers be effective or

14	better.

15	If you flip it, how can we in our

16	school district have 15 percent of our teachers

17	be identified as needs improvement or

18	unsatisfactory, which is 204 teachers, when we

19	have a high-performing district, as we have.

20	We had 15 schools identified as less than

21	effective.	That has huge implications for those

22	administrators.	We had six schools who were

23	identified as unsatisfactory, three of those

24	schools were B rated, three of them were an

25	A rating.	It doesn't pass the common sense
1	

1	test to us.

2	In fact, over the past five

3	years, cumulatively those schools have earned

4	one C, eight Bs and 21 As.	There's not a D or

5	an F score involved there, but because of the

6	trigger, those administrators at each of those

7	six schools would have been at risk, if the rule

8	were in place, to being terminated -- the entire

9	administrative team terminated -- the following

10	year if they earned another unsatisfactory.

11	That, to us, seems fundamentally unfair.	And

12	it's a human side to this that we want to bring

13	to everyone's attention.	I'm not complaining.

14	I'm saying it's really important for us to

15	understand the practical impact of these

16	distinctions; that you can get caught up in the

17	technicalities sometimes and forget about it.

18	So, I appreciate your patience.

19	And, the court reporter, I appreciate your

20	indulgence as well.

21	MS. HEBDA:	Anything else for the record

22	before the court reporter gets to go home?

23	Thank you very much.

24	We'll end the rule workshop here.

25	(The workshop ended at 7:05 p.m.)
1	

1

2	STATE OF FLORIDA: COUNTY OF ORANGE:
3

REPORTER'S CERTIFICATE

4	I, SANDRA DIANE EVANS, RPR, certify that I

5	was authorized to and did report stenographically the

6	foregoing proceedings, and that the transcript is a

7	true record.

8	I FURTHER CERTIFY that I am not of counsel

9	for, related to, or employed by any party hereto or

10	attorney involved herein, nor am I financially

11	interested in the outcome of the action.

12	Dated this 11th day of March, 2013.

13

14

15
SANDRA DIANE EVANS, RPR
16

17

18

19

20

21

22

23

24

25

'	
2.0 [1] - 94:16
20 [10] - 99:1, 100:14,	
6	
absences [2] - 35:13,
108:8	
Act [2] - 12:12, 69:5
actual [7] - 4:13, 13:8,	
		
6 [1] - 54:11			
	100:15, 102:1,		absolute [1] - 114:21	70:20, 103:18,	
'11 [1] - 136:17	145:6, 146:2, 170:6,		absolutely [8] - 43:21,	105:2, 141:10, 177:4	
'11-'12 [22] - 35:22,	170:9, 170:21,	60 [4] - 19:13, 19:19,	88:15, 115:7, 116:3,	add [10] - 7:16, 64:5,	
64:9, 66:2, 66:4,	181:24	32:17, 132:24	139:5, 139:24,	69:11, 94:8, 109:18,	
72:6, 76:5, 89:4,	2000 [1] - 184:11	60/40 [1] - 68:12	153:8, 159:11	114:19, 131:11,	
103:11, 123:15,	2009-'10 [1] - 65:2	68 [5] - 134:16,	Absolutely [9] - 26:15,	137:1, 142:2	
127:16, 127:19,	2011 [1] - 70:25	134:20, 152:5,	32:24, 32:25, 36:1,	added [34] - 80:18,	
127:23, 131:24,	2013 [1] - 161:10	153:17, 155:21	57:15, 90:18, 138:9,	82:22, 82:24, 86:18,	
135:15, 136:17,	2013-'14 [1] - 137:16	6A-5.030 [3] - 3:8,	153:5, 156:3	90:11, 94:20, 94:21,	
136:21, 148:12,
172:24, 184:14,
184:15, 184:16,
184:18
'11-'12-'13 [1] - 132:2
'12 [1] - 136:17
'12-'13 [4] - 127:17,
136:17, 150:15,
184:19
'13-'14 [3] - 150:10,
150:12, 184:19	2014-'15 [2] - 78:8,
78:13
2015 [1] - 60:14
2017 [1] - 184:22
204 [1] - 189:18
21 [2] - 120:7, 190:4
22 [4] - 119:21,
119:23, 120:6, 120:7
23 [1] - 175:24
27 [1] - 177:8	11:8, 13:8	abstract [2] - 154:16	96:8, 96:10, 100:16,	
		6A-5.0411 [1] - 3:8	abused [1] - 188:1	103:4, 109:19,	
		
7	accept [2] - 132:23,	109:22, 110:10,	
			153:18	114:5, 122:7, 122:9,	

7 [3] - 168:22, 168:23,	accommodations [1]	123:10, 123:16,	
			- 105:20	125:9, 125:11,	
			According [1] -	126:9, 126:18,	
		168:24			
			147:15	127:3, 127:20,	
		75 [4] - 152:12,			
			according [6] - 33:2,	137:6, 140:10,	
		171:23, 172:1			
			80:5, 166:15,	141:10, 142:1,	
		7:00 [1] - 168:21			
			166:18, 182:6,	143:11, 169:18,	
		7:05 [1] - 190:25			
	3		186:12	170:20, 171:17,	
0					
			accordingly [1] - 84:5	172:9	
		8			
	
3 [2] - 14:6, 155:2				

0411 [2] - 70:16,			account [17] - 86:20,	adding [1] - 145:9	

80 [3] - 152:9, 152:14,	88:16, 90:14, 98:21,	addition [2] - 21:7,	
	3.7 [1] - 144:2		99:5, 99:8, 110:2,	85:22	
149:10					
	30 [4] - 32:17, 120:24,	152:15	113:15, 116:17,	additional [4] - 32:11,	
1	120:25, 121:2	87 [1] - 76:14	116:20, 116:24,	63:19, 63:21, 135:11	
					
	300 [6] - 35:3, 35:4,		117:23, 117:24,	address [4] - 6:1,	

1 [5] - 40:10, 60:14,	89:9, 89:11, 89:12,	9	119:14, 143:21,	16:24, 38:8, 175:9	
	89:15		157:14, 181:9	adjust [7] - 86:1, 86:3,	
		
9 [1] - 145:17			
145:9, 155:2, 162:24	
4		accountability [6] -	130:12, 131:10,	
1012.34 [1] - 11:23			87:10, 92:7, 121:15,	131:19, 132:3, 174:2	
11 [7] - 16:1, 16:5,	

4 [2] - 14:7, 39:12	90 [7] - 31:8, 31:21,	178:10, 189:7, 189:8	adjusted [2] - 95:10,	
16:6, 16:7, 31:5		32:17, 33:9, 33:10,	accountable [1] -	102:23	
12 [2] - 15:25, 18:21		33:18, 97:17	78:18	adjustment [8] - 79:1,	
	40 [7] - 19:18, 39:21,				
120 [1] - 121:6		95 [18] - 129:11, 133:3,	accounted [3] - 86:24,	79:3, 85:24, 102:16,	
	59:17, 64:17, 65:13,				
125 [1] - 154:18		133:6, 133:17,	154:19, 157:10	113:5, 157:9,	
	65:18, 66:5				
13 [3] - 41:21, 177:17,		134:23, 140:16,	accounts [2] - 90:3,	160:17, 168:14	
	44 [1] - 175:23				
181:17		140:25, 141:14,	179:3	administer [4] - 26:8,	
	47 [1] - 125:6	143:5, 143:8, 152:4,			
15 [3] - 25:14, 189:16,			accuracy [4] - 22:1,	26:20, 28:4, 29:1	
189:20		153:7, 153:9,	22:6, 25:24, 30:5	administrative [4] -	
	5				
15th [1] - 157:12		155:20, 156:12,	Accuracy [1] - 26:9	4:8, 15:5, 15:9,	
16 [2] - 129:2, 129:23	
5 [3] - 120:24, 145:3,	156:13, 157:6	accurate [4] - 70:8,	190:9	
168,000 [1] - 128:11		98 [1] - 189:13	111:16, 121:18,	Administrative [1] -	
17 [2] - 158:4, 176:1		99.7 [1] - 157:6	165:23	13:14	
	155:2				
18 [1] - 154:20	50 [15] - 19:19, 59:12,		accurately [2] - 6:20,	Administrator [1] -	
		A			
			187:21	11:9	
180 [1] - 97:16	65:17, 84:7, 84:8,				
19 [1] - 170:8	124:13, 124:15,		achievement [19] -	administrator [4] -	
		
ability [3] - 98:21,			
	124:20, 124:22,		59:25, 60:8, 72:21,	12:17, 12:18, 22:8,	
19th [1] - 157:13					
	125:12, 137:8,		74:11, 77:11, 77:12,	42:16	
1st [1] - 12:8		137:16, 177:23			
	138:12, 152:8		81:24, 85:1, 85:25,	administrators [5] -	
		able [16] - 21:18, 24:6,			
2	50/50 [1] - 68:12		89:23, 89:24, 96:15,	48:14, 62:17, 69:21,	
		25:19, 33:22, 33:25,			
	500 [1] - 158:5		96:22, 97:4, 97:7,	189:22, 190:6	

2 [5] - 13:22, 21:25,		34:3, 35:15, 36:10,			
	53 [1] - 125:6		167:7, 167:11,	adopt [1] - 28:24	
		38:24, 44:24, 78:3,			
	54 [1] - 154:21		167:17, 171:11	adopted [7] - 80:5,	
		97:3, 112:24, 131:6,			
145:5, 155:2, 162:24			achieving [1] - 83:25	84:23, 85:15,	
		132:3, 161:12			
2,000 [3] - 137:7,			acknowledge [1] -	149:18, 150:14,	
		absence [1] - 59:17			
158:3			155:12	163:24, 164:20	

advanced [2] - 71:17,	
alone [1] - 63:3	
98:25	
59:15, 59:21, 59:24,	
40:2, 40:4, 40:8,	
73:24	Alternate [1] - 92:3	appreciate [5] - 3:1,	60:15, 60:22, 60:23,	40:10, 40:24, 41:19,	
advantages [1] - 89:5	alternate [1] - 81:22	111:2, 146:1,	67:18, 78:25, 79:2,	42:8, 43:10, 43:16,	
advisory [1] - 112:14	amend [2] - 31:21,	190:18, 190:19	84:22, 85:7, 160:8,	44:1, 44:15, 44:20,	
affect [3] - 10:14,	32:10	appropriate [7] -	161:13, 163:13,	45:7, 45:16, 47:5,	
87:24, 94:20	amending [1] - 31:6	48:16, 62:18, 62:19,	163:23, 164:9,	47:9, 47:12, 47:17,	
affected [2] - 181:1,	amendments [1] -	62:20, 85:4, 85:6,	165:20, 168:11	48:8, 48:11, 49:2,	
181:2	63:20	167:18	assign [2] - 61:9,	50:13, 51:5, 53:21,	
affects [2] - 9:1, 12:22	American [1] - 114:7	approval [8] - 16:15,	164:25	54:9, 54:11, 54:15,	
afford [1] - 39:4	amount [6] - 169:14,	16:21, 17:4, 19:1,	assigned [14] - 34:18,	55:6, 55:13, 55:20,	
afraid [1] - 65:4	175:17, 175:24,	51:12, 51:13, 51:18,	60:21, 86:16, 118:7,	56:20, 57:13, 64:7,	
age [2] - 93:19, 93:24	177:9, 177:12,	77:25	119:19, 122:15,	64:13, 64:20, 64:22,	
agency [2] - 4:7, 12:20	181:18	approve [12] - 12:13,	122:20, 122:24,	67:3, 67:21, 68:22,	
agenda [2] - 3:25,	anatomy [1] - 78:11	12:16, 13:2, 14:11,	123:19, 125:16,	69:16, 70:3, 70:4,	
10:18	anchor [8] - 127:23,	14:21, 28:19, 44:10,	125:18, 125:19,	71:15, 71:19, 72:14,	
aggregate [2] - 82:24,	128:1, 136:2,	51:10, 52:3, 52:24,	162:17	72:23, 73:1, 73:6,	
169:19
ago [4] - 5:5, 31:15,
33:8, 66:15
agree [5] - 26:13,
35:18, 112:25,
162:9, 181:3
agreeable [1] - 64:24
agreed [1] - 66:9
agreement [7] - 49:10,
50:10, 50:21, 51:17,
53:13, 53:23, 114:10
agreements [2] -
49:12, 50:17
ahead [8] - 48:23,
52:3, 57:9, 137:13,
142:19, 142:20,
142:21, 188:17
air [2] - 19:21, 132:22
Algebra [26] - 71:4,
71:8, 71:16, 72:18,
74:12, 74:15, 74:17,
74:20, 75:9, 75:20,
76:2, 77:1, 77:2,
77:20, 80:22, 81:6,
81:15, 81:25, 84:24,
116:7, 118:2, 159:2,
159:4, 159:6, 183:3,
183:4
algebra [23] - 71:6,
71:20, 71:23, 72:16,
73:7, 73:9, 73:15,
73:17, 73:19, 73:20,
73:23, 74:4, 74:16,
76:9, 76:10, 79:22,
79:23, 79:24, 80:6,
80:12, 80:24, 81:2,
81:8
allow [1] - 105:23
allowed [2] - 95:23,
131:4
allowing [1] - 150:18
almost [5] - 28:12,
75:12, 90:3, 158:12,
168:21	136:14, 136:18,
136:23, 137:1, 137:2
annual [13] - 12:7,
20:19, 21:4, 21:6,
21:18, 21:20, 22:15,
22:18, 23:7, 23:11,
63:11, 64:8, 128:21
annually [1] - 12:5
anomalies [2] - 23:17,
118:3
answer [13] - 6:8,
6:13, 30:9, 42:20,
50:15, 53:21, 97:18,
113:13, 124:22,
124:23, 133:25,
134:1, 181:20
answering [1] - 50:7
answers [3] - 5:8,
6:11, 47:3
anticipate [1] - 11:2
anyhow [1] - 88:2
anyway [2] - 68:20,
96:3
AP [15] - 166:15,
166:16, 166:25,
167:4, 167:19,
169:9, 172:16,
172:18, 172:25,
173:2, 173:5, 173:8,
173:16, 183:24
appeal [9] - 175:15,
175:16, 175:23,
175:25, 176:2,
176:4, 176:5,
176:17, 177:3
appealing [3] -
176:20, 176:21
applied [2] - 54:17,
137:6
applies [2] - 61:17
apply [6] - 9:23, 42:9,
45:17, 84:17, 139:6,
148:15
appointments [1] -	52:25, 74:8
approved [12] - 14:24,
15:1, 16:17, 16:18,
46:10, 52:16, 52:17,
61:2, 62:24, 70:21,
73:12, 81:10
approves [1] - 12:4
April [5] - 8:21, 8:24,
9:5, 10:24, 11:3
area [2] - 161:7,
177:11
areas [2] - 58:7, 162:1
argument [1] - 124:12
arrived [1] - 98:6
Arrived [1] - 135:6
art [4] - 159:21,
159:24, 160:3, 160:4
article [1] - 130:17
articles [1] - 130:18
arts [4] - 34:25, 89:7,
91:1, 125:10
aspects [1] - 189:5
assess [2] - 67:18,
164:8
assessed [1] - 178:17
assessing [1] - 163:25
assessment [28] -
23:8, 23:12, 28:8,
28:19, 28:23, 54:16,
54:23, 55:16, 55:17,
59:13, 60:19, 60:20,
67:19, 79:4, 136:12,
136:19, 160:16,
160:19, 161:22,
165:21, 166:11,
166:13, 169:8,
169:10, 169:12,
169:25, 170:20,
172:5
Assessment [1] - 92:4
assessments [28] -
27:6, 27:9, 27:11,
27:21, 27:22, 28:4,
28:5, 56:9, 59:14,	assignment [1] -
66:10
assistance [4] - 16:4,
28:2, 28:21, 63:18
assistant [2] - 31:14,
47:22
associated [6] -
55:17, 55:20, 72:8,
153:24, 170:4,
188:16
asthma [1] - 187:13
attached [6] - 41:22,
120:18, 181:18,
181:24, 182:18,
185:14
attend [1] - 5:9
Attendance [1] - 92:20
attendance [14] -
88:11, 92:21, 96:5,
97:10, 97:11, 97:12,
97:18, 97:20, 97:25,
98:9, 99:4, 101:1,
110:2
attendees [1] - 136:3
attention [1] - 190:13
attributed [1] - 86:25
AUDIENCE [310] -
10:20, 14:23, 16:6,
17:3, 17:6, 17:9,
17:13, 18:8, 18:15,
19:11, 19:18, 20:4,
20:8, 21:11, 21:23,
21:25, 22:4, 22:16,
23:18, 24:1, 24:21,
25:1, 25:17, 25:23,
26:1, 26:12, 26:15,
27:5, 29:6, 29:10,
29:20, 30:2, 31:5,
31:12, 32:22, 33:2,
33:10, 33:13, 33:16,
33:18, 33:22, 35:2,
35:19, 35:24, 36:15,
36:21, 37:7, 37:22,
38:10, 39:2, 39:9,	74:14, 74:20, 75:5,
75:23, 75:25, 76:4,
77:6, 77:16, 78:2,
78:7, 79:16, 79:19,
80:9, 80:24, 81:5,
81:14, 81:20, 83:16,
83:20, 84:6, 84:18,
87:15, 87:23, 88:5,
88:6, 88:10, 88:21,
89:12, 89:16, 90:16,
90:19, 91:18, 92:24,
93:2, 95:13, 95:25,
96:12, 96:15, 96:19,
96:24, 98:5, 98:17,
98:20, 99:7, 99:25,
100:6, 100:11,
101:7, 101:22,
103:1, 103:15,
103:18, 104:13,
104:19, 104:22,
105:3, 105:10,
107:6, 107:17,
107:19, 108:2,
108:7, 108:10,
110:12, 110:13,
110:18, 110:24,
111:5, 111:12,
114:18, 115:12,
115:17, 116:5,
116:21, 116:25,
117:6, 117:11,
117:17, 117:21,
118:13, 119:12,
119:17, 120:8,
120:11, 120:15,
120:21, 121:19,
128:25, 129:9,
129:15, 130:20,
131:17, 131:25,
132:17, 133:6,
133:12, 133:21,
135:2, 135:5,
135:15, 135:18,
137:5, 137:14,
137:19, 138:10,	

139:1, 139:23,	
automatic [2] -	
128:6, 128:20,	
166:6, 175:8	
109:19, 121:23,	
141:16, 143:21,	148:13, 148:18	131:19, 134:14,	biggest [3] - 75:16,	127:7	
143:25, 144:4,	available [5] - 31:17,	137:21, 147:3,	96:3, 142:11	calculus [6] - 78:11,	
144:13, 144:21,	31:22, 31:25, 54:23,	147:22, 148:5,	Bill [1] - 8:25	173:6, 173:12,	
144:23, 145:2,	64:18	152:23, 154:5,	bit [7] - 8:17, 69:17,	173:17, 185:11	
145:5, 145:8,	average [29] - 83:1,	160:14, 163:17,	117:4, 151:18,	calibration [2] - 22:22,	
145:12, 145:15,	102:3, 122:15,	165:8, 165:10,	157:7, 182:12	23:13	
145:17, 145:22,	122:19, 123:1,	167:14, 174:20,	black [1] - 140:5	candid [1] - 114:22	
145:25, 146:23,	128:3, 129:11,	185:4, 185:17,	block [3] - 109:24,	cannot [4] - 105:18,	
148:1, 148:6,	140:12, 140:21,	186:12	109:25, 110:23	109:16, 114:24,	
148:14, 148:19,	141:4, 141:12,	basic [1] - 71:21	board [3] - 19:7, 19:8,	118:6	
148:22, 149:17,	142:9, 143:16,	basis [9] - 46:17,	121:4	capability [2] - 113:8,	
150:4, 150:10,	145:7, 146:10,	46:19, 64:25, 78:15,	Board [9] - 7:12, 9:3,	161:12	
150:17, 151:1,	156:14, 170:6,	107:13, 113:7,	10:8, 10:12, 17:22,	capacity [1] - 113:9	
151:7, 151:13,	170:7, 170:9,	167:8, 167:11,	18:6, 18:14, 63:22,	capture [3] - 6:11,	
151:17, 151:23,	170:15, 171:4,	167:13	63:23	36:3, 165:21	
152:3, 152:7,	171:5, 171:21,	bathroom [1] - 166:23	book [3] - 50:23,	capturing [1] - 43:23	
152:14, 152:17,	171:22, 171:24,	beat [1] - 128:22	166:1, 166:2	card [3] - 4:25, 6:18,	
153:3, 153:6,	173:2, 173:3	became [2] - 31:17,	bottom [10] - 14:7,	56:21	
153:13, 154:2,	averages [1] - 165:10	138:16	18:20, 124:17,	cards [1] - 3:12	
154:14, 154:23,	avoid [2] - 145:12,	become [7] - 127:24,	127:12, 140:19,	care [4] - 119:1, 119:2,	
155:1, 156:22,	145:15	135:21, 139:14,	151:9, 151:18,	119:4, 177:13	
157:16, 157:22,
158:15, 159:8,		146:14, 149:18,	156:8, 175:6	career [4] - 76:12,	
	B				
		160:22, 162:20	brain [5] - 105:23,	112:4, 153:23, 157:9	
159:13, 159:17,					
		began [1] - 184:14	106:11, 106:15,	careers [3] - 111:17,	
	
Ba [1] - 184:6				
159:21, 161:1,					
		beginning [6] - 98:2,	106:19, 106:23	114:23, 189:2	
161:4, 162:11,					
		103:22, 103:25,	brand [1] - 108:23	Carolina [1] - 176:8	
162:19, 162:22,	ba [3] - 184:6				
		111:20, 117:18,	brand-new [1] -	carry [1] - 37:25	
162:23, 166:9,	background [1] -				
		179:22	108:23	case [4] - 18:17,	
166:14, 166:18,	133:14				
		begins [1] - 22:13	break [1] - 8:16	64:25, 141:21	
166:21, 167:1,	backtracking [1] -				
		behalf [1] - 79:20	bring [3] - 65:5, 70:10,	case-by-case [1] -	
167:4, 167:10,	183:14				
		bell [5] - 129:10,	190:12	64:25	
167:22, 167:25,	Bad [1] - 151:16				
		130:8, 130:10, 156:4	brings [1] - 87:1	cases [2] - 93:15,	
168:2, 172:13,	bad [1] - 134:8				
		belong [1] - 157:19	broken [2] - 178:2,	180:24	
172:17, 174:1,	band [3] - 141:14,				
		below [15] - 123:4,	187:7	categories [5] - 39:15,	
174:24, 175:13,	153:22, 155:11				
		140:24, 141:3,	brought [3] - 25:3,	157:19, 186:16,	
175:22, 176:19,	bank [2] - 28:10,				
		141:4, 142:1, 142:3,	63:20, 179:19	187:9, 187:17	
177:1, 177:6,	161:15				
		145:1, 145:7,	Broward [2] - 8:15,	categorize [1] - 153:8	
178:14, 178:16,	banks [1] - 27:16				
		152:11, 152:12,	136:4	category [12] - 51:16,	
181:17, 181:23,	bar [11] - 95:10, 95:20,				
		152:15, 154:6,	Bs [1] - 190:4	52:1, 105:11, 119:5,	
182:5, 182:12,	102:23, 113:11,				
		154:7, 156:14	build [3] - 28:3, 28:22,	129:24, 141:18,	
182:21, 182:25,	143:2, 143:6,				
		benchmark [1] - 28:15	121:22	156:7, 156:8,	
183:8, 183:13,	143:10, 144:17,				
		best [4] - 25:10, 65:23,	burn [1] - 118:10	186:13, 186:21,	
183:17, 183:25,	146:9, 152:18,				
		77:17, 161:15	but.. [1] - 47:4	187:2, 187:19	
184:3, 184:6, 184:8,	156:14				
		Beth [1] - 115:6	buy	- 39:20, 114:5,	caught	- 190:16	
184:11, 184:18,	bargaining [10] -				
		better [14] - 39:21,	132:25	certain [7] - 90:7,	
184:21, 184:25,	49:10, 49:11, 50:10,				
		46:8, 128:1, 128:5,
128:7, 136:20,		106:2, 107:1, 148:1,	
185:6, 185:9, 186:2,	50:17, 50:21, 51:17,				
			C	149:21, 175:17,	
					
186:6, 186:8,	53:8, 53:12, 53:22,				
		136:24, 141:13,		175:24	
186:15, 187:12,	177:2		
calculate [7] - 66:13,		
		163:6, 179:23,		certainly [12] - 25:10,	
187:22, 188:7,	base [1] - 78:14				
		189:14		39:13, 41:11, 46:18,	
188:11, 188:19	based [42] - 11:22,				
		Between [1] - 117:11	106:5, 110:10,	75:13, 90:6, 90:10,	
auditory [1] - 186:22	13:9, 34:10, 52:15,				
		between [11] - 12:22,	110:21, 110:22,	90:13, 113:3,	
authority [5] - 4:7,	59:12, 61:1, 68:2,		146:17, 147:13		
		21:19, 59:20, 73:14,		118:20, 153:8,	
11:20, 28:18, 49:14,	76:5, 76:19, 80:19,		calculated [5] - 72:15,		
		100:19, 116:12,		157:17	
70:17	81:8, 81:10, 86:4,		79:25, 81:8, 105:25,		
		118:18, 121:12,		certification [2] - 60:5,	
autism [2] - 91:20,	86:11, 94:10, 94:12,		113:7		
		125:6, 189:6, 189:11		162:2	
91:23	95:7, 95:10, 102:2,		calculating [1] - 82:20		
		beyond [2] - 63:22,		certifications [1] -	
autistic [2] - 107:21,	102:24, 106:21,		calculation [6] -		
		121:6		60:5	
107:23	107:2, 122:17,				
		big [4] - 52:7, 76:9,	54:16, 97:24, 99:19,	cetera [1] - 149:24	

[3]

[1]

challenge [3] - 76:9,	
39:12, 63:10,	
104:10, 104:24,	
160:22, 168:25,	
confident [17] - 125:3,	
118:14, 120:23	120:17, 126:6,	106:10, 119:25	169:7, 171:13,	125:22, 126:5,	
challenged [1] - 15:4	134:21, 135:4,	cognitive [1] - 106:11	172:7, 172:10,	126:12, 134:20,	
challenging [3] -	135:8, 141:1, 179:2,	cohorts [1] - 74:23	174:15, 185:23	134:23, 140:17,	
87:25, 120:16	181:13, 181:14	collect [2] - 92:21,	compare [1] - 106:20	140:25, 141:24,	
chance [3] - 99:16,	City [1] - 8:14	179:24	compared [3] - 97:1,	143:8, 143:12,	
138:12, 142:6	clarification [4] -	collective [10] - 49:9,	106:18, 128:14	144:7, 153:9,	
change [13] - 3:18,	47:13, 48:11, 48:18,	49:11, 50:10, 50:16,	compares [1] - 38:24	155:17, 155:20,	
15:19, 19:7, 19:18,	151:8	50:21, 51:17, 53:7,	competent [2] - 26:7,	156:12, 156:13	
20:5, 20:7, 48:7,	clarify [3] - 43:10,	53:12, 53:22, 177:2	26:20	confused [4] - 31:12,	
116:6, 126:2,	53:5, 79:19	college [1] - 167:21	complaining [1] -	31:23, 183:18,	
149:12, 150:12,	clarifying [1] - 4:16	colloquially [1] - 55:5	190:13	183:19	
150:14, 152:25	clarity [1] - 97:10	combine [5] - 58:14,	complaints [1] -	consequences [2] -	
changed [2] - 33:15,	class [35] - 34:17,	59:24, 68:10, 80:17,	189:12	51:6, 153:24	
176:2	41:24, 60:2, 63:6,	122:23	compliance [4] -	consider [8] - 29:12,	
changes [6] - 12:11,	73:13, 78:11, 78:12,	combined [2] -	17:23, 18:5, 21:13,	29:18, 64:1, 66:4,	
19:25, 20:18, 47:14,	79:6, 81:7, 81:9,	136:18, 148:9	53:19	83:22, 126:25,	
85:16, 149:14	88:14, 88:17, 91:16,	comfortable [1] -	complicated [3] -	156:18, 157:3	
characteristic [2] -	93:3, 93:25, 99:3,	165:1	91:11, 113:4, 115:18	considerably [1] -	
95:11, 102:4	100:14, 108:22,	coming [6] - 3:9, 3:20,	complies [1] - 50:20	135:22	
characteristics [11] -	116:10, 116:12,	35:10, 69:19,	comply [4] - 17:15,	consideration [13] -	
86:2, 89:20, 89:21,	116:15, 117:15,	157:11, 189:8	50:24, 53:3	3:25, 9:4, 19:24,	
94:11, 95:15, 102:4,	118:2, 119:15,	comment [9] - 3:11,	complying [1] - 51:3	20:13, 30:25, 43:1,	
104:25, 105:4,	119:20, 119:21,	4:17, 4:23, 9:7,	component [4] -	127:18, 129:22,	
111:6, 125:2	160:7, 170:23,	43:19, 88:19,	31:19, 55:23, 180:7,	132:11, 137:3,	
chart [1] - 113:11	177:9, 177:17,	108:14, 161:4,	180:14	156:23, 168:7,	
charter [10] - 42:10,	177:18, 185:13,	187:25	comprehensive [1] -	187:16	
42:11, 42:12, 44:8,	186:1	commentary [1] -	105:24	considerations [1] -	
44:10, 44:11, 44:13,	classes [17] - 71:21,	112:20	concept [2] - 136:22,	166:7	
151:1, 151:3	71:25, 72:3, 73:25,	comments [16] - 4:24,	147:14	considered [6] -	
charters [1] - 44:4	75:20, 79:21, 80:6,	5:15, 5:17, 5:20,	concern [7] - 111:14,	19:25, 62:3, 101:17,	
check [4] - 23:15,	116:7, 116:14,	5:24, 5:25, 6:3, 6:25,	137:15, 158:17,	101:20, 120:9,	
25:6, 25:7, 61:21	116:16, 138:15,	8:4, 10:21, 21:22,	177:13, 179:11,	184:15	
checking [4] - 22:22,	147:23, 163:12,	84:5, 161:2, 175:8,	188:23, 189:1	considering [1] -	
22:23, 59:9	164:25, 171:23	175:10	concerned [8] - 37:2,	180:15	
checklist [2] - 15:11,	classified [1] - 158:8	Commissioner [8] -	41:21, 48:21, 94:19,	consistency [10] -	
15:12	classroom [11] -	17:21, 18:13, 21:5,	119:7, 157:22,	13:13, 27:10, 28:5,	
chemistry [1] - 172:18	54:12, 60:15, 89:20,	70:22, 77:25, 85:14,	179:8, 179:14	29:2, 82:11, 82:19,	
cherry [1] - 65:22	105:20, 119:14,	85:15, 165:16	concerns [1] - 69:18	83:11, 84:1, 84:14,	
child [9] - 34:2, 87:19,	138:19, 157:11,	Commissioner's [2] -	conclusion [1] -	139:11	
88:14, 96:20, 97:15,	157:12, 164:6,	12:7, 18:2	153:19	Consistency [1] - 27:5	
98:23, 98:24, 100:3,	171:19, 180:20	Committee [2] - 71:3,	conditional [2] -	consistent [4] - 30:10,	
108:18	classrooms [2] -	85:13	51:12, 51:18	43:3, 180:1, 180:2	
child's [1] - 87:18	67:15, 163:12	committee [12] - 74:8,	conditionally [3] -	consistently [1] -	
children [5] - 88:23,	clear [3] - 8:5, 83:10,	85:19, 87:3, 91:5,	16:18, 52:2, 52:17	179:10	
109:6, 162:25,	84:4	91:12, 96:6, 96:7,	conditions [5] - 84:16,	consortium [1] -	
177:8, 178:5	clearly [1] - 6:19	114:8, 179:4,	139:15, 147:17,	160:18	
choose [7] - 28:9,	click [1] - 175:9	179:19, 180:17	148:15, 148:17	construct [5] - 126:20,	
32:14, 32:15, 74:12,	clicker [1] - 3:17	common [2] - 27:19,	conducive [1] - 64:22	134:16, 140:15,	
95:23, 164:11,	close [4] - 11:1, 31:8,	189:25	conduct [1] - 52:8	141:21, 142:3	
164:13	33:14, 33:17	commonly [2] - 152:3,	Conduct [1] - 63:25	constructed [4] -	
chooses [2] - 18:24,	closed [1] - 34:15	152:4	confidence [17] -	13:23, 32:8, 122:5,	
44:18	clue [3] - 22:4, 155:3,	communication [1] -	125:23, 126:4,	163:7	
choosing [2] - 59:23,	155:4	101:3	126:20, 133:3,	consulting [1] - 70:10	
59:25	Code [1] - 63:24	communities [1] -	133:7, 134:16,	contained [1] - 12:6	
chose [1] - 95:22	coefficient [3] -	39:24	140:16, 141:14,	contemplated [2] -	
chronicles [1] - 179:7	102:22, 106:14,	company [2] - 70:10,	142:4, 143:5, 152:2,	33:1, 45:13	
circumstance [1] -	106:15	124:9	152:5, 152:8, 152:9,	content [18] - 4:10,	
90:11	coefficients [7] - 78:5,	comparable [10] -	153:7, 153:17,	11:16, 12:1, 13:4,	
circumstances [11] -	102:15, 103:2,	123:7, 149:4,	155:11	13:10, 13:11, 14:17,	

14:23, 15:1, 15:10,	
109:17, 119:18,	
covariates [2] - 94:5,	
36:12, 38:19, 38:23,	
121:1, 121:6	
15:15, 15:16, 49:8,	120:11, 147:18,	179:22	54:16, 54:23, 55:16,	deadline [2] - 10:20,	
49:21, 50:11, 57:17,	147:20, 147:24,	cover [7] - 3:23, 4:10,	55:17, 55:22, 56:1,	161:10	
163:25	168:23, 178:4,	7:17, 10:3, 19:9,	56:8, 58:1, 59:13,	deal [3] - 24:19, 88:1,	
contents [1] - 9:2	178:17	81:13, 181:12	59:18, 59:21, 61:10,	105:12	
continue [2] - 76:12,	counted [2] - 105:15,	covering [1] - 58:16	61:13, 62:8, 62:10,	dealing [3] - 23:10,	
114:3	186:10	covers [1] - 172:4	64:18, 65:2, 65:24,	23:16, 188:15	
contract [5] - 46:23,	counties [1] - 39:3	cram [1] - 79:3	66:3, 66:12, 66:15,	deals [4] - 12:2, 20:11,	
51:1, 176:24, 177:1,	county [7] - 19:12,	crap [1] - 97:5	66:17, 67:3, 67:5,	22:17, 24:15	
177:3	37:8, 39:3, 39:6,	create [4] - 50:3,	67:13, 67:20, 67:22,	dealt [2] - 24:15, 26:22	
contracted [1] - 114:6	100:12, 108:10,	143:2, 183:14	67:23, 67:25, 68:3,	deans [1] - 138:20	
contribute [3] - 47:15,	178:1	created [1] - 163:24	68:6, 68:15, 68:20,	debate [2] - 5:17,	
115:5, 180:22	County [13] - 8:15,	creating [2] - 132:19,	70:25, 74:12, 76:21,	116:1	
contributed [1] -	28:7, 39:21, 93:22,	163:22	78:13, 80:3, 80:4,	decade [1] - 12:15	
130:17	129:1, 130:12,	credit [3] - 143:10,	81:12, 89:4, 89:10,	December [1] - 12:8	
contributing [2] -	130:16, 136:4,	144:12, 167:21	90:12, 92:1, 96:22,	decide [6] - 39:20,	
48:5, 87:12	142:25, 172:24,	crime [1] - 177:10	97:23, 98:19, 99:20,	42:25, 48:2, 48:3,	
control [3] - 179:16,
179:18, 180:6
conversation [1] -
77:3
convert [1] - 170:10
converted [2] - 82:25,
169:19
convoluted [1] -
185:10
Cook [1] - 162:6
copies [1] - 54:7
copy [1] - 3:5
core [1] - 27:19
Correct [4] - 20:9,
78:6, 152:16, 185:8
correct [23] - 14:25,
17:10, 17:12, 29:13,
36:22, 51:15, 81:11,
83:18, 83:19, 99:5,
99:6, 111:25, 115:7,
121:18, 148:16,
148:21, 149:25,
151:4, 151:22,
166:17, 166:20,
184:10, 185:5
corrected [1] - 34:20
correctly [6] - 88:24,
111:14, 112:2,
120:2, 131:17,
178:22
correspondence [1] -
189:10
Counseling [1] -
161:25
counselor [3] - 108:3,
148:19, 161:24
counselors [1] -
138:19
count [18] - 40:19,
77:22, 108:13,
108:17, 108:20,
109:4, 109:6,
109:13, 109:14,	189:5
couple [14] - 3:23, 4:1,
6:15, 7:6, 27:13,
69:20, 117:21,
122:3, 126:23,
161:20, 169:1,
178:12, 178:20,
186:16
course [41] - 27:8,
27:20, 28:14, 29:4,
32:21, 46:14, 71:23,
72:8, 72:9, 72:11,
72:19, 73:17, 76:11,
77:12, 78:12, 90:6,
91:1, 101:1, 110:7,
110:9, 110:11,
117:13, 137:17,
161:13, 164:1,
164:18, 164:19,
164:20, 164:24,
167:19, 167:24,
168:18, 169:8,
169:11, 174:11,
174:21, 174:22
courses [19] - 60:9,
79:11, 79:12, 80:14,
80:23, 81:2, 81:3,
90:25, 91:3, 109:24,
110:1, 110:23,
162:3, 167:18,
170:3, 171:6, 171:9,
172:11, 173:24
COURT [4] - 45:25,
149:11, 149:16,
188:8
court [8] - 5:1, 6:9,
6:19, 43:11, 188:1,
188:13, 190:19,
190:22
covariants [1] - 86:2
covariate [6] - 79:1,
79:3, 85:24, 113:4,
168:14, 180:15	criteria [5] - 48:17,
58:7, 58:16, 75:11,
160:14
criterion [5] - 126:13,
127:24, 128:20,
136:18, 139:12
crude [1] - 132:22
cumulatively [1] -
190:3
curiosity [1] - 30:2
curious [1] - 105:13
current [9] - 10:13,
10:14, 31:7, 45:11,
66:16, 86:9, 105:7,
132:1, 186:9
Current [1] - 103:17
curve [5] - 129:10,
129:11, 130:9,
130:10, 156:4
cut [12] - 94:17, 94:19,
121:6, 131:5,
131:10, 131:19,
143:17, 150:3,
171:3, 173:3, 173:4,
173:15
cut-off [1] - 121:6
cycle [3] - 20:23,
20:25, 21:2	101:20, 103:16,
104:23, 111:8,
112:12, 121:1,
121:2, 121:23,
124:1, 124:2, 130:5,
130:6, 130:15,
130:23, 130:24,
131:2, 131:6,
131:20, 133:8,
133:9, 134:15,
136:18, 136:19,
137:1, 138:4, 147:3,
147:4, 147:5, 147:6,
147:9, 147:11,
147:22, 149:1,
154:10, 156:19,
157:9, 157:25,
158:1, 158:2, 158:4,
158:21, 160:1,
161:15, 162:17,
162:25, 165:13,
165:21, 165:23,
165:24, 166:4,
166:25, 167:4,
169:21, 173:14,
174:2, 179:5, 179:9,
179:23, 180:1,
181:7, 183:9,
183:12, 183:14,
183:20, 183:24,
184:9, 184:17,
184:24, 185:15,
185:25, 186:2,
186:11, 189:4	113:23, 115:25
decision [4] - 68:22,
114:1, 115:15,
115:16
deep [1] - 119:20
deficiency [1] - 51:15
define [9] - 13:25,
16:16, 19:2, 40:21,
42:21, 43:4, 62:22,
63:1, 142:20
defined [2] - 41:8,
44:13
defines [1] - 51:11
definition [7] - 13:24,
14:5, 141:9, 142:15,
142:22, 143:3,
155:16
definitions [3] - 141:6,
155:15, 156:15
degree [2] - 152:1,
153:17
demonstrate [1] -
25:19
denied [6] - 16:19,
16:20, 17:7, 51:13,
51:14, 52:17
deny [1] - 51:23
Department [13] -
12:3, 12:9, 12:12,
12:23, 12:24, 19:1,
29:19, 44:2, 44:4,
44:9, 62:4, 113:9
Department's [1] -	
		D			
		
dad [1] - 87:20			
		Dade [1] - 93:21			
			database [3] - 35:11,	34:7	
		daily [3] - 92:20,			
			88:9, 90:17	dependent [1] -	
		97:20, 101:1			
			date [1] - 121:6	135:18	
		darn [1] - 155:22			
			days [22] - 8:10, 31:8,	described [1] - 146:8	
		data [145] - 9:9, 9:23,			
			31:21, 32:17, 33:9,	describing [2] -	
		12:6, 19:21, 22:24,			
			33:18, 51:14, 69:20,	144:15, 146:22	
		22:25, 23:9, 31:7,			
			97:10, 97:14, 97:15,	description [4] -	
		31:17, 31:18, 31:22,			
			98:4, 98:8, 98:9,	14:19, 28:15, 48:16,	
		31:25, 32:11, 32:12,			
			100:14, 100:15,	164:20	
		32:13, 33:4, 34:5,			
			104:1, 120:24,	descriptive [1] - 39:14	

detail [1] - 53:23	
140:25, 142:7,	
23:9, 23:10, 23:15,	
22:18, 23:7, 43:7,	
81:19	
detailed [1] - 59:3	154:11, 155:15,	23:19, 24:2, 24:5,	48:16, 49:8, 59:11,	double-spaced [1] -	
details [1] - 49:12	159:5, 159:6, 159:9,	25:18, 27:21, 29:14,	101:3, 160:16, 172:9	15:22	
detect [1] - 21:18	168:10, 172:14,	29:17, 31:9, 32:9,	districts [82] - 9:22,	down [17] - 5:24,	
detention [1] - 177:15	173:15, 177:24,	34:9, 37:15, 39:11,	10:2, 10:5, 12:2,	14:14, 26:25, 31:14,	
determine [3] - 41:1,	178:23, 180:10,	40:11, 40:14, 41:7,	12:5, 12:14, 12:23,	35:15, 43:12, 86:22,	
52:15, 92:23	181:8, 182:14, 187:9	41:15, 42:10, 42:11,	15:12, 20:21, 23:14,	87:10, 88:15,	
determined [4] - 91:5,	differently [5] - 9:16,	44:5, 44:18, 45:24,	26:3, 27:7, 27:15,	102:16, 102:24,	
152:21, 162:7,	15:7, 119:22,	46:4, 46:11, 46:17,	27:17, 28:1, 28:6,	111:24, 118:10,	
176:15	159:14, 182:14	46:19, 48:1, 49:5,	28:24, 29:2, 29:12,	124:17, 183:2,	
determining [3] -	difficult [4] - 32:7,	49:6, 50:18, 51:20,	30:19, 34:6, 35:22,	187:7, 188:3	
95:19, 111:7, 111:8	74:3, 88:22, 179:2	52:23, 53:9, 54:13,	38:20, 39:18, 40:18,	draft [1] - 9:8	
develop [1] - 27:8	difficulty [1] - 135:19	55:10, 56:5, 56:23,	42:25, 44:3, 44:11,	draw [1] - 68:16	
developed [4] - 38:20,	dilemma [1] - 32:4	58:1, 58:3, 58:4,	44:15, 47:25, 52:20,	drawback [1] - 13:16	
62:2, 70:24, 85:12	directly [1] - 100:24	59:9, 59:19, 59:23,	55:2, 55:4, 56:3,	drew [1] - 68:7	
developing [1] -	disabilities [2] -	60:7, 61:8, 61:15,	58:13, 58:14, 59:15,	drop [2] - 65:12, 65:18	
142:16	107:22, 186:12	62:7, 62:8, 62:15,	64:7, 66:1, 66:4,	dropped [1] - 137:24	
development [10] -
20:16, 20:19, 20:20,
57:24, 58:2, 59:5,
61:5, 62:10, 102:2,
102:16
developmental [7] -
86:8, 94:13, 94:23,
95:1, 113:22, 120:4,
122:17
develops [1] - 29:17
deviations [2] -
129:13, 149:21
diagnosis [1] - 108:2
diagram [1] - 140:2
diamond [3] - 134:11,
140:11, 151:10
died [1] - 87:20
dies [1] - 90:8
difference [7] - 56:24,
86:10, 94:2, 120:3,
120:5, 120:6, 120:7
Difference [1] - 93:19
differences [2] -
86:14, 180:16
Different [1] - 30:13
different [62] - 7:7,
13:15, 17:19, 24:24,
26:3, 26:4, 26:10,
30:14, 34:22, 39:17,
39:18, 41:24, 66:9,
67:1, 67:12, 72:23,
72:25, 75:10, 80:3,
93:11, 93:23, 94:6,
101:4, 107:9,
109:20, 117:11,
117:14, 119:6,
119:21, 124:21,
124:25, 125:16,
125:20, 125:21,
126:1, 126:5, 132:6,
135:3, 135:7, 135:8,
136:6, 137:1, 137:2,
137:22, 140:17,	disability [16] - 91:4,
91:7, 91:8, 96:4,
101:14, 102:21,
105:11, 105:21,
106:11, 107:4,
107:16, 108:1,
187:6, 187:8, 187:10
disabled [3] - 91:6,
105:17, 107:8
disadvantaged [1] -
97:1
disadvantages [1] -
89:5
disadvantaging [1] -
39:24
disagree [1] - 113:1
discipline [5] -
177:12, 178:3,
179:5, 179:9, 179:17
disconnect [1] - 189:6
discrepancies [1] -
39:2
discretion [1] - 43:8
discussed [2] - 156:1,
179:5
discussing [1] - 46:12
discussion [1] - 57:3
distance [1] - 151:9
distinctions [1] -
190:16
distribution [6] -
127:4, 127:6,
127:10, 127:15,
127:25, 128:9
District [1] - 11:8
district [173] - 11:24,
12:4, 12:9, 14:10,
14:20, 16:3, 16:11,
16:20, 16:24, 17:1,
17:14, 17:16, 18:4,
18:21, 18:23, 20:15,
20:23, 21:13, 22:14,
22:21, 22:24, 23:8,	62:21, 63:12, 64:2,
64:24, 65:4, 66:22,
67:17, 70:10, 72:1,
72:17, 74:11, 76:17,
76:25, 77:4, 79:22,
80:3, 80:16, 80:21,
81:21, 81:23, 83:21,
84:7, 85:1, 85:3,
90:14, 90:23, 92:22,
93:5, 99:8, 99:16,
99:25, 100:1, 100:5,
100:20, 100:21,
100:23, 100:25,
101:2, 108:5,
110:15, 110:19,
118:4, 118:19,
120:25, 123:7,
123:23, 125:13,
125:22, 126:20,
130:8, 131:18,
131:21, 132:6,
132:7, 132:18,
137:7, 137:20,
142:24, 148:10,
149:3, 149:9, 150:1,
158:1, 158:2,
158:25, 159:22,
160:8, 160:20,
161:11, 163:2,
163:3, 163:5,
163:20, 165:10,
167:16, 168:22,
168:24, 169:7,
169:11, 170:15,
170:18, 170:25,
171:19, 171:20,
172:7, 172:10,
174:14, 182:3,
182:9, 185:19,
186:15, 187:2,
187:3, 187:4,
187:20, 189:16,
189:19
district's [10] - 17:6,	67:8, 69:8, 71:9,
72:6, 72:12, 75:10,
79:9, 81:18, 82:12,
82:19, 85:2, 97:19,
98:21, 100:6, 110:3,
112:12, 121:14,
121:24, 123:13,
123:14, 130:6,
131:4, 146:18,
149:19, 150:18,
156:20, 160:18,
161:6, 161:17,
161:21, 165:4,
165:12, 165:13,
165:17, 165:19,
166:3, 166:7, 170:5,
170:12, 179:25,
183:23, 183:24
divide [1] - 170:14
divorce [1] - 87:19
document [4] - 49:19,
50:3, 50:6, 53:8
Documentation	- 49:4
documentation	- 16:3, 53:10
documents	- 3:6,
7:11, 49:11
DOE	- 26:16, 49:9,
49:14, 64:22, 68:23,
132:23, 132:25,
150:17, 186:20,
186:25
done	- 8:14, 36:7,
46:21, 52:22, 62:6,
69:12, 93:22, 99:23,
118:20, 121:22,
124:7, 142:12,
150:25, 160:23,
164:16, 165:18,
181:22, 182:2,
188:25
double [2] - 15:22,	drug [1] - 177:11
due [1] - 12:7
duly [1] - 49:15
during [11] - 4:14, 9:5,
10:13, 20:23, 24:5,
24:24, 25:2, 35:5,
56:11, 99:4, 181:15
dyslexia [1] - 186:25
dyslexic [1] - 186:22

E

e-mail [2] - 175:9
earliest [2] - 10:16,
11:1
early [7] - 55:9, 55:24,
57:8, 75:3, 75:5,
98:24, 161:9
earn [1] - 170:2
earned [5] - 125:9,
141:11, 190:3,
190:10
earning [2] - 163:15,
163:16
earth [3] - 27:9, 78:10,
78:16
easier [3] - 123:18,
135:22, 140:7
easy [1] - 74:7
economic [3] - 95:14,
95:22, 178:7
editing [1] - 7:15
Education [5] - 7:12,
10:13, 12:23, 18:6,
18:14
educational [1] - 45:4
educator [1] - 10:11
Educator's [1] - 58:25
effect [6] - 10:10,
38:2, 38:4, 132:3,
150:6, 183:5
effective [61] - 19:13,	

[1]

[2]

[3]

[10]

[19]

39:16, 39:22, 39:23,	
employees [1] - 49:7	
151:24, 152:10,	
55:25, 56:14, 57:22,	
96:4, 112:16	
59:6, 83:22, 83:23,	employment [3] -	152:11, 152:12,	58:8, 59:4, 59:9,	examined [1] - 67:24	
128:17, 137:8,	40:14, 64:2, 67:25	153:17, 153:20,	59:11, 59:12, 59:16,	example [24] - 12:21,	
137:23, 138:3,	encourage [2] - 7:1,	154:17, 157:23,	59:20, 60:10, 60:17,	19:5, 19:6, 19:12,	
138:11, 138:16,	68:14	157:24, 158:10	61:11, 61:20, 62:6,	20:9, 50:18, 60:4,	
138:17, 138:20,	end [29] - 10:23, 11:2,	errors [14] - 134:22,	62:9, 62:17, 63:10,	66:8, 80:15, 96:19,	
138:23, 140:9,	27:8, 43:20, 54:21,	141:19, 141:20,	63:12, 63:19, 63:20,	97:17, 98:22,	
140:22, 141:7,	56:1, 56:2, 56:18,	141:22, 144:5,	64:4, 64:9, 66:21,	102:24, 124:5,	
141:8, 141:23,	57:6, 57:14, 64:2,	145:9, 151:21,	67:20, 68:7, 68:10,	128:4, 135:20,	
141:24, 142:10,	72:19, 76:11, 78:12,	151:25, 152:25,	69:6, 83:9, 83:15,	149:6, 152:22,	
142:14, 143:3,	95:5, 95:6, 95:7,	153:14, 155:8,	84:9, 87:8, 88:17,	156:17, 160:9,	
143:4, 143:15,	97:22, 121:12,	157:5, 157:17	93:6, 102:10, 110:4,	160:12, 169:21,	
143:18, 144:25,	121:21, 127:4,	escapes [1] - 17:25	118:22, 126:14,	170:25, 172:18	
146:6, 148:3, 148:5,	137:17, 157:8,	ESE [1] - 186:11	128:6, 137:10,	exams [2] - 60:5,	
149:24, 152:20,	161:13, 169:11,	essentially [1] -	138:21, 139:8,	137:17	
152:22, 152:24,	188:4, 188:14,	180:16	139:15, 146:21,	exceed [3] - 143:18,	
153:15, 153:20,	190:24	establish [7] - 82:14,	147:13, 147:21,	149:22, 171:14	
153:22, 154:8,	End [1] - 77:11	83:6, 85:3, 96:17,	148:5, 155:25,	exceeded [6] - 38:18,	
158:7, 159:19,	end-course [1] - 27:8	128:18, 128:20,	163:21, 164:12,	123:12, 123:20,	
170:5, 170:21,	end-of-course [5] -	131:5	165:3, 165:22,	143:10, 171:18,	
170:23, 171:1,	76:11, 78:12,	established [3] -	166:5, 173:20	171:24	
171:3, 171:7,	137:17, 161:13,	60:25, 132:20,	evaluations [23] -	excellent [1] - 188:25	
171:22, 172:1,	169:11	169:15	9:12, 10:11, 31:7,	Except [1] - 152:1	
173:4, 173:16,	end-of-the-year [1] -	et [1] - 149:24	32:10, 40:22, 41:2,	except [3] - 20:21,	
183:2, 185:1,	56:1	Ethics [1] - 63:24	41:7, 41:11, 44:10,	77:22, 129:1	
189:13, 189:21	ended [4] - 121:7,	ethnicity [1] - 95:22	44:12, 47:15, 48:4,	exception [3] - 15:9,	
effectively [1] - 149:23	138:2, 138:23,	evals [1] - 40:18	48:6, 48:15, 58:10,	90:7, 178:24	
eight [7] - 87:5,	190:25	evaluate [2] - 22:5,	58:11, 59:20, 62:1,	exceptions [1] - 69:12	
119:20, 119:23,	ending [1] - 188:12	163:23	70:9, 87:8, 161:6,	excessive [1] - 35:13	
120:5, 123:14,	ends [4] - 18:16,	Evaluated [2] - 186:4	176:18, 185:1	exclude [1] - 95:14	
149:20, 190:4	66:14, 121:12, 140:6	evaluated [23] - 22:7,	evaluative [1] - 179:13	exempt [1] - 106:1	
eighth [17] - 61:13,	English [4] - 92:17,	24:4, 24:17, 33:24,	evaluator [9] - 21:25,	exercising [1] - 59:23	
71:16, 71:21, 75:17,
76:1, 76:7, 76:8,
76:13, 79:21, 80:7,
80:11, 80:19, 81:15,
81:16, 81:25, 97:2,
101:10
either [9] - 59:14,
105:22, 109:23,
125:5, 134:3, 157:2,
158:6, 172:12
elaborated [1] -
175:21
electronic [1] - 166:2
elementary [4] -
98:23, 110:12,
143:22, 154:20
elements [1] - 19:14
eleventh [2] - 75:2,
81:7
eligible [1] - 148:2
eliminate [1] - 109:25
ELL [1] - 112:2
elsewhere [2] - 40:16,
40:19
employed [3] - 39:5,
40:13, 46:16
employee [4] - 48:15,
54:3, 54:5, 83:8	96:4, 101:13, 135:1
ensure [3] - 17:22,
27:10, 30:4
entail [1] - 51:6
entering [3] - 106:21,
107:12, 157:12
entire [5] - 97:13,
130:4, 139:15,
141:14, 190:8
entirely [1] - 152:19
entity [1] - 45:23
EOC [3] - 55:21,
77:20, 79:23
equal [2] - 136:13,
169:14
equally [2] - 85:3, 85:5
equated [2] - 77:11,
170:21
equivalent [1] - 79:11
error [29] - 37:14,
124:3, 124:6,
125:15, 125:23,
125:24, 126:19,
134:12, 134:17,
134:18, 141:17,
142:4, 142:5,
143:19, 144:1,
145:9, 151:20,	46:18, 47:20, 54:20,
54:25, 69:18, 70:1,
71:20, 137:21,
149:1, 163:8,
166:25, 167:24,
183:12, 183:16,
184:16, 185:20,
185:24, 186:5
evaluates [1] - 46:9
evaluating [2] - 164:4,
164:17
Evaluation [1] - 11:9
evaluation [101] -
10:15, 11:25, 12:4,
12:10, 12:14, 12:17,
12:18, 13:3, 13:6,
14:10, 14:17, 14:24,
15:1, 15:13, 16:25,
18:23, 20:14, 21:6,
22:19, 26:4, 26:8,
26:18, 31:16, 31:21,
32:1, 32:16, 33:7,
33:8, 40:25, 46:10,
46:25, 47:23, 48:4,
49:8, 49:12, 49:20,
50:1, 50:3, 50:12,
50:19, 51:20, 52:9,
53:10, 54:18, 55:14,	22:6, 25:23, 26:6,
29:11, 56:16, 62:2,
69:17, 70:15
evaluators [5] - 26:19,
29:7, 29:16, 49:7,
63:17
event [1] - 17:16
eventual [1] - 97:1
eventually [1] - 139:14
everywhere [1] -
163:13
evidence [2] - 14:19,
47:15
exact [3] - 28:8, 107:3,
142:25
Exactly [1] - 140:3
exactly [16] - 27:1,
30:19, 30:25, 36:2,
39:25, 40:5, 47:17,
89:10, 130:2,
131:12, 138:25,
139:3, 140:18,
144:6, 146:21,
151:14
exam [4] - 76:11,
78:13, 105:25,
164:10
examine [3] - 11:17,	exist [1] - 186:3
existing [1] - 183:10
exists [1] - 115:8
expect [9] - 7:8, 22:20,
49:9, 86:3, 87:21,
95:20, 140:3, 156:3
expectation [3] -
80:23, 93:17, 95:7
expectations [2] -
114:16, 174:21
expected [19] - 37:19,
38:6, 38:17, 38:18,
44:16, 44:25, 69:5,
86:10, 86:15, 93:24,
107:2, 122:16,
122:21, 123:12,
123:20, 140:14,
171:14, 171:19,
171:25
expecting [1] - 69:11
expects [1] - 25:18
expensive [1] - 70:11
experience [3] -
40:15, 41:2, 41:12
experienced [1] -
32:19
experts [1] - 112:19
expire [1] - 69:12	

explain [6] - 36:17,	
151:11, 161:10,	
final [3] - 17:14, 57:7,	
47:7, 59:1, 92:3,	
free [3] - 64:4, 76:25,	
60:9, 61:16, 114:25,	175:13	164:10	94:12, 112:18,	77:1	
118:19, 124:6	faring [1] - 41:16	finalize [2] - 31:10,	161:17, 161:24	friend [1] - 79:20	
explained [3] - 16:22,	fault [1] - 115:1	31:15	flow [1] - 188:15	friends [2] - 7:1, 116:5	
85:21, 99:13	FCAT [48] - 65:6,	finalized [2] - 32:16,	flowing [1] - 135:12	front [5] - 3:22, 19:9,	
explaining [2] -	70:25, 71:23, 72:8,	33:8	flush [2] - 169:5,	102:9, 175:2, 175:4	
159:18, 188:25	73:18, 76:5, 76:6,	Finally [1] - 60:12	174:7	FTE [2] - 35:5, 108:11	
explicit [2] - 42:23,	76:12, 76:19, 79:1,	finally [3] - 26:25,	Flush [1] - 174:7	FTEs [1] - 108:13	
42:24	79:24, 80:19, 81:17,	77:23, 146:25	flying [1] - 13:16	full [6] - 16:21, 21:8,	
expo [1] - 184:12	82:21, 84:24, 90:1,	fine [3] - 52:4, 98:11,	folks [5] - 4:23, 7:3,	46:17, 46:18, 51:12,	
expressed [1] - 69:22	91:24, 91:25, 92:1,	176:19	30:13, 63:18, 118:19	51:18	
extend [1] - 161:5	92:5, 92:11, 94:16,	finish [1] - 4:22	follow [2] - 16:13,	full-time [2] - 46:17,	
extent [1] - 68:14	96:20, 98:1, 98:2,	fired [1] - 21:12	135:24	46:18	
extra [12] - 116:15,	104:8, 105:18,	first [49] - 11:7, 11:19,	following [2] - 10:10,	fully [2] - 16:17, 52:16	
156:9, 156:10,	105:19, 105:21,	13:24, 19:21, 24:3,	190:9	functioning [1] -	
156:11, 156:12,	106:3, 121:15,	32:6, 34:12, 35:22,	foot [1] - 138:18	39:11	
156:23, 156:24	135:20, 135:21,	40:11, 40:14, 40:22,	forbid [2] - 66:24,	functions [3] - 45:23,	
extract [1] - 100:8	135:23, 136:7,	41:7, 52:13, 54:18,	109:11	46:4, 169:10	
extracted [1] - 80:25	138:2, 157:13,	55:13, 56:21, 67:17,	forever [2] - 150:19,	fundamentally [1] -	
extremely [1] - 7:20	157:23, 157:24,	72:6, 73:12, 76:15,	184:4	190:11	
extremes [1] - 141:5	159:9, 159:10,	76:21, 77:21, 82:18,	forget [1] - 190:17	funds [1] - 18:9	
eyes [3] - 112:9,	160:10, 160:23,	92:10, 92:18, 92:25,	forgot [1] - 51:22		
112:17, 130:16	162:25, 170:4,	95:17, 96:20, 99:15,	form [4] - 15:11,	G	
					
	177:25, 184:3	122:5, 128:5, 141:9,	15:15, 15:17, 105:22		
				
gain [4] - 95:3, 95:4,	
F	fear [1] - 115:9
fearful [1] - 161:8	145:24, 147:1,
147:14, 162:6,	formative [1] - 27:23
formed [1] - 165:18		

fact [18] - 10:24, 26:7,
39:12, 123:14,
129:16, 129:17,
130:2, 130:17,
131:8, 133:16,
140:15, 140:24,
154:5, 158:22,
169:2, 174:19,
179:9, 190:2
facto [1] - 184:12
factored [1] - 119:22
factors [3] - 101:17,
115:9, 180:18	February [6] - 8:10,
37:10, 37:13,
108:12, 157:12,
157:13
feedback [6] - 7:23,
9:7, 82:8, 126:25,
131:9, 139:19
felt [3] - 68:15, 68:25,
177:14
FERPA [1] - 45:3
few [10] - 8:10, 8:22,
15:21, 22:8, 34:15,
47:24, 77:14,
152:22, 155:14,	163:11, 169:6,
173:6, 173:17,
175:3, 178:21,
182:5, 182:7,
184:15, 184:16,
184:20, 188:10
First [5] - 4:3, 7:7,
107:19, 140:1,
146:21
fit [1] - 178:6
five [35] - 16:8, 16:15,
16:16, 16:22, 20:22,
20:24, 21:9, 21:14,
21:21, 44:3, 44:5,	formerly [2] - 19:16,
19:17
forms [4] - 28:13,
49:5, 136:6, 136:14
formula [30] - 37:25,
38:1, 70:19, 71:1,
72:24, 72:25, 77:9,
78:17, 82:3, 82:5,
85:22, 85:23, 91:10,
91:21, 91:22, 92:1,
92:4, 93:4, 98:7,
104:13, 104:15,
104:20, 104:23,
105:5, 106:4,	100:7, 113:20	
					
				gains [3] - 115:21,	
				123:17, 149:23	
				game [1] - 57:10	
				gang [1] - 177:14	
				gender [1] - 95:23	
				generally [2] - 120:1,	
				120:3	
				generate [2] - 33:4,	
				184:12	
				gentleman [1] -	
				131:20	
				geography [3] -	
				172:25, 173:2, 173:8	
faculty [1] - 59:5	177:24	44:16, 74:2, 94:16,	114:20, 114:24,	Georgia [1] - 92:9	
failed [2] - 115:24,	fidelity [2] - 27:5,	94:18, 94:22, 97:21,	115:5, 138:6, 165:6	gifted [6] - 38:3, 38:4,	
185:2	27:10	98:12, 116:13,	formulas [5] - 73:2,	86:21, 102:21,	
fair [3] - 106:5,	field [4] - 77:21, 101:8,	117:6, 117:7, 117:9,	78:9, 84:21, 84:25,	102:24, 178:1	
168:15, 168:16	101:18, 119:14	117:11, 118:5,	85:4	Gifted [1] - 93:8	
fall [7] - 39:14, 51:16,	fields [2] - 63:4, 66:18	118:24, 119:9,	forth [5] - 12:24,	given [19] - 17:10,	
52:1, 116:25, 129:9,	fifth [12] - 44:21,	124:9, 124:10,	52:20, 111:17,	36:17, 39:13, 45:7,	
144:17, 186:16	44:22, 45:1, 45:2,	124:18, 125:1,	175:19, 177:3	98:1, 105:3, 125:4,	
Fallen [2] - 135:5,	177:8, 184:1, 184:3,	153:4, 167:6,	forward [1] - 161:20	125:21, 126:1,	
135:6	185:12, 185:14,	178:22, 190:2	four [15] - 16:8, 16:9,	126:5, 134:4,	
familiar [8] - 20:17,	185:16, 185:17,	five-year [2] - 20:22,	16:13, 39:15, 54:6,	134:21, 135:3,	
58:20, 59:8, 60:12,	186:1	20:24	75:7, 94:18, 94:22,	135:7, 135:9, 136:7,	
61:12, 84:18, 127:2,	figure [4] - 52:5, 78:3,	fix [2] - 55:8, 131:16	95:18, 95:24,	140:17, 140:25,	
133:4	113:13, 187:3	flag [1] - 93:21	106:23, 117:14,	142:7	
family [1] - 87:18	file [3] - 99:17, 110:16,	flip [1] - 189:15	136:8, 167:6, 185:11	Given [1] - 156:24	
fancy [1] - 168:14	121:22	floor [2] - 83:21, 83:22	fourth [2] - 8:13, 69:10	glad [3] - 111:1, 111:2,	
far [13] - 3:7, 30:25,	fill [4] - 3:12, 4:24,	Florida [17] - 11:23,	frame [3] - 17:10,	121:1	
35:21, 41:20, 56:3,	31:18, 49:25	13:14, 45:17, 45:19,	52:4, 54:25	goal [4] - 83:25,	
56:11, 56:17, 81:10,	filled [1] - 3:13	45:20, 46:3, 46:9,	frankly [2] - 131:3,	121:25, 167:19,	
89:25, 137:3,	filling [1] - 129:24	46:15, 46:22, 46:24,	138:6	167:23	

goals [7] - 61:1, 79:6,	
graphic [2] - 134:5,	
85:13	
35:21, 36:1, 36:19,	
137:18, 138:9,	
164:18, 164:19,	151:8	grunt [1] - 138:21	37:4, 37:21, 38:8,	138:25, 139:3,	
168:18, 173:24,	graphics [1] - 151:16	guarantee [1] - 90:20	38:12, 39:25, 40:3,	139:24, 141:18,	
174:20	grateful [1] - 131:1	guess [3] - 32:1,	40:5, 40:9, 40:20,	143:24, 144:3,	
God [2] - 66:24,	great [17] - 6:21, 22:9,	117:17, 178:9	41:5, 41:25, 42:14,	144:6, 144:20,	
109:11	26:21, 27:12, 32:5,	guidance [3] - 138:19,	43:15, 43:17, 44:7,	144:22, 144:24,	
GPAs [1] - 163:16	33:20, 33:21, 42:14,	148:19, 161:23	44:18, 45:6, 45:9,	145:4, 145:6,	
grade [93] - 28:9,	44:7, 45:6, 45:9,	Guidance [1] - 161:24	45:19, 46:2, 47:6,	145:11, 145:14,	
44:21, 44:22, 44:23,	49:17, 65:14, 65:15,	guidelines [8] - 28:1,	47:11, 47:16, 47:19,	145:16, 145:19,	
45:22, 46:5, 61:13,	98:13, 111:14,	28:23, 28:25, 29:7,	48:9, 48:19, 49:17,	145:23, 146:1,	
67:17, 71:5, 71:6,	154:22	29:10, 29:18, 29:25,	50:15, 51:8, 54:1,	146:24, 148:4,	
71:16, 71:21, 72:2,	Great [1] - 133:5	62:4	54:10, 54:14, 55:2,	148:8, 148:16,	
73:7, 73:9, 73:10,	greater [1] - 38:5	guys [2] - 23:20, 53:25	55:12, 55:18, 55:24,	148:21, 148:25,	
73:13, 73:16, 74:1,	Greek [1] - 113:5		56:22, 57:15, 64:10,
64:19, 64:21, 65:20,	149:13, 150:1,
150:6, 150:12,	
74:5, 74:6, 74:9,	green [4] - 95:9,	H			
					
74:15, 74:17, 74:25,	95:20, 102:23,		67:7, 68:5, 68:24,	150:21, 151:3,	
		
half [5] - 129:16,			
75:1, 75:2, 75:4,	113:11		70:13, 71:18, 71:24,	151:12, 151:15,	
75:6, 75:13, 75:14,	greened [1] - 98:11		72:17, 72:25, 73:4,	151:22, 152:1,	
		129:18, 129:25,			
75:16, 75:17, 76:1,	grew [1] - 170:6		73:11, 74:19, 74:22,	152:6, 152:13,	
		153:2, 153:3			
76:8, 76:13, 78:3,	grieving [1] - 177:4		75:15, 75:24, 76:3,	152:16, 153:2,	
		hammering [1] -			
79:21, 80:7, 80:11,	group [13] - 98:7,		76:24, 77:13, 78:1,	153:5, 153:11,	
80:19, 80:21, 81:9,	98:8, 112:15,	174:19	78:6, 78:20, 79:17,	154:1, 154:3,	
					
81:15, 81:17, 81:25,	125:16, 125:21,	hand [1] - 115:23	80:1, 80:10, 81:1,	154:22, 154:24,	
					
83:2, 96:21, 96:25,	126:1, 129:20,	handbooks [1] - 49:4	81:10, 81:18, 81:23,	155:4, 157:15,	
					
97:2, 101:10,	140:17, 142:7,	handle [1] - 134:3	83:18, 83:21, 84:11,	157:20, 158:11,	
					
102:14, 102:19,	142:10, 142:11,	handout [2] - 6:24,	84:20, 87:22, 88:3,	158:18, 159:11,	
					
102:25, 108:22,	158:8, 178:23	13:7	88:8, 88:19, 88:25,	159:15, 159:22,	
					
109:13, 122:12,	groups [2] - 25:8,	handouts [4] - 3:4,	89:14, 89:17, 90:18,	161:3, 162:9,	
					
		3:15, 3:21, 7:4	90:22, 91:22, 93:1,	162:12, 162:20,	
128:4, 135:21,	165:18				
		happy [1] - 63:2	93:5, 95:16, 96:1,	163:2, 166:12,	
135:23, 137:25,	growth [72] - 9:11,				
		hard [2] - 116:2, 188:9	96:14, 96:18, 96:23,	166:17, 166:20,	
138:1, 138:15,	27:7, 31:18, 31:19,				
		head [4] - 88:15,	97:18, 98:13, 98:18,	166:24, 167:3,	
140:4, 141:13,	31:22, 31:25, 32:3,				
		98:14, 154:25, 155:5	99:6, 99:10, 100:4,	167:9, 167:15,	
148:12, 160:9,	32:12, 33:4, 34:24,				
		heading [1] - 46:7	100:10, 100:18,	167:23, 168:1,	
160:15, 160:21,	37:19, 38:6, 38:7,				
		health [6] - 105:22,	101:19, 101:23,	168:6, 172:15,	
162:7, 163:11,	57:25, 60:2, 60:6,				
		186:17, 186:19,	103:4, 103:17,	172:21, 174:3,	
165:5, 165:8, 165:9,	64:17, 70:21, 71:22,				
		186:21, 187:14,	103:20, 104:15,	175:2, 175:18,	
166:1, 166:2,	72:13, 74:3, 74:8,				
		187:15	104:21, 105:1,	176:13, 176:23,	
169:15, 177:8,	77:10, 82:15, 82:25,				
		hear [5] - 3:16, 5:6,	105:6, 106:7,	177:2, 178:12,	
177:16, 177:19,	83:1, 83:7, 84:8,				
		6:10, 45:25, 111:2	107:12, 107:18,	178:15, 178:19,	
177:21, 184:1,	84:21, 84:25, 85:4,				
		heard [5] - 133:16,	107:25, 108:5,	181:20, 182:1,	
184:3, 185:12,	85:5, 87:1, 92:12,				
		135:11, 137:3,	108:9, 109:18,	182:8, 182:20,	
185:14, 186:1	96:13, 96:17, 101:9,				
		146:5, 189:11	110:14, 110:20,	182:24, 183:7,	
graded [1] - 166:15	103:6, 103:7,				
		hearing [2] - 36:4,	111:1, 111:11,	183:11, 183:15,	
grader [4] - 45:1, 45:2,	111:10, 111:23,				
		188:9	112:7, 115:11,	183:22, 184:2,	
45:8, 74:16	113:10, 114:8,				
		heart [1] - 178:2	115:15, 115:18,	184:5, 184:7,	
graders [5] - 75:18,	119:2, 127:21,				
		HEBDA [302] - 10:22,	116:19, 116:23,	184:10, 184:14,	
76:7, 81:7, 81:13,	127:22, 128:3,				
			117:2, 117:9,	184:19, 184:23,	
170:16	128:21, 135:15,	14:25, 16:7, 17:5,			
			117:16, 117:20,	185:5, 185:8,	
grades [10] - 56:21,	167:5, 167:12,	17:8, 17:12, 17:16,			
			117:24, 118:16,	185:19, 186:4,	
73:23, 78:5, 94:1,	167:14, 169:14,	18:10, 18:16, 19:16,			
			119:16, 119:24,	186:7, 186:14,	
109:20, 109:21,	169:20, 170:9,	19:23, 20:6, 20:9,			
			120:10, 120:12,	187:4, 187:20,	
113:19, 163:15,	170:11, 170:15,	21:16, 21:24, 22:3,			
			120:20, 121:8,	187:23, 188:17,	
164:25, 189:7	170:22, 170:24,	22:9, 22:17, 23:21,			
			121:20, 129:8,	190:21	
graduate [2] - 74:21,	171:4, 171:5,	24:9, 24:25, 25:6,			
			129:14, 130:2,	held [3] - 96:25,	
75:9	171:10, 171:15,	25:21, 25:25, 26:11,			
			130:22, 131:23,	167:11, 167:12	
grained [1] - 26:25	171:25, 173:16,	26:14, 26:21, 27:12,			
			132:1, 133:5,	help [12] - 18:4, 28:1,	
grandchild [2] -	174:15, 179:4,	29:9, 29:13, 29:21,			
			133:11, 133:15,	47:7, 88:22, 90:22,	
108:18, 109:11	180:11, 180:12,	30:13, 31:11, 32:5,			
			133:23, 135:3,	91:3, 109:15, 135:1,	
grandson [1] - 96:21	181:10, 185:4	32:23, 33:7, 33:11,			
			135:6, 135:17,	153:13, 157:17,	
Granted [1] - 115:3	Growth [2] - 71:2,	33:20, 34:4, 35:18,			
			135:25, 137:12,	159:17, 162:12	

helpful [2] - 7:20, 8:5	
hold [2] - 36:3, 136:5	
27:6, 27:11, 32:6,	
161:22	
79:5, 173:19	
helps [3] - 5:10, 27:21,	holding [1] - 78:17	82:4	individually [1] -	instrument [3] - 24:7,	
47:5	home [2] - 146:3,	implemented [1] -	163:4	24:22, 119:3	
hidden [2] - 114:10,	190:22	67:10	indulgence [1] -	instruments [3] -	
158:16	homebound [2] -	implements [1] -	190:20	26:20, 52:10, 61:24	
high [13] - 75:8, 81:5,	66:11, 66:12	52:21	industry [1] - 60:4	intend [2] - 7:22,	
107:11, 107:14,	honest [1] - 115:9	implications [1] -	inequitable [1] - 139:2	11:16	
143:9, 143:14,	honors [1] - 76:7	189:21	influence [2] - 113:16,	intending [1] - 4:13	
172:20, 177:10,	hoopla [1] - 88:13	important [16] - 9:25,	180:21	inter [12] - 22:1, 22:6,	
177:11, 180:11,	hoot [1] - 76:13	10:5, 24:18, 24:19,	inform [3] - 49:6, 50:4,	22:23, 24:6, 24:22,	
180:24, 189:19	hoping [4] - 7:6, 7:14,	37:23, 52:8, 68:16,	50:5	25:5, 25:8, 25:20,	
high-crime [1] -	84:24	82:16, 94:9, 123:25,	information [25] -	26:9, 30:4, 30:11,	
177:10	hospital [2] - 66:11,	130:13, 146:12,	37:23, 39:1, 53:4,	30:14	
high-drug [1] - 177:11	66:12	155:17, 158:19,	57:5, 59:3, 71:10,	Inter [1] - 26:12	
high-growth [1] -	hours [2] - 4:1, 188:2	168:6, 190:14	79:9, 102:10, 104:6,	inter-rater [12] - 22:1,	
180:11	house [1] - 39:20	improvement [13] -	105:24, 108:6,	22:6, 22:23, 24:6,	
high-performing [1] -	huge [5] - 106:15,	39:16, 58:3, 61:2,	112:10, 121:25,	24:22, 25:5, 25:8,	
189:19	143:25, 177:9,	62:11, 128:19,	123:15, 123:22,	25:20, 26:9, 30:4,	
high-poverty [1] -	177:11, 189:21	129:4, 141:8,	123:24, 126:9,	30:11, 30:14	
177:11	human [2] - 188:22,	142:16, 142:22,	128:12, 135:9,	Inter-rater [1] - 26:12	
higher [6] - 107:10,	190:12	153:25, 156:9,	146:17, 165:3,	interest [2] - 6:5, 45:4	
129:21, 135:22,	hundred [3] - 30:6,	157:4, 189:17	165:5, 168:24, 175:6	interesting [1] - 73:3	
144:18, 170:9, 173:4	73:20, 162:9	improvements [1] -	informing [2] - 58:2,	interests [1] - 77:18	
higher-performing [1]	hunt [1] - 29:24	5:12	62:5	interim [2] - 27:23,	
- 107:10	hurricanes [1] - 66:25	inaudible [3] - 42:13,	initial [4] - 16:9, 52:19,	164:9	
highest [4] - 76:16,		61:25, 115:20	135:10, 142:15	interrupt [1] - 188:15	
127:5, 141:1, 177:25	I	inaudible) [3] -	injury [6] - 105:23,	interval [12] - 125:23,	
highly [40] - 19:13,		148:24, 153:10,	106:11, 106:15,	126:4, 126:21,	
	
idea [5] - 41:16, 55:6,				
39:15, 39:22, 39:23,		186:13	106:19, 106:24,	133:17, 134:17,	
83:23, 128:17,		include [17] - 12:5,	107:4	140:16, 143:5,	
137:8, 137:23,	56:15, 136:20,	16:4, 31:24, 32:3,	innovating [1] - 19:14	152:5, 152:9, 153:7	
138:3, 138:11,	154:14	48:3, 51:23, 55:14,	input [20] - 6:15, 7:5,	investigate [1] - 23:1	
138:16, 138:20,	identical [3] - 3:7,	63:17, 67:5, 67:6,	7:7, 8:7, 8:18, 48:12,	involved [1] - 190:5	
140:9, 140:22,	136:7, 168:13	67:7, 69:9, 95:19,	48:15, 48:17, 51:20,	issue [4] - 97:9,	
141:22, 141:24,	identified [7] - 101:8,	129:25, 180:2, 180:6	51:22, 51:23, 51:25,	103:21, 165:22,	
142:14, 143:3,	101:14, 153:24,	included [17] - 4:11,	52:6, 62:13, 62:16,	188:23	
143:15, 143:18,	163:24, 189:17,	13:5, 34:23, 37:11,	62:24, 63:18, 68:19,	issues [6] - 35:14,	
146:6, 148:3, 148:5,	189:20, 189:23	49:10, 59:4, 67:20,	175:1, 175:5	38:9, 66:19, 88:1,	
149:24, 152:20,	identifying [1] -	72:10, 86:5, 86:21,	inputs [1] - 10:18	111:15, 115:12	
152:22, 152:24,	153:15	89:2, 89:22, 92:11,	instance [1] - 184:9	item [2] - 27:16, 28:10	
153:15, 158:6,	IEP [2] - 108:24, 109:2	92:15, 105:4, 105:7,	instances [1] - 94:5	items [1] - 27:18	
159:19, 170:5,	ignore [1] - 144:10	106:12	instead [3] - 60:1,	iterative [1] - 18:4	
170:20, 170:23,	II [2] - 74:21, 75:9	includes [3] - 46:25,	80:22, 149:20	itself [7] - 13:11,	
171:1, 171:22,	illogical [1] - 30:9	49:4, 57:24	Instead [1] - 152:10	70:20, 85:22,	
172:1, 173:4,	imagine [4] - 44:20,	including [4] - 47:23,	Institute [1] - 114:7	140:10, 140:12,	
173:16, 183:2, 185:1	75:11, 106:24, 174:3	59:22, 153:22, 180:3	instruction [2] - 59:6,	140:24, 142:1	
Hillsborough [3] -	impact [7] - 10:2,	incorporates [1] -	181:5		
39:21, 179:12	10:4, 17:14, 89:4,	107:8	Instructional [1] -	J	
hired [12] - 24:16,	111:9, 182:14,	increase [2] - 152:19,	11:8		
				
Jacksonville [1] - 8:14	
40:11, 40:13, 40:21,	190:15	153:21	instructional [23] -		
41:6, 41:15, 42:9,	impacts [1] - 189:1	indicated [1] - 54:19	12:13, 32:13, 40:12,		
54:13, 54:19, 54:24,	impaired [6] - 105:22,	indicates [1] - 17:9	42:16, 42:17, 42:18,	Janette [1] - 107:20	
62:1, 188:1	186:17, 186:19,	indicator [1] - 93:20	48:13, 58:10, 58:19,	January [2] - 55:3,	
historical [3] - 66:3,	186:21, 187:14,	indicators [1] - 61:24	58:23, 58:24, 61:9,	97:17	
78:13, 172:18	187:15	individual [16] - 10:2,	61:10, 68:12, 79:12,	job [13] - 35:15, 41:4,	
historically [1] - 97:19	implement [1] - 52:3	10:4, 23:14, 28:19,	147:20, 147:25,	50:18, 58:12, 68:11,	
hit [5] - 97:5, 97:7,	Implementation [2] -	37:20, 56:9, 60:24,	148:10, 162:17,	76:14, 76:23, 124:8,	
144:20, 144:21,	71:2, 85:13	61:5, 61:10, 87:9,	164:6, 168:18,	124:11, 124:16,	
144:24	implementation [7] -	91:7, 91:15, 94:25,	174:16, 174:18	157:8, 168:19,	
hits [1] - 57:7	12:10, 17:19, 17:20,	100:12, 100:17,	instructionally [2] -	188:25	

jobs [1] - 78:18	
kids' [2] - 35:12,	
100:15, 109:24,	
146:25, 169:5,	
literally [1] - 149:18	
Joe [1] - 39:19	186:12	113:12, 113:18,	174:7, 174:8	literature [3] - 24:10,	
Johnny [1] - 100:13	Kim [1] - 162:6	117:15, 117:18,	legislative [3] - 8:23,	68:5, 90:1	
judge [2] - 15:5, 15:9	kind [34] - 5:9, 7:5,	129:3, 131:4, 131:5,	9:6, 10:25	live [2] - 76:22, 76:24	
judgment [4] - 126:12,	7:7, 8:6, 8:17, 39:7,	132:18, 135:24,	legislature [1] -	livelihood [1] - 112:4	
133:20, 144:8,	49:19, 52:10, 54:22,	168:20	115:14	livelihoods [1] -	
144:10	55:5, 55:8, 55:21,	laundry [2] - 52:11,	length [2] - 151:11,	111:17	
judgments [1] -	57:18, 62:24, 73:13,	61:18	151:14	lives [3] - 78:18,	
126:17	73:22, 73:25, 85:22,	law [57] - 4:4, 9:1,	less [6] - 38:6, 64:17,	130:7, 189:3	
juices [1] - 135:12	90:10, 90:16, 91:8,	11:14, 12:15, 13:5,	70:11, 122:21,	load [1] - 185:16	
July [5] - 10:9, 60:14,	103:2, 107:6,	15:5, 15:9, 16:25,	165:13, 189:20	local [6] - 9:23, 56:23,	
121:24, 121:25,	112:19, 114:13,	17:19, 17:20, 17:24,	lesson [2] - 25:11,	59:14, 59:24, 60:20,	
167:2	117:25, 139:7,	18:18, 18:24, 24:15,	25:13	160:16	
jump [1] - 139:4	170:3, 175:20,	29:14, 30:20, 32:3,	letters [2] - 113:6,	logic [2] - 129:15,	
jumping [1] - 161:8	176:17, 178:2,	32:8, 32:9, 32:25,	175:8	154:2	
June [4] - 9:4, 10:9,	179:6, 179:25,	41:6, 46:25, 47:19,	letting [1] - 150:9	logical [1] - 78:14	
10:13, 70:25	182:16	50:20, 50:25, 51:4,	level [27] - 19:15,	long-term [1] - 117:22	
justify [2] - 100:2,	kindergarten [7] -	51:21, 53:4, 56:13,	39:13, 92:22, 94:17,	longest [1] - 14:22	
100:5	67:17, 163:12,	58:5, 58:8, 58:18,	94:18, 94:19, 95:2,	look [62] - 3:7, 7:10,	
juvenile [1] - 177:14	168:1, 168:8, 168:9,	58:22, 59:2, 61:20,	95:3, 101:2, 106:22,	7:25, 9:6, 11:8,	
	168:11, 172:16
kinds [19] - 4:21, 8:4,	62:11, 62:15, 83:6,
83:12, 84:6, 85:2,	107:1, 107:14,
108:22, 113:20,	11:13, 11:15, 13:7,
13:10, 14:20, 20:15,	
K					
	18:6, 27:24, 51:2,	95:18, 132:9,	113:21, 113:22,	20:25, 23:6, 23:9,	

K-12 [1] - 47:1					
	57:25, 63:8, 66:21,	142:13, 147:15,	115:22, 119:13,	34:17, 36:1, 39:20,	
	71:25, 72:3, 80:3,	150:7, 150:19,	135:19, 142:4,	41:14, 52:24, 53:14,	
keep [9] - 6:1, 13:17,	113:6, 146:11,	150:22, 150:23,	165:9, 168:3,	53:16, 53:25, 54:6,	
83:24, 134:2, 143:4,	163:14, 165:24,	164:2, 164:3,	168:22, 169:15,	56:11, 57:21, 57:22,	
144:4, 153:6,	170:17, 181:6,	167:13, 167:15,	172:11	61:15, 61:23, 73:14,	
173:22, 174:19	181:14, 187:9	175:18, 176:14	Leveling [1] - 101:7	85:5, 86:6, 86:9,	
kept [1] - 101:2	knowing [3] - 10:4,	laws [4] - 17:23,	leveling [1] - 101:17	95:24, 96:10, 97:3,	
keyboard [1] - 3:18	37:10, 42:20	44:13, 44:14, 45:3	levels [3] - 94:16,	98:7, 98:15, 100:13,	
kid [9] - 75:12, 88:17,	known [1] - 57:8	lead [1] - 22:25	94:22, 107:7	102:20, 103:8,	
97:16, 99:2, 103:24,	knows [2] - 14:20,	leadership [5] - 58:11,	levelwide [1] - 61:13	104:10, 106:7,	
107:14, 107:15,	136:23	58:23, 59:2, 61:25	Levy [1] - 28:7	106:10, 106:13,	
180:20, 186:24		learner [4] - 92:17,
96:5, 101:13	license [1] - 114:10
lie [1] - 140:19	112:10, 112:16,
115:19, 115:21,	
kid's [2] - 104:7,	L				
					
165:25		learning [23] - 57:25,	Lied [1] - 135:2	118:22, 119:24,	
	
labeled [2] - 3:8				
kidding [1] - 131:1		59:13, 60:2, 60:24,	lied [2] - 15:25, 134:25	123:13, 136:20,	
kids [49] - 33:24,		70:21, 85:4, 85:5,	lies [1] - 14:18	140:7, 149:8,	
	laboratory [1] - 45:22				
34:18, 35:3, 35:4,		86:20, 87:2, 101:14,
107:7, 113:16,
113:19, 115:20,
115:21, 123:17,
149:23, 164:19,
169:14, 173:24,
174:20, 177:13
least [11] - 6:13, 9:4,
10:23, 59:12, 68:17,
75:8, 117:14,
120:24, 120:25,
121:2, 185:21
leave [9] - 10:22,
21:14, 62:21, 63:3,
63:8, 66:20, 110:7,
168:5, 174:25
leaving [1] - 176:7
Lee [3] - 8:15, 136:3,
142:25
left [8] - 5:20, 43:7,
100:21, 119:9,	life [2] - 109:8, 110:25
likely [3] - 28:11, 72:5,
121:2
Likewise [2] - 140:23,
186:21
limit [1] - 128:16
limited [1] - 143:23
line [16] - 5:4, 5:5,
5:25, 6:25, 8:20,
10:21, 27:19, 37:7,
43:22, 45:3, 68:7,
68:16, 85:17,
114:23, 140:5,
152:19
link [1] - 175:1
linked [1] - 138:4
list [10] - 18:12, 19:23,
36:22, 61:18, 86:22,
105:3, 106:12,
165:2, 180:19
lists [1] - 52:11	149:10, 151:8,
151:9, 165:20,
168:22, 169:3,
173:14
Look [1] - 171:12
looked [8] - 35:25,
71:3, 87:5, 137:5,
160:3, 161:14,
173:10, 187:5
looking [32] - 10:25,
13:1, 13:2, 20:18,
22:24, 25:11, 27:1,
28:14, 39:19, 45:11,
59:11, 71:5, 77:17,
78:7, 97:25, 100:2,
104:6, 111:8,
112:23, 112:24,
114:6, 122:6, 122:7,
122:25, 123:2,
138:7, 139:12,
155:23, 158:19,	
36:22, 56:11, 56:17,	lack [1] - 46:8				
56:20, 77:22, 80:19,	laid [2] - 32:8, 130:16				
80:25, 81:16, 88:12,	lain [1] - 134:25				
92:24, 93:25, 98:23,	language [20] - 29:14,				
100:8, 102:3,	33:14, 34:25, 39:14,				
103:13, 104:11,	49:15, 65:10, 70:13,				
104:16, 107:10,	70:14, 89:7, 91:1,				
115:24, 119:22,	92:17, 96:5, 101:13,				
119:23, 120:3,	107:21, 107:23,				
120:18, 120:22,	119:12, 119:17,				
137:21, 138:2,	120:9, 125:10, 149:9				
138:22, 142:7,	large [4] - 39:6, 155:8,				
147:8, 162:7, 163:1,	158:2, 172:4				
163:13, 164:9,	last [31] - 8:12, 15:3,				
165:25, 166:21,	18:10, 18:11, 29:7,				
167:20, 168:19,	31:16, 52:22, 65:5,				
171:18, 173:1,	69:19, 69:23, 70:2,				
173:12, 174:12,	77:2, 77:9, 86:8,				
174:21, 177:22,	86:11, 96:21, 97:5,				
177:24, 181:2	97:21, 99:11,				

158:20, 173:23,	
materials [1] - 181:5	
167:17, 171:10	
71:15, 71:19, 72:14,	
151:7, 151:13,	
180:14	maternity [2] - 63:7,	measured [6] - 88:2,	72:23, 73:1, 73:6,	151:17, 151:23,	
looks [4] - 10:8, 53:2,	66:20	88:24, 89:1, 89:8,	74:14, 74:20, 75:5,	152:3, 152:7,	
53:3, 156:19	math [24] - 70:20,	94:6, 147:8	75:23, 75:25, 76:4,	152:14, 152:17,	
lose [3] - 168:3, 168:4,	73:18, 75:7, 76:13,	measurement [4] -	77:6, 77:16, 78:2,	153:3, 153:6,	
177:18	76:19, 80:7, 80:11,	124:4, 124:6,	78:7, 79:16, 79:19,	153:13, 154:2,	
losing [1] - 157:8	81:3, 81:17, 81:21,	134:13, 143:19	80:9, 80:24, 81:5,	154:14, 154:23,	
lost [1] - 117:4	85:10, 89:12, 91:2,	measures [6] - 60:1,	81:14, 81:20, 83:16,	155:1, 156:22,	
loudly [1] - 6:9	97:2, 97:4, 128:4,	69:6, 72:13, 74:11,	83:20, 84:6, 84:18,	157:16, 157:22,	
love [1] - 160:5	133:22, 157:24,	149:1	87:15, 87:23, 88:5,	158:15, 159:8,	
low [9] - 83:8, 132:21,	158:5, 159:10,	measuring [3] - 60:2,	88:6, 88:10, 88:21,	159:13, 159:17,	
156:14, 172:18,	170:16, 184:3,	70:20, 170:24	89:12, 89:16, 90:16,	159:21, 161:1,	
177:18, 180:12,	185:13	meat [1] - 14:16	90:19, 91:18, 92:24,	161:4, 162:11,	
180:25, 181:10	mathematics [3] -	mechanism [1] -	93:2, 95:13, 95:25,	162:19, 162:22,	
low-growth [2] -	34:25, 89:7, 125:11	48:14	96:12, 96:15, 96:19,	162:23, 166:9,	
180:12, 181:10	mathematics' [2] -	mechanisms [1] -	96:24, 98:5, 98:17,	166:14, 166:18,	
lower [4] - 19:22,	169:23, 169:25	48:12	98:20, 99:7, 99:25,	166:21, 167:1,	
93:17, 146:9, 157:24	matter [7] - 39:10,	meet [11] - 75:11,	100:6, 100:11,	167:4, 167:10,	
lowers [1] - 93:15	53:7, 76:13, 91:2,	114:15, 142:17,	101:7, 101:22,	167:22, 167:25,	
lowest [2] - 127:5,	91:13, 91:14, 126:10	148:15, 148:17,	103:1, 103:15,	168:2, 172:13,	
129:19	matters [5] - 91:7,	149:22, 156:16,	103:18, 104:13,	172:17, 174:1,	
	93:14, 94:7, 181:8,	171:14, 183:1,	104:19, 104:22,	174:24, 175:13,	
M	186:5	183:6, 185:2	105:3, 105:10,	175:22, 176:19,	
	mean [41] - 4:20, 14:4,	meeting [4] - 5:3,	107:6, 107:17,	177:1, 177:6,	

ma'am [28] - 10:19,					
	16:16, 16:17, 16:18,	119:9, 188:5, 188:13	107:19, 108:2,	178:14, 178:16,	
	19:2, 28:6, 37:13,	meetings [1] - 179:6	108:7, 108:10,	181:17, 181:23,	
21:10, 25:22, 27:4,	38:3, 38:5, 40:17,	meets [2] - 51:21,	110:12, 110:13,	182:5, 182:12,	
37:6, 40:3, 41:18,	47:18, 54:2, 68:9,	79:11	110:18, 110:24,	182:21, 182:25,	
43:25, 48:10, 64:6,	73:8, 75:9, 81:11,
85:23, 96:16, 108:3,
109:1, 109:14,
111:17, 111:18,
111:21, 111:23,
118:9, 121:19,
122:19, 122:21,
124:19, 128:12,
129:12, 133:9,
137:8, 149:3,
149:19, 156:12,
166:2, 186:23
Meaning [1] - 140:12
meaning [5] - 14:16,
68:1, 89:6, 140:2,
168:13
meanings [1] - 14:2
means [21] - 8:3,
34:12, 39:17, 40:11,
40:20, 48:18, 53:5,
62:21, 68:25, 73:1,
81:12, 85:24, 90:25,
102:15, 122:13,
122:14, 124:19,
125:15, 145:6,
160:16, 174:8
meant [1] - 69:1
meantime [4] - 69:24,
74:10, 114:2, 162:13
measure [9] - 27:6,
60:6, 60:8, 72:21,
74:3, 92:12, 147:7,	MEMBER [310] -
10:20, 14:23, 16:6,
17:3, 17:6, 17:9,
17:13, 18:8, 18:15,
19:11, 19:18, 20:4,
20:8, 21:11, 21:23,
21:25, 22:4, 22:16,
23:18, 24:1, 24:21,
25:1, 25:17, 25:23,
26:1, 26:12, 26:15,
27:5, 29:6, 29:10,
29:20, 30:2, 31:5,
31:12, 32:22, 33:2,
33:10, 33:13, 33:16,
33:18, 33:22, 35:2,
35:19, 35:24, 36:15,
36:21, 37:7, 37:22,
38:10, 39:2, 39:9,
40:2, 40:4, 40:8,
40:10, 40:24, 41:19,
42:8, 43:10, 43:16,
44:1, 44:15, 44:20,
45:7, 45:16, 47:5,
47:9, 47:12, 47:17,
48:8, 48:11, 49:2,
50:13, 51:5, 53:21,
54:9, 54:11, 54:15,
55:6, 55:13, 55:20,
56:20, 57:13, 64:7,
64:13, 64:20, 64:22,
67:3, 67:21, 68:22,
69:16, 70:3, 70:4,	111:5, 111:12,
114:18, 115:12,
115:17, 116:5,
116:21, 116:25,
117:6, 117:11,
117:17, 117:21,
118:13, 119:12,
119:17, 120:8,
120:11, 120:15,
120:21, 121:19,
128:25, 129:9,
129:15, 130:20,
131:17, 131:25,
132:17, 133:6,
133:12, 133:21,
135:2, 135:5,
135:15, 135:18,
137:5, 137:14,
137:19, 138:10,
139:1, 139:23,
141:16, 143:21,
143:25, 144:4,
144:13, 144:21,
144:23, 145:2,
145:5, 145:8,
145:12, 145:15,
145:17, 145:22,
145:25, 146:23,
148:1, 148:6,
148:14, 148:19,
148:22, 149:17,
150:4, 150:10,
150:17, 151:1,	183:8, 183:13,
183:17, 183:25,
184:3, 184:6, 184:8,
184:11, 184:18,
184:21, 184:25,
185:6, 185:9, 186:2,
186:6, 186:8,
186:15, 187:12,
187:22, 188:7,
188:11, 188:19
members [3] - 19:7,
19:8, 137:20
mention [2] - 20:10,
98:20
mentioned [5] - 18:22,
51:19, 61:19, 63:16,
179:23
merge [1] - 73:2
met [8] - 38:17, 56:19,
71:3, 83:14, 123:11,
123:20, 125:1,
171:24
method [3] - 20:2,
129:6, 160:17
methods [1] - 54:16
metric [4] - 69:6,
69:10, 82:13
mid [4] - 54:20, 55:4,
55:5, 168:3
mid-level [1] - 168:3
mid-year [3] - 54:20,
55:4, 55:5	
69:15, 71:14, 73:5,					
75:22, 79:17, 79:18,					
81:4, 87:14, 91:17,					
101:6, 105:9, 137:4,					
156:21, 157:21,					
159:16, 175:12,					
177:5, 186:7					
made-up [1] - 133:1					
magic [1] - 151:23					
mail [2] - 175:9					
maintaining [1] -					
135:19					
majority [3] - 72:7,					
75:15, 185:22					
makeup [1] - 41:10					
mandatory [2] - 150:5,					
150:7					
manner [1] - 106:5					
manual [7] - 50:19,					
53:8, 53:13, 53:15,					
54:2, 54:3, 54:5					
manuals [1] - 49:5					
March [6] - 8:21, 8:24,					
9:5, 9:18, 10:23,					
11:2					
mastery [2] - 163:25,					
164:8					
match [3] - 36:24,					
109:21, 110:5					
matched [1] - 121:16					
matching [1] - 99:17					

middle [8] - 16:2,	
168:25, 169:18,	
43:15, 43:17, 44:7,	
144:22, 144:24,	
must [9] - 19:14,	
66:8, 66:16, 71:15,	171:17, 172:4,	44:18, 45:6, 45:9,	145:4, 145:6,	39:14, 55:14, 67:23,	
140:8, 140:11,	179:3, 179:15,	45:19, 46:2, 47:6,	145:11, 145:14,	144:16, 144:17,	
156:2, 180:8	180:4, 180:5, 181:9,	47:11, 47:16, 47:19,	145:16, 145:19,	169:13, 171:13,	
midpoint [1] - 41:14	181:11	48:9, 48:19, 49:17,	145:23, 146:1,	183:6	
might [16] - 3:10, 5:11,	modeled [1] - 15:11	50:15, 51:8, 54:1,	146:24, 148:4,		
				N	
10:12, 42:9, 45:4,	modeling [3] - 66:3,	54:10, 54:14, 55:2,	148:8, 148:16,		
65:2, 92:15, 113:1,	123:10, 127:4	55:12, 55:18, 55:24,	148:21, 148:25,		
				
name [4] - 4:18, 4:25,	
128:14, 138:11,	models [3] - 41:20,	56:22, 57:15, 64:10,	149:13, 150:1,		
143:8, 152:17,	74:14, 90:1	64:19, 64:21, 65:20,	150:6, 150:12,		
				6:17, 120:19	
152:19, 153:6,	modifications [3] -	67:7, 68:5, 68:24,	150:21, 151:3,		
				names [2] - 19:7,	
154:16, 155:12	16:24, 18:22, 47:10	70:13, 71:18, 71:24,	151:12, 151:15,		
				35:12	
migrant [1] - 101:11	modifies [1] - 174:8	72:17, 72:25, 73:4,	151:22, 152:1,		
				narrative [2] - 15:16,	
mild [1] - 107:9	modify [4] - 17:1,	73:11, 74:19, 74:22,	152:6, 152:13,		
				15:17	
mildly [1] - 105:17	18:23, 49:15, 133:13	75:15, 75:24, 76:3,	152:16, 153:2,		
				national [7] - 112:15,	
million [1] - 88:8	monitor [19] - 9:6,	76:24, 77:13, 78:1,	153:5, 153:11,		
				112:19, 158:13,	
mind [4] - 48:19,	12:9, 12:19, 13:1,	78:6, 78:20, 79:17,	154:1, 154:3,		
				169:9, 172:19,	
83:24, 136:11,	20:11, 20:12, 20:21,	80:1, 80:10, 81:1,	154:22, 154:24,		
				173:10, 173:14	
145:24	22:2, 23:6, 26:17,	81:10, 81:18, 81:23,	155:4, 157:15,		
				nature [1] - 158:9	
mindset [1] - 177:14	26:24, 27:2, 28:20,	83:18, 83:21, 84:11,	157:20, 158:11,		
				near [2] - 14:7, 31:5	
mine [2] - 69:19, 79:20	30:8, 44:2, 44:5,	84:20, 87:22, 88:3,	158:18, 159:11,		
				necessarily [10] -	
Minus [1] - 145:5	44:11, 44:12, 44:16	88:8, 88:19, 88:25,	159:15, 159:22,		
				28:7, 45:14, 106:23,	
minus [7] - 124:17,
124:18, 124:19,
125:24, 134:18,
134:19, 145:3
minute [1] - 80:16
minutes [5] - 77:14,
100:14, 119:9,
146:2, 181:24
miss [2] - 42:6, 92:24
missed [2] - 75:22,
99:3
misses [1] - 99:1
missing [4] - 31:3,
34:19, 36:23, 93:2
mitigation [1] - 181:10
mix [2] - 72:2, 137:1
mobile [1] - 101:10
Mobility [1] - 93:10
mobility [2] - 104:2,
117:1
modal [1] - 93:19
model [50] - 71:5,
71:8, 73:8, 73:14,
74:8, 77:10, 77:24,
78:11, 79:1, 79:4,
81:11, 82:22, 85:11,
85:24, 86:6, 87:4,
87:13, 89:22, 90:12,
96:11, 98:2, 99:4,
100:16, 103:4,
112:16, 112:20,
113:5, 113:10,
114:5, 114:9,
116:24, 118:25,
119:2, 122:18,
123:16, 129:20,
129:23, 137:6,
159:2, 168:15,	monitored [1] - 26:2
monitoring [7] - 12:8,
20:14, 21:21, 22:21,
23:9, 56:8, 63:13
month [5] - 6:2, 8:9,
9:18, 31:15, 33:8
months [2] - 41:23,
177:20
most [11] - 22:7,
41:23, 47:21, 72:5,
73:18, 92:2, 93:3,
93:15, 105:12,
156:1, 156:6
Most [2] - 35:21, 180:7
move [5] - 42:10,
113:20, 113:21,
120:4, 161:20
moved [1] - 146:23
moves [1] - 42:16
MS [302] - 10:22,
14:25, 16:7, 17:5,
17:8, 17:12, 17:16,
18:10, 18:16, 19:16,
19:23, 20:6, 20:9,
21:16, 21:24, 22:3,
22:9, 22:17, 23:21,
24:9, 24:25, 25:6,
25:21, 25:25, 26:11,
26:14, 26:21, 27:12,
29:9, 29:13, 29:21,
30:13, 31:11, 32:5,
32:23, 33:7, 33:11,
33:20, 34:4, 35:18,
35:21, 36:1, 36:19,
37:4, 37:21, 38:8,
38:12, 39:25, 40:3,
40:5, 40:9, 40:20,
41:5, 41:25, 42:14,	89:14, 89:17, 90:18,
90:22, 91:22, 93:1,
93:5, 95:16, 96:1,
96:14, 96:18, 96:23,
97:18, 98:13, 98:18,
99:6, 99:10, 100:4,
100:10, 100:18,
101:19, 101:23,
103:4, 103:17,
103:20, 104:15,
104:21, 105:1,
105:6, 106:7,
107:12, 107:18,
107:25, 108:5,
108:9, 109:18,
110:14, 110:20,
111:1, 111:11,
112:7, 115:11,
115:15, 115:18,
116:19, 116:23,
117:2, 117:9,
117:16, 117:20,
117:24, 118:16,
119:16, 119:24,
120:10, 120:12,
120:20, 121:8,
121:20, 129:8,
129:14, 130:2,
130:22, 131:23,
132:1, 133:5,
133:11, 133:15,
133:23, 135:3,
135:6, 135:17,
135:25, 137:12,
137:18, 138:9,
138:25, 139:3,
139:24, 141:18,
143:24, 144:3,
144:6, 144:20,	161:3, 162:9,
162:12, 162:20,
163:2, 166:12,
166:17, 166:20,
166:24, 167:3,
167:9, 167:15,
167:23, 168:1,
168:6, 172:15,
172:21, 174:3,
175:2, 175:18,
176:13, 176:23,
177:2, 178:12,
178:15, 178:19,
181:20, 182:1,
182:8, 182:20,
182:24, 183:7,
183:11, 183:15,
183:22, 184:2,
184:5, 184:7,
184:10, 184:14,
184:19, 184:23,
185:5, 185:8,
185:19, 186:4,
186:7, 186:14,
187:4, 187:20,
187:23, 188:17,
190:21
multi [1] - 69:6
multi-metric [1] - 69:6
multiple [13] - 62:8,
67:3, 67:4, 67:13,
67:18, 67:22, 67:23,
67:25, 68:4, 68:25,
69:5, 91:15, 156:15
Multiple	- 68:25
multiply [1] - 102:1
music [3] - 160:5,
160:6, 160:7		
				107:25, 118:24,	
				121:8, 130:8,	
				137:12, 166:3,	
				188:23	
				necessary [1] - 131:19	
				need [56] - 3:18, 9:12,	
				21:1, 23:1, 26:25,	
				29:12, 31:1, 34:20,	
				35:15, 41:1, 41:13,	
				42:21, 42:22, 43:13,	
				47:13, 50:9, 51:12,	
				52:8, 54:6, 55:8,	
				57:22, 58:4, 58:6,	
				63:18, 64:15, 68:6,	
				78:21, 82:8, 90:10,	
				90:17, 97:9, 99:22,	
				102:8, 103:12,	
				106:20, 109:3,	
				109:15, 111:21,	
				113:25, 115:19,	
				115:25, 116:2,	
				122:3, 133:24,	
				147:2, 149:11,	
				156:18, 159:5,	
				159:17, 161:5,	
				162:18, 162:20,	
				168:12, 188:8	
				needed [3] - 13:25,	
				15:6, 15:15	
				needing [1] - 63:4	
				needs [19] - 33:15,	
				39:16, 55:8, 56:13,	
				57:17, 70:5, 79:12,	
				111:18, 126:20,	
				128:19, 129:4,	
				141:7, 142:16,	
				142:22, 153:25,	
				156:9, 157:4,	

[1]

174:19, 189:17	
normal [5] - 129:7,	
October [2] - 55:1,	
147:10, 149:19,	
order [3] - 19:12, 25:4,	
negative [3] - 111:9,	129:10, 130:8,	108:11	151:17, 151:20,	76:11	
122:19, 153:23	164:24	offended [1] - 134:9	151:24, 152:10,	orient [2] - 4:9, 13:9	
negotiate [2] - 51:1,	North [1] - 176:7	offering [2] - 173:5,	152:11, 152:23,	original [1] - 128:8	
77:7	note [2] - 31:13,	173:18	152:25, 153:3,	orthodontists [1] -	
negotiated [1] - 49:16	123:25	Office [1] - 58:25	153:4, 153:16,	98:24	
negotiations [1] -	nothing [6] - 57:5,	office [1] - 77:4	153:20, 155:25,	ought [4] - 7:18, 8:3,	
132:20	88:15, 88:18, 110:5,	officer [1] - 177:15	164:10, 165:15,	73:2, 131:10	
neighborhood [1] -	150:13, 151:23	official [1] - 4:17	167:6, 168:2,	outcome [1] - 59:21	
181:6	notice [4] - 11:11,	often [2] - 24:12, 82:8	168:20, 169:4,	outlines [1] - 53:23	
never [3] - 88:9, 162:8,	41:9, 119:14, 120:23	OHI [2] - 187:12,	169:17, 170:12,	outperformed [1] -	
176:1	notified [1] - 175:20	187:14	170:25, 171:17,	122:16	
new [14] - 9:8, 9:15,	notion [1] - 39:10	old [3] - 15:18, 65:2,	172:12, 174:5,	outside [2] - 33:6,	
12:15, 19:8, 21:2,	November [2] - 31:23,	115:21	174:10, 174:13,	178:1	
48:5, 108:23, 117:7,	55:3	on-line [8] - 5:4, 5:5,	176:6, 176:12,	overall [5] - 4:10,	
127:6, 127:7,	number [45] - 5:19,	5:25, 6:25, 8:20,	178:23, 179:6,	83:9, 83:15, 118:22,	
127:10, 164:14,	13:19, 17:25, 20:1,	10:21, 43:22, 85:17	179:20, 179:24,	157:1	
165:16
newly [11] - 24:16,
40:11, 40:13, 40:21,
41:6, 41:15, 42:8,
54:13, 54:19, 54:24,
62:1
newly-hired [4] -
24:16, 54:19, 54:24,
62:1
news [2] - 130:17,	24:14, 25:9, 29:8,
30:4, 36:16, 36:18,
47:12, 48:12, 67:8,
90:24, 93:10, 98:4,
98:6, 98:8, 106:25,
107:11, 111:25,
112:1, 119:18,
119:22, 120:22,
122:13, 122:14,
122:19, 123:11,	once [9] - 20:22,
20:23, 21:8, 24:4,
24:11, 36:6, 54:20,
84:23
one [145] - 3:7, 3:8,
3:13, 5:16, 7:21,
11:6, 11:7, 13:12,
15:8, 18:1, 18:2,
22:7, 22:20, 23:5,
29:24, 34:19, 35:21,	180:13, 182:6,
185:12, 185:16,
185:17, 186:1,
187:24, 189:5, 190:4
One [11] - 49:1, 69:17,
69:21, 94:14, 111:4,
117:3, 123:5, 123:9,
153:2, 156:22, 177:7
One-and-a-half [1] -
153:2	overrides [1] - 84:9
overview [3] - 4:3,
4:12, 7:21
overwhelm [1] -
180:21
own [18] - 18:23,
22:18, 27:8, 34:8,
38:16, 38:21, 39:11,
42:25, 44:6, 45:23,
46:4, 46:5, 90:1,	
130:18	125:25, 128:16,	35:23, 36:5, 38:8,	one-and-a-half [1] -	130:6, 131:5,	
next [18] - 4:1, 10:15,	134:14, 134:15,	38:14, 38:23, 39:15,	153:3	150:18, 165:18,	
14:13, 20:1, 31:23,	152:20, 157:2,	46:10, 47:13, 48:23,	one-fifth [2] - 185:16,	172:9	
36:7, 48:2, 48:20,	165:8, 165:9,	49:19, 49:23, 64:11,	185:17		
48:22, 57:16, 71:10,	171:12, 171:13,	65:9, 66:2, 66:4,	ones [20] - 9:14,	P	
					
86:23, 104:17,	171:18, 174:10,	66:7, 66:15, 67:11,	34:23, 35:8, 35:12,		
				
p.m [1] - 190:25	
131:25, 132:7,	174:13, 174:14,	70:24, 71:5, 72:8,	35:13, 45:12, 46:8,		
134:6, 177:16,	186:23	72:9, 72:11, 73:12,	47:1, 74:10, 75:6,		
182:22	numbered [1] - 15:25	75:25, 76:16, 77:17,	76:16, 84:22, 98:12,	Page [19] - 13:22,	
Next [2] - 63:4, 70:16	numbers [8] - 13:17,	79:13, 84:24, 86:13,	103:18, 105:14,	14:6, 14:7, 15:25,	
nine [1] - 29:8	78:16, 82:23, 129:2,	92:16, 94:17, 94:22,	109:2, 110:21,	16:1, 16:5, 16:6,	
ninth [19] - 71:5, 71:6,	132:21, 132:22,	95:21, 98:25, 99:11,	158:25, 159:6, 170:1	16:7, 18:21, 31:5,	
73:7, 73:10, 73:13,	154:24, 155:1	99:16, 102:6,	open [2] - 10:22,	39:12, 54:11,	
74:5, 74:9, 74:15,		102:17, 107:6,	121:7	120:24, 168:22,	
	O				
74:17, 74:25, 75:6,		108:24, 109:18,	open-ended [1] -	168:23	
75:12, 75:14, 75:16,		110:8, 110:10,	121:7	page [11] - 13:17,	
	
objective [2] - 26:6,				
78:2, 81:6, 81:9,		110:11, 110:22,	opened [1] - 69:20	13:22, 14:13, 20:1,	
81:13, 170:16		113:22, 114:4,	opinion [1] - 134:19	29:7, 102:9, 103:6,	
	26:18				
nobody [4] - 36:17,		114:25, 115:22,	opportunities [1] - 6:2	105:8, 132:16,	
	obligation [2] - 17:21,				
39:5, 114:11, 114:21		116:9, 117:7,	opportunity [1] -	132:18, 150:21	
	17:22				
non [1] - 164:5		117:15, 118:8,	79:14	pages [2] - 13:18,	
	observation [4] - 52:9,				
nonclassroom [2] -		121:9, 122:6,	opposed [1] - 187:18	15:21	
	56:14, 61:24, 119:3				
147:20, 147:25		123:13, 123:25,	option [13] - 32:9,	Panama [1] - 8:14	
	observations [4] -				
noncompliant [1] -		124:19, 126:13,	47:23, 59:15, 60:3,	paper [5] - 113:14,	
	24:10, 24:14, 25:2,				
17:17		130:14, 131:11,	60:8, 61:7, 64:16,	114:9, 149:12,	
	67:12				
none [1] - 26:5		132:7, 134:3,	66:2, 79:23, 113:3,	149:15, 179:7	
	observe [1] - 25:4				
nonFCAT [1] - 159:18		134:17, 135:25,	162:21, 167:16,	paragraph [4] - 8:2,	
	observed [3] - 19:14,				
noninstructional [1] -		136:2, 137:2,	173:21	31:6, 54:12, 120:24	
	24:23, 54:24				
59:19		137:20, 139:11,	optional [1] - 71:9	parameters [2] - 78:4,	
	occasionally [1] -				
nonsubstantial [1] -		139:12, 141:7,	options [4] - 18:13,	121:9	
	46:22				
19:9		141:16, 142:4,	59:23, 63:15, 63:17	paraprofessional [1] -	
	occurs [2] - 55:25,				
nonsubstantive [1] -		142:5, 142:18,	Options [1] - 63:16	42:18	
	157:1				
19:6		143:5, 146:7, 147:4,	Orange [1] - 28:7	Paren [1] - 168:22	

parent [6] - 51:19,	
126:16, 127:17,	
performance [37] -	
picking [2] - 65:23,	
poorly [1] - 100:9	
51:22, 51:23, 51:24,	127:25, 128:14,	27:7, 39:13, 54:16,	132:22	pops [1] - 175:10	
52:5, 90:8	128:17, 128:18,	54:17, 54:22, 55:15,	picture [1] - 144:14	population [4] - 129:6,	
parental [4] - 48:17,	128:21, 129:11,	55:16, 58:9, 59:16,	piece [3] - 123:25,	130:1, 130:7, 174:12	
62:12, 62:24, 68:19	129:17, 129:18,	60:1, 67:22, 67:23,	124:2, 159:23	portfolio [1] - 166:1	
parents [3] - 48:14,	129:24, 131:8,	72:21, 80:13, 80:14,	pieces [1] - 37:23	portion [3] - 45:20,	
62:16, 101:11	133:24, 139:18,	80:22, 82:20, 83:12,	piloting [2] - 38:14,	46:3, 162:7	
Part [1] - 112:11	150:9, 150:24,	85:1, 94:15, 126:13,	45:11	positive [3] - 111:9,	
part [23] - 6:8, 6:22,	153:22, 156:1,	128:6, 139:13,	place [5] - 17:18,	122:14, 179:17	
10:5, 32:1, 46:15,	157:11, 157:19,	140:3, 141:3, 142:8,	74:24, 116:7, 162:1,	possibilities [1] -	
50:17, 53:2, 53:9,	159:25, 164:22,	143:1, 146:16,	190:8	160:25	
53:11, 53:16, 59:8,	174:25, 175:19,	147:13, 158:23,	places [3] - 14:2,	possibility [1] -	
64:4, 82:7, 82:17,	182:14, 189:2	159:1, 159:3,	22:25, 119:11	141:25	
118:15, 139:16,	people's [3] - 78:18,	160:10, 160:13,	plain [1] - 115:21	possible [4] - 65:16,	
139:20, 140:20,	114:23, 163:17	160:20, 171:11,	plan [6] - 9:2, 44:6,	114:14, 141:1,	
146:25, 147:21,	per [2] - 89:23, 170:16	172:2	61:2, 61:5, 80:5,	156:22	
148:7, 148:9, 151:13	perceived [1] - 121:5	performed [6] - 90:7,	110:4	possibly [5] - 22:5,	
participate [1] - 34:6	percent [72] - 19:13,	122:11, 122:20,	plans [2] - 58:3,	118:6, 142:8, 157:3,	
participation [2] -	19:19, 19:20, 39:21,	140:13, 141:12,	163:18	174:25	
6:16, 175:5	39:23, 59:12, 59:17,	159:13	platform [1] - 27:20	post [3] - 5:3, 170:24,	
particular [6] - 12:1,	64:17, 65:13, 65:17,	performing [7] -	platforms [1] - 27:16	171:6	
60:2, 94:11, 102:20,	66:5, 73:20, 76:14,	87:17, 100:9,	play [1] - 106:6	posted [2] - 5:4, 8:19	
116:9, 120:16	84:7, 84:8, 97:12,	107:10, 107:11,	playing [3] - 101:7,	poverty [1] - 177:11	
parties [1] - 88:13	97:13, 97:14, 99:1,	180:25, 189:19	101:18, 119:13	PowerPoint [7] - 3:5,	
parts [3] - 4:2, 6:6,	101:16, 101:22,	perhaps [3] - 17:15,	plenty [1] - 6:2	3:17, 4:2, 6:23,	
54:3
Pasco [1] - 39:22
pass [8] - 9:1, 76:11,
166:22, 167:20,
167:21, 172:19,
173:11, 189:25
pass/fail [2] - 60:6,
60:7
passed [2] - 10:12,
76:15
Passing [1] - 76:12
past [7] - 45:12, 46:6,
76:4, 111:15, 134:7,
185:13, 190:2
patience [1] - 190:18
patterned [2] - 20:12,
20:13
peer [6] - 16:4, 47:23,
47:24, 48:3, 63:17,
67:11
penalized [1] - 88:23
pencil [2] - 13:19,
113:14
People [1] - 94:19
people [56] - 5:10,
5:11, 5:19, 5:21,
9:15, 10:21, 23:14,
25:8, 25:9, 25:12,
25:15, 28:5, 37:2,
38:25, 42:20, 54:20,
65:1, 65:16, 65:17,
66:25, 67:13, 68:10,
68:15, 69:14, 70:6,
90:25, 112:18,
114:3, 114:6, 125:2,	106:3, 123:10,
123:11, 124:13,
124:16, 124:20,
124:22, 125:12,
129:2, 129:11,
129:16, 129:18,
129:23, 130:1,
132:24, 133:3,
133:6, 133:17,
134:16, 134:20,
134:23, 138:12,
140:16, 140:25,
141:14, 143:5,
143:8, 145:6, 152:4,
152:5, 152:8, 152:9,
153:9, 153:17,
155:20, 156:12,
156:13, 158:5,
162:10, 170:6,
170:9, 170:21,
171:13, 171:18,
171:23, 172:1,
189:13, 189:16
percentage [7] - 98:9,
106:2, 107:23,
107:24, 128:18,
148:2, 149:22
percentages [1] -
59:16
percentile [1] - 127:13
percentiles [1] - 87:6
perform [3] - 94:11,
98:10, 105:19
Performance [1] -
59:7	96:25, 121:3
period [6] - 20:24,
92:25, 93:1, 97:21,
100:14, 110:22
periods [1] - 185:11
permission [1] -
131:22
permitted [1] - 27:8
person [14] - 25:3,
48:5, 55:7, 64:19,
64:20, 83:22, 87:9,
133:16, 136:12,
146:20, 147:3,
157:8, 183:9
person's [2] - 140:13,
155:24
personal [2] - 31:13,
44:12
Personnel [1] - 11:9
personnel [18] -
11:24, 12:4, 12:14,
40:12, 40:13, 42:17,
42:19, 47:14, 48:13,
59:20, 61:9, 62:2,
63:19, 87:8, 147:20,
147:25, 164:6
pertain	- 29:22
pertained [1] - 180:20
pertains [1] - 53:16
pertinent [1] - 54:4
ph [4] - 34:7, 35:5,
98:11, 172:19
ph) [1] - 58:25
phenomenal [1] -
76:16	plus [11] - 59:2, 95:8,
112:23, 124:17,
124:18, 125:24,
134:17, 134:18,
172:19, 189:6
Plus [2] - 5:10, 96:7
point [27] - 5:16, 5:21,
24:9, 28:17, 41:25,
46:20, 50:25, 53:20,
55:7, 57:1, 70:23,
72:5, 73:11, 77:23,
78:19, 99:21, 100:4,
120:4, 135:16,
136:6, 143:17,
149:7, 163:3,
169:24, 171:3,
173:4, 183:21
points [13] - 94:8,
102:2, 102:17,
122:17, 124:18,
125:4, 128:4, 128:8,
131:5, 131:11,
134:15, 173:3,
173:15
police [1] - 178:4
policies [2] - 49:5,
92:23
policy [2] - 110:15,
182:9
poll [5] - 124:9,
124:13, 125:13,
134:19
polled [1] - 124:20
polling [2] - 124:7,
124:8	85:18, 102:9, 175:3
PowerPoints [1] -
103:8
practical [3] - 77:17,
78:23, 190:15
practice [6] - 32:13,
58:10, 58:19, 58:24,
68:12, 148:10
practices [2] - 32:2,
58:25
pre [3] - 163:12,
170:24, 171:5
pre-K [1] - 163:12
precisely [1] - 131:13
precursor [1] - 78:14
predicted [4] - 86:7,
103:23, 105:1, 122:8
prediction [2] - 95:6,
107:13
predictor [3] - 90:2,
96:3, 101:9
premise [1] - 166:10
prepare [1] - 76:6
present [2] - 98:4,
104:1
presentation [8] - 4:4,
4:12, 4:14, 4:15, 7:9,
7:25, 82:17, 89:3
President [3] - 124:8,
124:11, 124:16
presumably [1] -
121:19
presume [1] - 30:9
presumes [1] - 120:17
pretest [3] - 73:16,	

[1]

73:20, 165:7	
proficient [1] - 165:10	
47:2	
raise [1] - 115:23	
161:8, 161:18,	
pretty [4] - 14:12,	profound [1] - 92:2	Public [1] - 39:19	ran [2] - 89:4, 129:1	161:21, 167:4,	
61:22, 74:7, 155:22	program [4] - 29:15,	publish [1] - 103:12	random [1] - 101:25	178:2, 179:1,	
preventing [1] - 110:6	29:16, 29:17, 88:11	published [1] - 169:22	randomly [2] - 125:18,	179:15, 180:17,	
previous [4] - 36:11,	programs [1] - 162:1	pulled [2] - 19:20,	125:19	180:24, 180:25,	
40:25, 41:3	progress [2] - 57:1,	100:3	range [1] - 126:1	182:22, 183:18,	
primary [4] - 162:23,	57:2	pulling [1] - 100:7	rank [2] - 127:12,	185:19, 185:24,	
187:5, 187:8, 187:10	progression [1] -	pure [1] - 129:16	129:5	187:16, 190:14	
principal [13] - 24:4,	163:17	purpose [2] - 35:1,	rankings [1] - 127:13	reason [19] - 17:17,	
25:1, 31:14, 31:15,	progressive [1] - 56:8	57:22	ranks [1] - 87:6	49:22, 56:4, 60:3,	
47:22, 56:16, 61:3,	prohibits [1] - 150:17	purposely [1] - 95:14	rate [2] - 172:19,	82:18, 83:4, 84:1,	
100:22, 118:17,	project [1] - 3:19	purposes [15] - 14:1,	173:11	84:11, 84:17, 87:3,	
137:11, 138:16,	projected [1] - 111:23	14:3, 27:22, 34:23,	rated [4] - 129:3,	126:8, 132:24,	
181:4	projects [1] - 163:14	61:11, 109:19,	170:4, 171:21,	139:13, 152:7,	
principal's [1] - 59:4	promised [1] - 57:11	109:22, 121:16,	189:24	162:15, 165:1,	
principals [4] - 30:6,
50:5, 59:1, 164:5
Principles [1] - 63:25
print [1] - 6:18
printed [1] - 13:14
probability [1] -
153:21
problem [2] - 24:2,
97:8
procedure [1] - 119:3
procedures [6] - 49:7,
49:13, 49:20, 63:5,
82:4, 119:5	promoted [1] - 163:16
promotion [1] - 93:20
properties [1] - 28:11
proportion [4] - 82:25,
127:21, 169:20,
170:10
proposal [2] - 21:7,
135:10
propose [1] - 146:4
proposed [12] - 7:11,
9:14, 9:24, 19:24,
24:3, 127:16,
129:20, 130:11,	139:11, 160:4,
163:21, 164:12,
164:17, 165:3,
165:22
pushing [1] - 88:12
put [9] - 11:16, 24:13,
29:21, 54:5, 90:11,
104:23, 108:22,
121:9, 147:17
Put [1] - 30:23
putting [3] - 88:21,
132:13, 157:18	rater [13] - 22:1, 22:6,
22:23, 24:6, 24:22,
25:5, 25:8, 25:20,
26:9, 26:12, 30:4,
30:11, 30:14
raters [1] - 24:24
Rather [1] - 93:20
rather [1] - 123:16
ratified [1] - 49:16
rating [5] - 137:24,
138:15, 138:17,
141:2, 189:25
ratings [1] - 58:5
rationale [2] - 95:13,
95:16
raw [1] - 120:21
reach [2] - 132:21,
135:22
read [4] - 8:2, 24:10,
109:15, 130:17
reading [18] - 34:24,
72:2, 89:6, 91:1,
98:23, 99:2, 99:3,
125:10, 155:23,
157:25, 158:8,
158:12, 158:13,
158:15, 159:10,
169:25
reads [1] - 49:4
ready [3] - 77:24,
134:5, 149:16
real [6] - 50:8, 97:19,
109:8, 110:25,
128:25, 184:17	174:17, 174:18,
185:13
reasons [16] - 7:21,
34:22, 82:15, 82:16,
83:11, 83:24, 92:14,
99:23, 114:4,
125:20, 136:9,
136:10, 136:25,
139:10, 165:2,
180:13
receive [2] - 45:21,
83:9
received [5] - 31:8,
125:14, 136:1,
142:18, 142:23
receiving [2] - 61:12,
92:18
recent [1] - 73:18
recently [1] - 169:22
recommended [5] -
71:5, 85:14, 85:15,
96:6, 96:7
record [10] - 30:23,
43:17, 108:15,
109:10, 110:8,
149:14, 186:11,
187:25, 188:3,
190:21
recordings [1] - 5:5
recourse [1] - 33:5
red [3] - 151:10,
152:11, 175:8
reduce [2] - 59:15,
64:16	
		Q			
process [47] - 10:6,	149:18, 188:24,				
12:8, 12:25, 14:3,	189:1, 189:8				
		
Q&A [3] - 5:16, 43:20,			
14:9, 15:3, 15:6,	proposing [6] - 21:3,				
15:18, 16:4, 16:10,	64:11, 132:4, 132:8,				
		43:24			
16:15, 17:4, 17:6,	133:12, 146:14				
		quadray [1] - 98:10			
17:18, 18:17, 20:14,	protocol [1] - 20:18				
		qualify [1] - 68:4			
20:15, 28:20, 31:6,	prove [1] - 111:19				
		quality [1] - 28:22			
32:10, 34:8, 34:9,	provide [21] - 9:7, 9:8,				
		question-and-			
36:13, 47:24, 49:3,	28:20, 32:10, 34:5,				
		answer [1] - 6:8			
49:13, 52:19, 53:24,	38:21, 48:15, 48:18,				
		questions [14] - 4:16,			
62:5, 63:13, 67:11,	55:22, 62:16, 63:9,				
		4:19, 5:8, 6:11,			
70:15, 71:4, 84:10,	66:23, 67:1, 84:15,				
		10:17, 11:4, 48:24,			
100:18, 101:3,	98:16, 104:6,				
		50:7, 57:14, 94:14,			
118:16, 121:11,	126:19, 142:19,				
		107:17, 136:1,			
131:4, 175:15,	142:21, 174:16				
		136:2, 174:6			
175:16, 175:19,	provided [9] - 29:18,				
		quick [2] - 97:19,			
176:21, 177:4,	32:11, 32:14, 62:4,				
		174:5			
179:13	62:12, 141:6,				
		quickly [3] - 57:19,			
processes [1] - 18:4	142:15, 175:7,				
		79:20, 174:24			
processing [1] -	176:14				
		quite [6] - 138:6,			
186:23	provides [4] - 28:2,				
		141:15, 141:23,			
produces [1] - 86:17	59:2, 60:17, 60:18		reality [1] - 161:16
realized [1] - 19:19
really [40] - 23:2,
41:24, 52:8, 52:21,
57:4, 60:6, 65:4,	reevaluated [1] -
115:13
reference [1] - 127:24
referrals [3] - 177:12,
178:3, 179:17	
		180:9			
Professional [1] -	providing [4] - 27:17,				
		quote [2] - 105:17,			
63:25	28:22, 28:23, 132:10				
		130:20			
professional [14] -	provision [2] - 65:22,				
20:16, 20:19, 20:20,	93:6				
		R			
32:2, 57:23, 58:2,	psychologist [1] -		66:11, 68:5, 68:9,	referred [1] - 149:20	
58:12, 61:5, 62:10,	108:3	
Race [4] - 15:13,
49:24, 69:3, 69:4
race [1] - 95:22	71:7, 73:7, 79:20,	Referring [1] - 45:16	
					
63:21, 63:23, 64:1,	psychometric [1] -		87:15, 93:23, 100:6,	reflect [1] - 169:13	
68:11, 160:2	136:9		106:4, 111:7,	reflects [1] - 177:23	
professionally [1] -	public [7] - 4:23, 5:15,		126:15, 129:25,	Regardless [1] - 83:16	
128:13	9:7, 45:21, 46:16,		142:1, 156:13,	regardless [3] - 40:15,	

137:10, 143:18	
reporter [9] - 5:1, 6:9,	
157:9, 170:20,	
11:20, 11:21, 12:1,	
saw [3] - 122:8,	
registered [2] - 124:9,	6:20, 43:12, 149:14,	171:13, 176:18	12:21, 12:22, 13:8,	125:13, 129:2	
124:25	188:1, 188:13,	resulting [1] - 82:23	13:14, 13:21, 13:23,	scale [12] - 86:8,	
regular [6] - 46:15,	190:19, 190:22	results [22] - 9:11,	14:1, 14:16, 15:3,	94:13, 94:23, 95:1,	
46:23, 47:1, 76:8,	reporting [5] - 21:5,	21:5, 25:16, 33:23,	15:16, 15:17, 19:24,	102:17, 113:22,	
80:7, 173:12	21:18, 21:20, 111:3,	34:3, 72:19, 72:20,	27:3, 32:7, 33:3,	120:4, 122:17,	
regulate [1] - 50:16	187:17	72:22, 73:7, 75:19,	38:9, 38:13, 40:1,	135:22, 150:2,	
regulations [1] - 49:6	reports [3] - 23:7,	77:1, 77:2, 80:12,	40:2, 40:6, 40:8,	150:3, 167:5	
regulatory [1] - 28:18	100:23, 187:4	80:22, 82:21, 82:23,	42:2, 42:7, 43:4,	scenario [2] - 98:22,	
reject [1] - 49:15	represent [3] - 26:3,	112:16, 165:5,	48:22, 52:15, 52:21,	178:21	
relate [2] - 125:7,	151:19, 188:22	165:8, 172:3, 172:9	53:17, 57:18, 59:10,	scheduled [3] -	
186:9	representation [1] -	retain [1] - 105:24	63:23, 64:11, 70:13,	109:24, 110:1, 119:9	
related [2] - 59:5,	126:3	retained [1] - 163:17	70:16, 70:17, 70:18,	scheme [1] - 142:25	
94:21	represents [4] -	retention [2] - 93:20,	71:1, 71:11, 82:1,	School [11] - 11:9,	
relationship [2] -	125:11, 134:12,	93:21	82:7, 84:12, 85:10,	45:18, 45:19, 45:20,	
90:23, 112:11	140:6, 185:16	retested [1] - 127:6	85:21, 89:3, 114:10,	46:3, 46:9, 46:15,	
released [1] - 121:3	reprint [1] - 19:8	retire [1] - 182:22	133:12, 133:13,	46:22, 46:24, 47:8,	
relevant [1] - 90:24	require [4] - 64:3,	review [23] - 12:13,	139:6, 149:8, 150:6,	161:24	
reliability [13] - 22:1,	67:24, 109:21, 110:4	13:2, 14:11, 14:21,	152:23, 159:23,	school [110] - 9:22,	
22:6, 22:23, 24:6,
24:22, 25:5, 25:8,
25:20, 26:9, 26:12,
30:5, 30:11, 30:14
reliable [1] - 97:4
remain [1] - 158:16
remainder [2] -
142:20, 142:21
remaining [1] - 116:12
Remember [5] - 11:13,
127:20, 132:12,
134:10, 171:16
remember [10] -
20:10, 22:14, 98:14,
120:1, 120:2, 122:7,
134:18, 139:10,
140:2, 140:11
reminder [3] - 84:21,
85:9, 85:11
removed [1] - 100:16
removing [1] - 180:14
renowned [1] - 114:21
replicate [2] - 114:11,
114:12
report [23] - 12:6,
12:7, 18:5, 18:14,	required [17] - 12:13,
12:19, 29:11, 29:20,
29:21, 30:3, 31:9,
47:20, 58:5, 64:8,
71:11, 75:7, 75:8,
96:1, 96:4, 96:10,
105:18
requirement [6] -
24:13, 30:6, 30:12,
74:23, 74:24, 150:22
requirements [2] -
11:23, 64:3
requires [3] - 83:13,
111:12, 150:7
research [5] - 89:25,
93:22, 153:18,
155:19, 155:21
Research [1] - 114:7
researchers [1] -
112:15
resigned [4] - 116:8,
116:10, 117:15,
117:22
resources [2] -
165:19, 188:22
responded [1] -	16:10, 16:11, 16:12,
19:1, 21:8, 22:15,
22:18, 47:24, 51:9,
52:19, 53:11, 53:12,
53:13, 63:11, 64:8,
67:11, 71:4, 121:1
reviews [1] - 12:3
revised [1] - 21:1
revision [1] - 19:3
revisions [1] - 18:25
reward [1] - 88:13
rhetorical [1] - 134:2
rid [1] - 113:11
risk [1] - 190:7
road [1] - 183:3
rolled [1] - 86:16
rolling [4] - 77:20,
78:12, 98:18, 182:13
room [2] - 119:23,
166:22
Rosa [4] - 172:22,
172:23, 172:24,
174:4
roster [21] - 34:6,
34:8, 34:14, 34:15,
34:21, 35:5, 36:3,	160:23, 160:25,
165:17, 175:6,
175:13, 175:21,
183:5, 188:24,
189:2, 189:9, 190:7,
190:24
rule-making [1] - 4:7
rulemaking [1] - 15:7
rules [15] - 3:24, 4:5,
4:6, 9:2, 9:3, 9:10,
10:1, 10:3, 10:8,
11:14, 11:22, 29:22,
29:25, 63:22, 84:2
ruling [1] - 176:10
run [19] - 9:9, 9:18,
81:1, 81:2, 81:3,
86:22, 99:19,
112:21, 121:22,
130:5, 130:6,
130:15, 131:6,
131:21, 133:9,
154:10, 154:12,
157:25, 189:4
running [1] - 131:2
runs [1] - 158:1	10:10, 10:14, 10:15,
12:14, 12:17, 12:22,
17:23, 19:7, 19:8,
23:3, 27:14, 31:7,
31:9, 31:18, 33:14,
33:17, 36:7, 36:9,
42:10, 42:11, 43:7,
44:3, 44:4, 44:10,
45:22, 45:23, 46:4,
46:5, 46:16, 47:2,
48:13, 52:20, 56:23,
58:2, 58:11, 61:1,
61:3, 61:8, 62:11,
62:21, 63:7, 71:16,
75:8, 81:5, 87:9,
88:11, 88:12, 88:16,
89:21, 92:7, 93:11,
93:12, 93:13, 97:13,
97:22, 98:3, 98:23,
100:19, 100:20,
101:2, 101:12,
108:4, 109:20,
112:12, 113:19,
117:13, 118:10,
118:18, 120:25,
121:13, 121:15,	
			S		
				121:21, 121:22,	
20:20, 20:24, 21:6,	124:12	37:8, 37:10, 82:6,			
				122:8, 132:1,	
23:19, 23:22, 23:24,	response [1] - 6:14	99:10, 99:14, 99:22,	
sake [1] - 124:12		
				137:20, 137:22,	
56:21, 97:23, 98:15,	responses [1] - 9:8	100:18, 110:16,			
				137:25, 138:7,	
100:22, 102:11,	responsibilities [7] -	111:22, 112:23,	sample [2] - 129:7,		
				138:8, 138:14,	
102:12, 102:13,	18:2, 18:3, 58:12,	118:15, 121:10,			
			143:23	138:18, 143:22,	
103:5, 103:9, 155:6,	63:21, 63:24, 64:1,	181:21, 182:2			
			sampled [1] - 124:25	162:23, 164:9,	
179:25, 187:13,	68:11	rosters [3] - 34:11,			
			sand [1] - 68:16	166:19, 173:13,	
187:21	responsible [2] -	34:17, 112:13			
			Santa [4] - 172:22,	177:10, 177:25,	
reported [9] - 93:14,	163:9, 168:17	roughly [1] - 129:7			
			172:23, 172:24,	178:11, 179:1,	
97:20, 108:6, 111:1,	rest [3] - 80:6, 82:9,	routinely [1] - 118:2			
			174:4	180:7, 180:14,	
112:25, 124:15,	140:14	RTI [2] - 179:23,			
			sat [1] - 31:13	180:17, 180:19,	
179:9, 187:7, 187:10	resubmit [1] - 43:14	179:25			
			SAT-10 [1] - 79:2	180:22, 180:23,	
REPORTER [4] -	result [11] - 68:2,	rule [77] - 3:6, 4:7, 4:8,			
			satisfied [1] - 164:15	180:24, 181:10,	
45:25, 149:11,	80:18, 80:19, 83:3,	4:10, 5:6, 5:12, 9:14,			
			save [1] - 42:4	182:2, 189:6, 189:16	
149:16, 188:8	83:14, 125:4, 125:5,	11:6, 11:16, 11:17,			

schools [21] - 44:8,	
111:16, 116:18,	
38:24, 39:1, 44:25,	
84:13, 84:21, 91:19,	
5:24, 7:4	
44:11, 44:13, 45:21,	127:7, 131:19,	50:6, 50:9, 52:18,	91:20, 91:22, 108:1,	sheets [1] - 35:5	
87:25, 125:18,	150:3, 154:16,	56:19, 61:15, 71:1,	143:17, 159:3,	shoot [1] - 97:6	
151:1, 151:3, 180:7,	166:16, 171:19,	77:5, 82:3, 82:4,	164:16, 189:9	shooting [2] - 128:23,	
180:8, 180:10,	173:2, 176:19,	84:15, 85:18, 86:7,	separated [1] - 116:16	136:24	
180:11, 180:12,	176:20, 176:22,	86:9, 89:4, 90:4,	separately [1] - 81:3	Short [1] - 30:19	
181:8, 189:12,	177:25, 182:18,	90:19, 94:1, 96:6,	series [1] - 18:3	short [2] - 14:12,	
189:20, 189:22,	183:14, 184:12,	102:18, 102:20,	services [1] - 92:18	181:18	
189:24, 190:3, 190:7	185:15, 188:16	102:21, 102:25,	session [2] - 8:23, 9:6	shorter [1] - 151:18	
schoolteacher [1] -	scoring [1] - 20:2	103:7, 103:10,	sessions [1] - 10:25	show [12] - 5:25, 8:8,	
154:20	screen [1] - 42:4	104:7, 104:11,	Set [1] - 171:1	26:7, 37:12, 56:6,	
schoolwide [6] -	second [42] - 13:22,	104:16, 105:2,	set [45] - 12:24, 13:9,	93:4, 102:14,	
61:12, 89:10, 147:8,	14:8, 25:3, 29:6,	106:13, 106:14,	19:16, 39:11, 60:8,	111:21, 126:22,	
160:1, 170:2, 185:25	34:13, 36:9, 38:9,	106:16, 119:25,	64:24, 66:24, 72:20,	131:14, 134:6,	
score [98] - 33:5,	39:25, 40:2, 40:6,	121:1, 124:15,	81:23, 83:13, 85:6,	160:24	
34:24, 35:9, 38:17,	42:2, 42:3, 49:3,	124:17, 136:5,	100:20, 101:25,	showed [3] - 97:16,	
38:18, 41:21, 44:25,	50:14, 50:15, 75:16,	149:8, 154:13,	112:9, 112:17,	175:24, 176:1	
73:18, 74:2, 80:21,	83:4, 84:1, 84:11,	166:22, 169:3,	123:7, 124:25,	shows [1] - 54:7	
80:25, 82:24, 85:25,	84:17, 97:9, 99:15,	169:5, 175:7, 179:6,	126:5, 130:4,	sic [1] - 149:23	
86:4, 86:7, 86:9,	104:18, 109:12,	180:4, 185:9	134:21, 135:3,	side [4] - 125:5, 169:5,	
86:11, 86:18, 86:21,	112:9, 112:17,	seeing [1] - 4:19	135:7, 135:8, 136:1,	188:22, 190:12	
90:4, 90:5, 92:8,	116:9, 139:4,	seek [1] - 27:21	141:1, 146:9, 149:4,	sign [5] - 3:10, 3:21,	
92:13, 94:3, 94:18,	139:16, 139:20,	seeking [1] - 8:6	150:7, 150:8,	7:3, 33:25, 37:1	
94:20, 94:21, 95:1,	143:2, 143:3,	seem [2] - 24:1, 91:3	158:23, 159:3,	sign-in [3] - 3:10,	
95:8, 96:2, 102:17,	143:10, 145:20,	segment [1] - 172:4	159:9, 160:20,	3:21, 7:3	
103:23, 104:8,	146:6, 146:7,	selected [2] - 87:4,	161:9, 169:7,	signed [3] - 9:1, 35:4,	
104:12, 104:17,	148:23, 155:15,	87:13	170:22, 171:7,	50:20	
104:18, 105:2,
106:5, 106:8,	158:25, 177:21,
180:5, 182:6	self [4] - 23:8, 23:12,
63:12, 63:13	171:25, 172:6,
173:3, 173:4,	significant [2] - 91:14,
106:17	
106:21, 106:25,
107:12, 107:14,
120:4, 121:17,
122:9, 122:17,
123:12, 123:16,
123:20, 123:23,
125:10, 125:11,
125:14, 125:15,
125:25, 126:2,
126:10, 126:11,
126:14, 126:18,
127:5, 127:21,
134:10, 134:11,
134:13, 135:23,
140:10, 140:11,
140:18, 140:20,
140:24, 141:10,
141:11, 142:1,
142:2, 142:6, 143:8,
143:9, 143:11,
143:14, 144:1,
145:2, 154:6, 155:9,
167:6, 167:7,
169:19, 170:2,
170:8, 175:23,
175:24, 176:1, 190:5
scores [30] - 40:19,
41:3, 45:5, 76:5,
82:24, 89:23, 89:24,
94:19, 96:15, 97:8,
106:8, 106:9,	secondary [2] -
154:18, 187:6
Secondly [2] - 27:25,
108:7
section [36] - 4:15,
11:24, 13:24, 14:5,
14:8, 14:14, 14:15,
14:16, 14:18, 14:22,
15:10, 16:8, 16:13,
16:16, 16:22, 16:23,
17:24, 18:20, 19:4,
20:11, 22:11, 22:13,
47:9, 49:2, 51:9,
51:11, 57:12, 57:17,
57:21, 60:13, 61:21,
102:18, 174:9
Section [3] - 11:23,
16:9, 16:15
sections [2] - 58:16,
68:9
security [1] - 136:10
see [78] - 4:2, 5:7,
6:24, 7:17, 8:17,
9:19, 9:24, 13:23,
14:7, 14:8, 14:13,
20:23, 20:25, 21:19,
22:23, 22:24, 23:2,
25:15, 26:13, 33:22,
34:2, 34:3, 35:8,
35:15, 36:11, 36:14,
37:4, 38:16, 38:22,	self-assessment [2] -
23:8, 23:12
self-evaluation [1] -
63:12
self-monitoring [1] -
63:13
semester [11] - 34:12,
34:13, 36:8, 99:15,
109:25, 110:8,
110:9, 117:18,
117:19
Seminole [5] - 129:1,
130:5, 130:12,
130:15, 189:4
send [7] - 5:23, 5:24,
14:10, 43:22, 102:7,
112:12, 175:10
sends [1] - 52:23
sense [11] - 30:7,
53:2, 54:8, 78:24,
79:5, 88:2, 115:4,
138:24, 173:18,
173:19, 189:25
sensitive [1] - 188:20
sentence [10] - 16:2,
16:3, 49:3, 61:14,
162:16, 165:14,
169:4, 170:13,
174:6, 174:8
separate [14] - 3:6,
51:18, 52:7, 58:15,	173:15, 174:15,
177:3, 179:21
sets [2] - 168:11,
175:18
setting [3] - 9:10,
94:15, 171:2
settled [2] - 176:9,
176:11
seven [6] - 20:11,
22:11, 22:13, 30:4,
47:12, 128:8
seventh [12] - 28:9,
44:23, 45:8, 72:2,
80:20, 128:4,
135:21, 135:23,
160:9, 160:15,
160:21, 163:10
seventy [1] - 153:4
seventy-five [1] -
153:4
several [5] - 26:4,
67:7, 92:14, 183:2,
183:4
severe [5] - 91:19,
91:23, 92:2, 105:15,
153:23
SGIC [1] - 85:15
shall [1] - 16:3
shape [1] - 105:22
share [2] - 165:19
sheet [4] - 3:10, 3:21,	significantly [1] -
93:16
signing [2] - 36:16,
37:2
similar [6] - 28:12,
42:15, 61:6, 123:17,
169:15, 178:22
simple [3] - 113:10,
113:25, 160:12
simply [1] - 152:24
simulation [1] - 129:1
simulations [3] - 9:13,
9:18, 154:10
single [9] - 22:8, 29:3,
79:4, 98:19, 118:2,
122:22, 128:22,
161:13, 163:3
sit [2] - 88:18, 137:19
site [3] - 20:21, 21:8,
23:6
sitting [1] - 88:14
situation [10] - 25:10,
32:19, 64:24, 66:6,
75:11, 78:8, 80:8,
88:22, 118:21,
183:23
situations [2] - 118:1,
146:8
six [11] - 16:23, 18:20,
19:4, 41:23, 47:9,
54:12, 134:7, 146:2,	

177:20, 189:22,	
sounded [1] - 178:22	
171:7, 171:11,	
169:24, 170:1,	
72:19, 74:9, 74:25,	
190:7	sounds [5] - 89:9,	171:12, 172:1,	170:7, 170:18,	79:24, 84:13, 85:17,	
sixth [4] - 72:1, 73:16,	118:20, 128:25,	172:2, 172:11	170:19, 170:20,	105:25, 114:15,	
75:4, 93:1	178:24, 181:14	standardized [1] -	170:21, 171:1,	123:23, 128:9,	
size [4] - 119:15,	source [1] - 68:20	105:16	171:4, 172:4, 172:5,	140:19, 140:21,	
143:23, 154:23,	sources [11] - 62:8,	standards [48] - 9:10,	179:10, 179:13,	141:3, 141:19,	
177:9	67:4, 67:5, 67:13,	9:24, 20:25, 27:20,	180:9, 182:6	141:23, 142:5,	
skills [1] - 102:2	67:22, 67:23, 68:1,	54:17, 59:2, 61:25,	state [27] - 4:7, 10:4,	148:4, 159:17,	
SLD [8] - 101:15,	68:2, 68:4	81:24, 82:10, 82:14,	12:20, 21:11, 45:8,	159:19, 169:10,	
107:21, 186:13,	spaced [1] - 15:22	83:5, 83:7, 84:2,	59:20, 59:24, 83:2,	169:22, 181:24	
186:22, 186:24,	speaker [3] - 3:11,	84:14, 85:2, 94:16,	89:6, 91:9, 91:14,	stop [1] - 96:9	
187:10, 187:11	4:24, 6:18	119:8, 122:1, 122:4,	94:3, 94:11, 95:11,	store [2] - 134:15,	
slice [1] - 130:9	special [7] - 18:17,	123:8, 126:23,	102:5, 107:4,	134:24	
slide [2] - 48:20, 86:23	63:5, 63:6, 66:18,	130:4, 130:11,	113:23, 115:8,	stored [1] - 36:12	
slides [3] - 64:13,	119:5, 174:12	131:15, 132:19,	118:3, 122:12,	straight [2] - 103:9,	
64:14, 175:4	specific [8] - 4:7,	139:11, 139:18,	123:15, 130:4,	103:10	
small [1] - 39:3	30:17, 31:2, 107:7,	142:17, 146:4,	130:9, 140:4,	stronger [1] - 70:14	
smaller [1] - 158:7	153:19, 186:20,	146:15, 147:15,	161:17, 168:4,	strongest [2] - 90:2,	
snap [1] - 126:17	186:25, 187:18	149:4, 149:10,	169:12	101:8	
so.. [1] - 74:21	specifically [3] -	149:17, 154:15,	State's [1] - 135:19	struggle [1] - 164:22	
social [6] - 28:9,	26:22, 34:22, 53:23	158:24, 159:1,	State-adopted [1] -	stuck [1] - 155:11	
160:9, 160:11,	specify [1] - 186:18	159:4, 159:9,	164:20	student [88] - 9:11,	
160:15, 160:21,	spelling [1] - 4:25	160:20, 168:22,	statement [1] - 153:11	27:7, 31:18, 31:19,	
163:11	spend [2] - 58:20,	169:1, 169:7, 172:6,	states [7] - 113:17,	31:22, 31:25, 32:3,	
socio [3] - 95:14,	166:14	172:7, 173:15,	124:9, 124:10,	32:12, 33:4, 34:19,	
95:22, 178:7	spent [1] - 76:10	174:16, 185:23	125:1, 127:13,	34:24, 37:18, 37:19,	
socio-economic [3] -	split [7] - 59:19, 73:14,	Standards [1] - 161:25	158:13	37:20, 38:4, 44:22,	
95:14, 95:22, 178:7	116:12, 116:14,	start [13] - 21:1, 34:16,	statewide [13] - 9:13,	54:15, 54:22, 55:15,	
software [1] - 27:21	117:9, 117:12,	65:20, 66:2, 66:5,	9:20, 38:22, 38:25,	55:16, 57:2, 57:24,	
someone [5] - 41:14,
42:15, 66:20,
115:23, 156:25	177:17
spot [2] - 29:24,
107:15	69:9, 75:12, 75:13,
85:25, 95:5, 128:5,
140:6, 180:3	59:14, 60:19, 60:22,
60:23, 93:14, 113:7,
147:22, 166:10,	59:13, 59:21, 60:15,
64:17, 70:20, 71:22,
73:25, 74:4, 74:7,	
someplace [2] -
46:23, 50:8
sometime [1] - 31:23
sometimes [4] -
93:15, 165:4, 181:4,
190:17
Sometimes [1] - 181:4
somewhere [9] - 14:7,
31:2, 36:12, 37:14,
114:11, 125:6,
156:2, 178:10, 180:7
soon [4] - 38:15, 71:8,
84:25, 183:5
sooner [1] - 97:24
Sorry [3] - 40:4,
109:14, 151:16
sorry [12] - 15:25,
40:5, 46:2, 65:10,
75:22, 108:19,
109:6, 109:13,
109:16, 145:20,
146:1, 178:15
sort [6] - 24:12, 27:9,
67:9, 175:14,
176:10, 189:10
sorts [1] - 15:22
sound [2] - 126:15,
126:16	spur [1] - 5:12
staff [1] - 137:11
stand [2] - 3:16, 5:18
standard [62] - 28:15,
83:13, 83:14, 124:3,
124:6, 125:14,
125:23, 125:24,
126:19, 128:20,
129:12, 134:12,
134:17, 134:18,
134:22, 140:9,
141:16, 141:19,
141:20, 141:22,
142:4, 142:5,
143:19, 144:1,
144:5, 145:8, 145:9,
149:21, 151:20,
151:21, 151:24,
152:10, 152:11,
152:12, 152:25,
153:14, 153:16,
153:20, 154:17,
155:2, 155:8, 157:5,
157:17, 157:23,
157:24, 159:23,
169:13, 170:17,
170:18, 170:19,
170:22, 171:5,	started [13] - 49:24,
69:3, 69:4, 69:21,
74:25, 95:8, 106:22,
107:15, 117:17,
131:3, 136:21,
179:12, 179:21
starting [5] - 18:5,
18:13, 68:13, 72:6,
135:16
Starts [1] - 14:25
starts [5] - 13:21,
14:23, 94:24, 94:25,
182:23
State [51] - 7:12, 9:3,
10:8, 10:12, 17:22,
18:6, 18:14, 25:17,
28:18, 28:25, 63:22,
63:23, 79:10, 84:22,
85:6, 85:7, 92:21,
94:15, 97:20, 100:2,
106:1, 111:13,
111:18, 140:14,
140:21, 141:4,
141:13, 142:9,
143:16, 145:7,
146:10, 156:14,
164:20, 167:13,
167:15, 169:16,	166:13
statistical [5] - 78:15,
112:1, 129:16,
133:14, 133:21
statistically [2] -
91:13, 93:16
statistician [2] - 39:5,
114:21
statistics [4] - 126:15,
126:16, 133:17,
144:9
status [12] - 16:21,
52:15, 86:22, 91:4,
92:17, 93:8, 95:14,
95:22, 96:4, 96:5,
102:21, 178:8
statute [2] - 21:12,
67:21
Statutes [1] - 11:23
stay [4] - 11:2, 113:20,
119:10, 188:5
stayed [1] - 179:1
stays [2] - 10:7, 128:2
step [3] - 138:18,
164:4, 173:11
stick [2] - 28:24, 65:7
still [27] - 36:12, 43:7,
64:11, 70:1, 72:18,	74:17, 79:7, 80:12,
80:14, 82:15, 83:7,
84:8, 86:1, 86:3,
86:5, 86:8, 86:11,
86:13, 86:20, 87:1,
87:16, 87:20, 89:19,
89:23, 89:24, 90:4,
90:7, 90:13, 90:21,
93:23, 94:24, 94:25,
95:15, 95:21, 101:9,
101:10, 103:6,
104:14, 104:16,
104:22, 105:4,
105:10, 106:18,
106:21, 107:8,
109:22, 112:25,
114:8, 118:7,
120:16, 122:8,
122:22, 147:22,
148:7, 148:8,
160:14, 164:1,
169:14, 179:4,
182:18, 185:4
Student [4] - 12:12,
69:5, 71:2, 85:13
Students [1] - 91:4
students [123] - 27:11,
35:6, 35:7, 35:8,	

37:11, 38:3, 38:16,
41:21, 58:9, 59:7,	
stupid [1] - 185:13
subject [18] - 7:16,	
43:11, 43:13, 43:21,
116:3, 132:14,	
T	
179:18, 180:21,
181:1	
			
talks [4] - 18:1, 22:13,		
59:17, 60:18, 60:21,	83:2, 90:24, 102:14,	146:5, 146:11,		teachers [68] - 22:7,	
60:23, 60:25, 66:12,	102:19, 102:25,	154:12, 158:20		38:22, 39:14, 39:22,	
67:16, 67:18, 71:6,	114:24, 122:12,	suggests [1] - 19:22	40:10, 97:12	42:9, 46:7, 46:13,	
73:15, 73:22, 73:24,	137:17, 140:5,	sum [1] - 137:10	target [2] - 96:17, 97:5	46:16, 46:21, 47:21,	
74:15, 74:22, 75:1,	141:13, 148:11,	summated [4] - 27:23,	targets [2] - 60:24	50:4, 54:12, 58:23,	
75:3, 75:6, 75:15,	161:7, 169:15,	31:16, 31:21, 56:1	taught [8] - 38:25,	60:10, 60:15, 62:17,	
75:18, 76:6, 76:8,	172:14, 183:1,	summer [2] - 97:23,	73:9, 125:10,	63:7, 75:25, 80:4,	
76:15, 79:6, 79:22,	185:17	121:3	138:14, 160:11,	89:6, 99:18, 103:22,	
79:24, 81:9, 82:21,	subject-relevant [1] -	superintendent [1] -	170:3, 173:10, 186:1	116:13, 116:14,	
83:12, 86:16, 89:9,	90:24	61:8	TBI [3] - 186:16,	116:21, 117:5,	
89:11, 89:13, 90:17,	subjected [1] - 182:18	supervisor [1] - 47:21	186:18, 187:13	117:6, 117:7,	
91:9, 91:16, 91:19,	subjects [2] - 38:25,	supply [1] - 175:16	teach [19] - 46:7,	117:10, 117:12,	
91:23, 91:24, 92:3,	94:2	support [2] - 59:5,	46:14, 46:21, 71:25,	117:14, 118:5,	
92:5, 92:15, 94:10,	submission [2] - 14:9,	79:1	76:1, 81:16, 85:19,	118:13, 118:24,	
95:10, 98:7, 98:8,	49:3	suppose [1] - 100:10	87:24, 91:19, 138:2,	120:18, 121:3,	
99:18, 99:22,	submit [7] - 10:21,	supposed [13] - 4:4,	147:8, 147:23,	125:19, 125:20,	
102:23, 104:19,	12:2, 14:21, 15:13,	4:10, 11:20, 22:2,	160:5, 161:7,	128:11, 129:3,	
106:9, 106:17,	23:21, 23:23, 64:8	30:8, 57:1, 57:4,	161:23, 178:18,	129:6, 129:7, 137:7,	
107:3, 107:22,	submits [2] - 23:19,	62:12, 87:16,	179:2, 181:11	138:17, 139:6,	
108:12, 108:17,	53:9	107:10, 151:19,	teacher [84] - 21:12,	148:2, 152:22,	
108:21, 108:25,	subs [2] - 116:10,	160:3, 167:13	21:15, 24:3, 24:16,	158:4, 158:5, 158:8,	
109:3, 110:7,	117:22	survey [13] - 34:11,	24:23, 25:4, 37:17,	160:2, 162:1,	
111:22, 112:2,	Subsection [2] -	34:13, 34:14, 34:15,	38:15, 41:2, 44:21,	163:23, 164:4,	
117:8, 117:12,	22:12, 48:12	97:20, 97:21, 99:19,	54:24, 56:9, 56:10,	168:3, 169:23,	
118:6, 118:14,	subsection [2] -	109:23, 110:5,	56:15, 56:24, 57:4,	170:1, 170:8,	
119:19, 121:16,	22:17, 23:4	110:11, 110:22,	58:10, 60:21, 61:1,	171:21, 172:5,	
122:11, 122:15,
122:20, 122:24,
123:11, 123:19,
125:17, 125:18,
125:22, 126:1,
127:1, 127:6,
134:14, 135:8,
138:5, 139:13,
140:3, 140:4,
140:13, 140:18,
141:3, 141:12,
143:1, 146:16,
147:13, 149:22,
154:18, 155:14,
158:3, 160:15,
164:8, 165:8, 165:9,
170:6, 170:16,
171:14, 171:22,
171:24, 175:23,
176:1, 177:12,
177:17, 177:20,
178:23, 181:17
students' [1] - 116:18
studies [7] - 28:9,
160:9, 160:11,
160:15, 160:21,
163:11
stuff [9] - 52:22,
87:11, 113:6, 133:2,
160:1, 164:14,
174:4, 182:16,
186:10
stunted [1] - 178:9	subsections [1] -
22:12
substantial [8] -
18:25, 19:3, 19:11,
20:5, 20:6, 26:18,
47:10, 48:6
substantive [4] -
18:25, 19:3, 19:25,
48:7
subtract [1] - 113:12
Success [2] - 12:12,
69:5
sucked [2] - 65:9,
65:11
sudden [3] - 57:6,
90:8, 175:25
suddenly [1] - 135:21
suggest [4] - 24:11,
70:14, 157:3, 174:14
suggested [1] - 143:2
suggesting [2] - 5:11,
147:19
suggestion [15] -
21:22, 24:18, 27:2,
36:4, 43:6, 43:8,
130:12, 142:19,
142:24, 147:24,
154:4, 154:8, 154:9,
161:19, 162:11
suggestions [15] -
5:14, 7:19, 9:15,
9:19, 30:16, 30:18,	121:12
surveyed [1] - 125:3
switched [1] - 42:13
switching [1] - 181:15
system [52] - 12:17,
12:18, 13:3, 13:6,
14:10, 14:18, 16:11,
16:25, 17:1, 18:23,
22:19, 26:19, 46:10,
46:25, 48:4, 49:8,
49:20, 50:1, 50:3,
50:8, 50:12, 50:19,
51:21, 51:24, 52:16,
57:23, 59:9, 60:10,
60:17, 61:20, 62:3,
62:6, 62:9, 62:10,
62:24, 63:10, 63:19,
64:4, 64:25, 67:20,
68:7, 68:10, 69:7,
92:10, 93:7, 100:7,
102:10, 110:4,
118:22, 126:14,
133:19, 186:18
Systems [1] - 11:10
systems [12] - 11:25,
12:4, 12:10, 12:14,
14:24, 15:1, 15:14,
20:14, 20:16, 21:6,
26:4, 64:9	66:7, 66:14, 71:19,	176:6, 178:23,	
			72:15, 80:6, 81:6,	180:22, 181:11,	
			81:15, 86:17, 86:25,	189:13, 189:16,	
			87:1, 87:12, 89:12,	189:18	
			91:15, 96:25, 97:2,	teachers' [2] - 60:18,	
			100:12, 100:19,	60:22	
			100:22, 100:23,	teaches [4] - 72:1,	
			104:9, 108:23,	79:21, 138:5, 162:6	
			110:7, 112:23,	teaching [21] - 34:24,	
			116:8, 116:9,	47:1, 56:25, 63:4,	
			117:22, 118:7,	66:9, 66:18, 71:20,	
			118:17, 119:19,	76:10, 116:22,	
			122:15, 122:20,	138:22, 160:2,	
			122:25, 123:20,	160:4, 163:9,	
			133:22, 137:21,	166:15, 168:17,	
			137:23, 137:25,	168:19, 183:4,	
			138:2, 138:13,	185:11, 185:12,	
			141:2, 143:7,	185:16, 185:21	
			143:22, 153:25,	team [6] - 61:9, 61:10,	
			154:18, 155:13,	149:1, 162:17,	
			158:16, 159:18,	165:14, 190:9	
			159:21, 159:24,	technical [12] - 15:6,	
			161:22, 164:24,	28:1, 28:21, 98:15,	
			168:1, 168:8, 168:9,	102:11, 102:12,	
			176:14, 176:15,	102:13, 103:5,	
			177:15, 177:19,	103:9, 112:14,	
			177:23, 181:1,	114:9, 155:6	
			182:17, 183:3,	technicalities [3] -	
			184:1, 185:11, 189:7	188:16, 188:24,	
			teacher's [10] - 72:7,	190:17	
			117:7, 122:9,	technically [1] -	
			123:19, 126:13,	106:16	
			141:12, 179:16,		

technology [4] -	
177:16, 177:19,	
45:10, 45:14, 112:24	
112:9, 128:21	
167:6, 169:4,	
25:18, 39:7, 161:25,	184:1	tools [1] - 38:21	trying [18] - 28:21,	171:12, 174:10,	
181:5	thousands [1] - 18:1	Top [4] - 15:13, 49:24,	29:24, 39:20, 44:20,	174:14, 177:20,	
template [1] - 49:25	three [79] - 3:4, 3:15,	69:3, 69:4	114:13, 121:9,	182:7, 182:9,	
temporary [2] - 147:6,	4:2, 6:6, 7:4, 14:14,	top [11] - 16:1, 16:5,	127:16, 128:15,	182:10, 188:2, 189:9	
149:1	14:15, 14:16, 14:22,	20:1, 31:5, 98:14,	133:18, 136:1,	two-and-a-half [3] -	
ten [5] - 25:14, 94:8,	15:10, 21:12, 21:23,	127:11, 151:10,	136:22, 144:7,	129:16, 129:18,	
102:1, 120:3, 128:4	31:6, 34:13, 34:15,	151:19, 154:25,	158:23, 159:1,	129:25	
tend [1] - 104:12	35:7, 36:4, 39:23,	155:5, 156:7	160:24, 161:15,	two/three [2] - 109:21,	
tended [1] - 104:17	45:14, 55:11, 57:12,	total [1] - 36:24	174:18, 188:19	110:5	
tens [1] - 98:12	57:17, 57:21, 58:7,	totally [1] - 141:21	turn [2] - 132:15,	type [3] - 26:5, 26:17,	
tenth [4] - 75:1, 75:17,	59:18, 60:17, 61:21,	tracked [1] - 73:24	173:7	43:22	
81:6, 138:15	64:17, 65:1, 65:6,	traditionally [1] -	turned [1] - 126:6	typical [4] - 24:3,	
term [1] - 117:22	65:24, 66:8, 66:15,	173:13	turning [1] - 132:18	123:3, 140:2, 154:17	
terminated [2] - 190:8,	68:17, 69:1, 76:21,	tragic [1] - 90:9	turns [2] - 154:13,	typically [3] - 74:6,	
190:9	89:13, 89:14, 94:18,	train [1] - 70:11	171:1	98:10, 122:11	
terms [2] - 97:6, 97:7	94:22, 95:4, 99:19,	trained [3] - 30:6,	tutoring [1] - 108:20	Typically [1] - 55:4	
test [30] - 26:6, 26:18,	106:2, 106:22,	30:12, 69:22	twelfth [7] - 73:16,		
27:16, 27:20, 28:11,
28:13, 29:3, 34:3,	109:23, 111:4,
112:8, 113:21,	training [10] - 29:7,
29:11, 29:15, 29:16,	74:1, 74:4, 74:16,
75:2, 75:18, 81:7	U	
60:7, 70:6, 75:19,
77:11, 77:12, 81:19,
103:13, 107:12,
116:18, 121:17,
136:10, 148:6,
148:9, 158:10,
158:12, 159:5,
167:20, 167:21,
169:11, 170:24,
171:6, 190:1
testing [3] - 33:23,
77:21, 98:3
tests [5] - 105:16,
136:9, 158:13,
159:13, 162:2
text [6] - 4:13, 4:20,
7:11, 9:8, 11:17,
13:21
texts [1] - 7:10
THE [4] - 45:25,
149:11, 149:16,
188:8
themselves [2] -
108:22, 165:20
theoretically [1] -
101:9
therefore [1] - 65:12
Therefore [1] - 96:24
They've [1] - 165:14
they've [6] - 56:6,
101:13, 116:10,
184:4, 184:8
thinking [2] - 5:13,
94:6
thinks [1] - 24:19
third [15] - 8:13, 14:14,
61:7, 68:19, 69:10,
96:21, 96:24,
109:13, 152:18,
165:5, 165:7, 174:6,	115:23, 116:14,
121:12, 124:18,
125:4, 134:19,
136:8, 147:2, 147:5,
147:10, 154:20,
157:2, 157:4, 157:5,
157:16, 158:4,
167:6, 177:6, 182:7,
182:9, 182:10,
183:8, 183:20,
184:13, 185:14,
185:15, 185:18,
185:25, 189:23,
189:24
Three [1] - 183:7
threshold [4] - 182:15,
183:1, 183:6, 185:2
throughout [5] -
67:19, 93:13, 115:8,
116:11, 117:13
tied [7] - 34:2, 35:3,
35:9, 35:16, 57:23,
58:24, 59:1
timeline [9] - 8:8,
10:7, 10:17, 16:12,
69:9, 132:12, 161:5,
176:13, 176:16
tip [1] - 144:19
today [6] - 3:3, 3:24,
6:4, 6:13, 8:15,
46:13
today's [1] - 10:18
together [4] - 58:14,
86:16, 122:24, 142:2
took [9] - 3:2, 15:9,
80:20, 92:11, 116:7,
116:11, 116:15,
173:11, 173:14
tool [8] - 28:22, 34:7,
34:10, 38:15, 38:23,	29:17, 62:3, 69:17,
69:24, 69:25, 70:15
transcript [2] - 5:2,
136:5
transcripts [2] - 5:3,
8:19
transitions [1] - 93:10
transparent [1] -
114:14
traumatic [4] - 105:23,
106:11, 106:14,
106:19
treat [1] - 44:4
trenches [1] - 138:22
tried [7] - 9:23, 13:12,
52:7, 70:10, 84:12,
84:15, 147:17
trigger [13] - 144:15,
144:16, 145:12,
145:15, 145:20,
147:11, 148:14,
148:20, 148:24,
157:1, 182:15,
185:17, 190:6
triggered [1] - 185:3
triggers [1] - 146:14
troubling [1] - 189:5
true [4] - 99:17, 139:5,
158:12, 185:6
true-up [1] - 99:17
truly [1] - 138:4
trust [5] - 77:18,
111:13, 112:5,
115:6, 115:8
Trust [1] - 84:19
trusting [1] - 111:13
truth [2] - 124:23,
159:24
try [7] - 3:18, 13:17,
51:17, 79:3, 79:14,	twice [3] - 24:4, 24:17,
54:20
Two [1] - 89:22
two [113] - 3:6, 3:24,
5:4, 5:5, 7:10, 8:11,
11:12, 14:8, 20:1,
22:11, 24:23, 25:2,
26:10, 28:13, 34:12,
34:14, 40:22, 41:1,
41:7, 41:10, 41:13,
41:23, 44:22, 48:12,
49:2, 52:22, 58:14,
62:1, 64:13, 64:14,
65:12, 65:14, 65:15,
66:14, 67:12, 68:2,
68:3, 68:8, 68:9,
69:1, 69:13, 69:14,
70:8, 73:2, 76:23,
76:25, 79:21, 80:2,
80:6, 80:17, 82:15,
83:5, 83:10, 83:24,
84:13, 84:21, 89:24,
91:2, 92:18, 94:8,
94:18, 94:22, 95:2,
95:17, 96:12, 97:3,
100:14, 107:17,
109:23, 111:4,
112:22, 115:22,
116:15, 129:12,
129:16, 129:18,
129:25, 134:22,
136:19, 139:10,
140:6, 141:5, 141:6,
141:18, 141:19,
141:20, 141:22,
143:4, 144:4, 145:9,
147:4, 151:20,
151:24, 152:25,
153:14, 153:21,
157:2, 157:5, 164:4,	
ultimate [1] - 17:13
unable [1] - 17:14
unattainable [1] -
19:20
unclear [1] - 120:9
Under [5] - 21:25,
25:23, 40:10, 47:9,
49:2
under [18] - 39:11,
46:7, 46:9, 69:3,
84:16, 103:6,
116:25, 144:17,
147:17, 166:10,
169:3, 175:4,
186:16, 186:24,
186:25, 187:12,
187:15
underneath [2] -
22:13, 174:7
understood [1] -
178:21
unfair [4] - 39:8,
139:1, 162:5, 190:11
unique [1] - 39:11
unknown [1] - 37:3
unknowns [1] - 34:1
unless [7] - 37:12,
53:14, 106:23,
110:15, 114:20,
134:1, 182:9
unsatisfactory [27] -
39:17, 76:18, 76:22,
83:9, 83:15, 128:19,
129:4, 129:19,
137:9, 137:24,
140:23, 142:14,
144:16, 146:7,
146:10, 153:7,
153:16, 156:10,	

156:16, 156:25,	
86:18, 90:11, 94:20,	
versions [1] - 13:15	
163:7, 166:14,	
40:14, 40:22, 41:8,	
157:4, 158:7,	94:21, 100:16,	versus [3] - 119:21,	175:19, 177:7,	41:17, 45:11, 45:12,	
176:15, 185:3,	103:4, 109:19,	154:19, 172:19	178:21, 182:12	48:2, 53:25, 54:20,	
189:18, 189:23,	109:22, 114:5,	vetted [1] - 27:18	window [4] - 33:6,	55:4, 55:5, 55:24,	
190:10	122:9, 123:10,	video [4] - 25:12,	69:21, 149:25,	56:1, 56:2, 56:12,	
unusual [1] - 120:17	123:16, 125:9,	25:13, 25:14, 25:15	175:10	56:18, 57:7, 62:1,	
unwilling [1] - 17:15	125:11, 126:9,	Virtual [9] - 45:18,	wisely [1] - 144:9	65:5, 65:10, 66:2,	
up [63] - 3:21, 6:1,	126:18, 127:3,	45:19, 45:20, 46:3,	wish [1] - 57:7	66:4, 66:7, 66:8,	
13:9, 13:17, 18:16,	127:20, 137:6,	46:9, 46:15, 46:22,	withholding [1] - 18:8	66:10, 66:16, 66:21,	
24:6, 36:24, 43:22,	140:10, 141:10,	46:24, 47:7	wondering [1] - 45:2	67:19, 69:19, 69:23,	
54:7, 56:23, 58:9,	142:1, 143:11,	vital [1] - 10:1	word [7] - 7:18, 46:8,	70:2, 71:9, 71:10,	
63:20, 64:24, 66:14,	169:18, 170:20,	voters [4] - 124:10,	64:23, 100:2, 115:6,	72:14, 76:4, 76:10,	
69:20, 73:14, 78:15,	171:17, 172:9	124:12, 124:20,	134:25, 145:21	76:21, 77:2, 77:9,	
81:22, 86:16, 92:22,	VAM [46] - 34:25, 35:9,	124:25	worded [1] - 7:17	79:22, 80:20, 85:25,	
93:4, 95:5, 95:6,
95:7, 97:17, 97:25,	37:12, 40:18, 41:20,
42:2, 44:24, 48:24,		words [4] - 7:24,	86:9, 87:6, 89:15,	
		W	26:22, 55:15, 65:1	91:2, 91:3, 92:10,	
					
98:15, 99:17,	55:17, 55:18, 55:19,		workers [1] - 101:11	93:13, 96:21, 97:5,	
		
waited [1] - 188:14			
100:11, 100:16,	59:10, 73:10, 76:18,		works [4] - 93:22,	97:13, 97:22, 98:3,	
100:20, 102:16,	77:8, 77:24, 78:11,		100:24, 164:18,	99:11, 99:13,	
102:23, 103:16,	78:16, 78:20, 78:22,	waiting [2] - 56:18,	182:4	103:13, 103:14,	
105:14, 111:19,	79:10, 79:25, 80:10,	186:8	workshop [4] - 3:3,	103:17, 103:19,	
111:25, 112:15,	80:13, 80:25, 81:8,	walk [1] - 3:25	31:1, 190:24, 190:25	103:23, 103:25,	
113:20, 113:21,	81:13, 81:21, 82:22,	wants [3] - 163:20,	workshops [8] - 5:6,
8:11, 8:12, 8:13,
8:17, 8:18, 130:3,
131:12
world [1] - 114:21
world-renowned [1] -
114:21
worn [1] - 188:20
worried [2] - 82:2,
89:17
worry [5] - 113:25,
143:12, 148:20,
148:23, 149:2
worse [1] - 185:10
worth [6] - 35:7,
59:18, 65:24,
113:24, 183:9,
183:20
Wow [1] - 70:3
wow [1] - 70:4
wrestling [1] - 166:7
write [5] - 5:23, 6:18,
6:19, 53:1, 188:3	104:11, 104:17,
112:17, 113:12,
113:18, 113:19,
116:11, 117:13,
118:8, 121:13,
121:21, 125:12,
125:17, 127:5,
127:9, 127:15,
127:25, 128:5,
128:7, 128:22,
129:3, 131:4, 131:5,
132:1, 132:18,
135:20, 135:23,
135:24, 137:23,
141:11, 143:11,
147:4, 147:9,
148:23, 150:13,
161:18, 170:14,
173:6, 176:12,
177:7, 177:16,
181:16, 182:23,
184:15, 184:16,
184:20, 190:10
year's [16] - 10:15,	
116:12, 116:14,	99:19, 102:11,	188:2, 188:6			
117:9, 117:12,	110:15, 111:5,	watch [2] - 25:15,			
127:4, 129:24,	116:17, 117:23,	52:12			
133:1, 135:24,	117:25, 145:2,	watching [3] - 15:3,			
138:2, 138:23,	162:7, 165:6, 170:2,	15:18, 25:14			
142:2, 157:8,	170:7, 175:14,	ways [7] - 25:7, 25:9,			
170:14, 173:22,	177:7, 184:12,	30:15, 79:9, 85:4,			
175:10, 175:24,	185:15, 188:16	165:19, 179:23			
176:1, 176:21,	variability [1] - 157:10	web [10] - 5:25, 29:25,			
179:19, 187:2,	variable [4] - 38:5,	34:10, 38:15, 45:10,			
188:9, 190:16	93:9, 98:18, 119:18	45:14, 103:5, 105:8,			
Up [1] - 96:12	Variables [1] - 120:11	112:24, 162:14			
uses [7] - 29:3, 49:6,	variables [4] - 37:25,	webinar [3] - 5:4,			
84:13, 94:23, 115:6,	42:5, 115:4, 118:23	8:11, 133:4			
123:23, 142:25	variants [1] - 90:3	website [5] - 6:24, 7:2,			
	vary [2] - 41:11,	85:18, 102:7, 169:23			
V					
	102:19	weeded [1] - 87:11			
	vendor [1] - 114:5	weeds [1] - 174:4			

valid [1] - 161:21					
	verbally [2] - 108:19,	week [6] - 8:12, 8:14,			
	109:5	33:3, 99:1, 100:14,			
value [32] - 80:18,					
	verbiage [1] - 175:14	117:15			
82:22, 82:24, 86:18,					
	verification [15] -	Weeklies [1] - 13:15			
90:11, 94:20, 94:21,					
	34:8, 34:11, 34:14,	weeks [4] - 5:5, 34:15,			
100:16, 103:4,					
	34:16, 34:21, 36:3,	41:23, 177:20	Y	36:11, 40:25, 82:25,	
109:19, 109:22,					
	37:8, 82:6, 99:10,	weighting [1] - 20:2		83:1, 86:9, 86:11,	
110:10, 114:5,	99:14, 100:19,	weights [2] - 94:7,	
y'all [1] - 146:3	103:16, 127:21,	
					
122:7, 122:9,					
	110:17, 121:10,	101:25		128:3, 128:21,	
123:10, 123:16,					
	181:22, 182:2	whack [1] - 57:9	year [143] - 10:10,	150:14, 169:20,	
125:9, 125:11,					
	verifications [1] - 34:7	whereas [1] - 183:3	10:14, 10:16, 15:3,	170:9, 170:11,	
126:9, 126:18,					
	verified [5] - 35:6,	white [1] - 179:7	20:22, 20:24, 24:5,	170:15	
127:3, 127:20,					
	36:6, 37:9, 37:13,	whole [22] - 5:2, 10:3,	24:11, 24:17, 24:24,	years [60] - 20:22,	
137:6, 140:10,					
	99:20	39:10, 51:24, 53:12,	25:2, 31:8, 31:9,	21:9, 21:12, 21:14,	
141:10, 142:1,					
	verify [5] - 36:8, 37:15,	53:13, 53:14, 91:14,	31:16, 32:6, 32:20,	21:21, 21:23, 36:4,	
143:11, 169:18,				44:3, 44:5, 44:16,	
	37:17, 44:24, 112:13	107:13, 111:5,	33:14, 33:17, 35:5,		
170:20, 171:17,				44:22, 45:14, 52:23,	
	verifying [1] - 121:15	116:13, 120:15,	35:21, 35:22, 35:23,		
172:9				64:18, 65:2, 65:6,	
	version [2] - 13:13,	122:11, 130:9,	36:6, 36:7, 36:8,		
value-added [30] -				65:9, 65:14, 65:15,	
	15:19	146:21, 148:5,	36:9, 36:14, 40:12,		
80:18, 82:22, 82:24,				65:23, 66:8, 66:15,	

70:9, 74:2, 75:7,
76:21, 76:23, 76:25,
89:13, 89:14, 89:22,
89:24, 90:8, 92:19,
97:3, 106:25,
136:19, 147:2,
147:4, 147:5, 147:6,
147:10, 154:20,
157:2, 158:4,
161:20, 177:6,
177:24, 183:2,
183:5, 183:7,
183:20, 184:13,
185:14, 185:15,
185:18, 185:25,
190:3
years' [6] - 35:7,
59:18, 65:24, 66:15,
96:12, 183:8
yellow [4] - 144:19,
151:13, 151:17,
151:18
yesterday [4] - 71:3,
71:13, 116:6, 130:18
younger [1] - 67:16	
Z	

zero [25] - 122:9,
122:14, 123:2,
123:3, 123:4, 140:2,
140:12, 140:24,
141:4, 141:10,
141:11, 141:15,
142:1, 142:3, 142:5,
143:9, 144:21,
144:23, 144:24,
144:25, 145:1,
145:10, 154:6, 154:7