A call for transparency of judging in FLL

[Sarge]

Originally Posted by Dean Hystad

My lads were interviewed for the Champion's award at the Minnesota State tournament this year. This being our first year together I was filled with dread that they might win............ In a year or two they'll have earned a Champions award.

Great discussion! I have no ax to grind here. I was just pleasantly surprised by the statement above.

Dean, it sounds like your team of 4th graders made it to the Champion's final AND you were concerned they might win. I didn't even know 4th and 5th grade teams were seriously considered for the Championship. (Regardless of merit.) Nice job!

[timdavid]

Originally Posted by FLLmom1

...But, then again, what could the judges comment about, if the team did not demonstrate excellence in the areas of Robot Design and Project?

Originally Posted by Jkyaths

...Kids of several teams are baffled with the selection of both the champions at the California Capital Championship. The first champion as stated earlier were 3 girls 10 or younger..

I can certainly understand the desire for more transparency in the awards process - that was the initial focus of this thread. What I don’t like is insinuating that the named champion is undeserving of the award. I haven’t read any conclusive evidence supporting that assertion.

• According to official FLL guidelines, the Champion’s Award criteria requires that the team performs well in all three judged areas (Project, Robot Design and Core Values) and receives a Robot Performance score in the top 40% of official Robot rounds among other requirements. If you don’t know how well the Champion team was evaluated in those three areas, how can you claim they are an undeserving champion?
• As Dean noted, a short announcement for the Champions award is common. At the end of a long day, everyone is tired, and people are ready to go home.
• Cute, not extremely informative remarks about why a team won a particular award are also very common. Give the judges a break – they are trying to be entertaining at a time when their brains are totally fried.
• Suggesting that a team is not worthy of an award because it consists of 10 year old 5th grade girls is insulting. I’ve coached teams of 4th and 5th graders that were comprised of 9 and 10 year olds, and they were capable of competing with some of the best teams in the country.

I can understand feeling disappointed that your team didn’t win a particular award. I can understand wanting more information and transparency in the judging process. However, stating that the overall champion is not deserving of their award is not gracious and not professional.

While this forum is a great place to share information and discuss FLL, the people running FLL at both national and regional levels don’t pay much attention to discussions here. If you are really unhappy with the judging process, contact your local FLL partner and get involved. Talk to them and have a discussion. It looks like Northern California holds 4 championship tournaments in a month. I suspect they would love to have some new volunteers.

[Dean Hystad]

Originally Posted by Sarge

Great discussion! I have no ax to grind here. I was just pleasantly surprised by the statement above.

Dean, it sounds like your team of 4th graders made it to the Champion's final AND you were concerned they might win. I didn't even know 4th and 5th grade teams were seriously considered for the Championship. (Regardless of merit.) Nice job!

All congratulations belong to the lads, but you overstate their accomplishment. Though this is our first year together, all the boys are 6th graders and one has some prior FLL experience. In Minnesota we break teams into two groups based on the age of the oldest member. The lads are one of the older teams in the younger age group.

I've been judging FLL for a very long time and have seen a lot of Champion's Award teams. It is not uncommon for the younger division champ to be as good as any of the older teams. I think their lack of distractions compensates somewhat for their lack of maturity. Confident, intelligent, inquisitive kids have a lot on their plate once they reach 8th or 9th grade. Minnesota crowns younger and older champions not because we don't think the youngsters can compete, we do it to have twice as many award winners.

[dcsastry]

I agree with timdavid - I believe that it is impossible, maybe a bit reckless, and perhaps unfair to state that a team was undeserving of a Champions award. We have no access to the relevant data & discussions, and I can see nothing that this team has done that would show that they were not deserving. There may have been flashier teams that display their Core Values more visibly and maybe worked harder and longer. But all that highly visible enthusiasm is mostly irrelevant when it comes to judging, which is based on the teams' interviews during judging sessions.

I feel that with the large number of judges, many dedicated volunteers for many years, it is highly unlikely that any kind of "funny business" went on during the tournament, and that the team did so well because of how they presented themselves.

That being said there is still a perception problem with the process, and it is a byproduct of the fact that the judging process is kept secret and what parents, coaches and all teams are able to visibly see & experience during a competiton (robot game performance, pit designs & project ideas, team enthusiasm, etc) is not factored into the judging, or is weighted fairly low. Also, the secret judges that roam the tournament can potentially take a team out of the running for the Championship, but that will invariably not be communicated to the team as improvement feedback.

This has had the unintended consequence of sending a message, intended or not, appropriate or not, true or not, that hard work is not a requirement for being the championship team. Regardless of the fact that "work hard" is not an FLL Core Value, it will be difficult for coaches to motivate kids & their parents to put in the extra time required to truly learn new STEM subject matter. I do plan to address this with our Regional Partner and our Tournament Director once the season is complete.

My concern with this transparency issue is mostly geared toward keeping up the motivation of kids and their parents for robotics, FLL and STEM activities it introduces. Having well-run and highly visible teams publicly communicating their frustration and dropping out of FLL will have serious consequences.

[Dean Hystad]

Originally Posted by BlueCheesyFlamingos

This. Exactly my problem. I simply don't know what to tell them.

Why do you feel you have to tell them anything other than "Great Job"? I've done quite a few callbacks, both as a coach and judge. When you whittle 200 or more teams down to the top four, you don't have any clunkers. Every one is very good. Every one is worthy of the award. By any objective measure they are likely all equal.

Last year I think we spent around half an hour trying to decide who should get the teamwork award. What would I write? "We finally decided that team A's ability to recognize a bottleneck and reorganize on the fly trumped team B's better initial planning"? Did the team somehow change between initial and callback judging? Are the earlier comments invalid?

That was just a four judge Teamwork award panel. The Champion's award panel is twice as large, their decision twice as hard, and it is the last thing that happens before the awards are distributed. It is little wonder that they aren't writing a detailed analysis of their decision and then composing an eloquent tribute to be read at the award ceremony, which is already delayed.

[Dean Hystad]

Originally Posted by baybots2012

I will bet two bricks, a motor and a presentation board that if the Champion award was decided by a vote of the Coaches it would be more accurate and far less subject to criticism.

I will wager that if the kids themselves were told it was their responsibility to pay attention to what was going on at the tournament and ask other teams about their research and bot and to vote on a Champion that it would also be more accurate, less subject to criticism and way more fun.

You know I llike that idea, the teams vote in the Champion. Maybe just the teams who won some other award.

That's how they do it in the wrestling tournaments at States. It's the weight class winners themselves who decide on the Champion of Champions and that's the most covetted award of them all.

Teams have no idea what other teams have done. You don't see the research presentation or technical judging. You don't see the teamwork activity. You maybe see one or two robot runs. Coaches and teams have no information on which to base a decision. It may make sense in wrestling, but it makes no sense for FLL. Being acknowledged by your peers is only meaningful when your peers know who you are.

[Dean Hystad]

Originally Posted by Jkyaths

a) Outcome 1 - Good news here is that there will be more teams participating in robotics however, the quality of performance at the competition will be very poor since we are promoting mediocracy – and the message that the older team members are taking is that mediocracy and limited effort is ok…
b) Outcome 2 – Teams with older kids – 12/13/14, kids who are smart, energetic, intelligent, motivated who prioritized robotics over – math/science Olympiads, swim championships, ..etc will prefer to quit robotics or prioritize in an area where there is a higher correlation between effort and outcome.

Outcome 3. More teams are exposed to FLL. More kids get interested in science and math. More kids grow up to be technically, mathematically and scientifically literate. They may not take up STEM careers, but they will be an informed populace less likely to be swayed by demagoguery. They are aware and hard working and make short work solving all the problems created by antiquated, winner take all thinking. We achieve utopia.

I can make up outcomes with the best of them.

As for older, smart, energetic, motivated robot nuts, they are participating in FTC or FIRST or some other more demanding robotics competition. FLL is open for kids from 3rd through 9th grade. Now that there are so many choices I don't think FLL is something you should be doing all six years.

[BlueCheesyFlamingos]

Originally Posted by Dean Hystad

Why do you feel you have to tell them anything other than "Great Job"?

Because the kids are asking. They want to know and telling them all the teams are good is never enough. These are highly intelligent, motivated kids and telling them that all the teams are amazing only brings up more questions.

Maybe our situation is a bit different but when the kids saw they had scored a perfect score across all categories in the project, they wanted to know what made the other teams better. I wish "great job" was all I had to say. It would make my job as coach a lot easier.

[Dean Hystad]

Originally Posted by BlueCheesyFlamingos

Because the kids are asking. They want to know and telling them all the teams are good is never enough. These are highly intelligent, motivated kids and telling them that all the teams are amazing only brings up more questions.

Maybe our situation is a bit different but when the kids saw they had scored a perfect score across all categories in the project, they wanted to know what made the other teams better. I wish "great job" was all I had to say. It would make my job as coach a lot easier.

In FLL the kids come up with the answers. Turn the question around on them and ask what they think they could do to make the project better. It isn't your job to tell them what to do.

At the leve of teams being considered for awards at the state tournament, the answer to "Why did team A win an award instead of us?" is of no value. Next year is another team, another problem, different judges.

[Jkyaths]

[QUOTE=timdavid;56999]I can certainly understand the desire for more transparency in the awards process - that was the initial focus of this thread. What I don’t like is insinuating that the named champion is undeserving of the award. I haven’t read any conclusive evidence supporting that assertion.

• According to official FLL guidelines, the Champion’s Award criteria requires that the team performs well in all three judged areas (Project, Robot Design and Core Values) and receives a Robot Performance score in the top 40% of official Robot rounds among other requirements. If you don’t know how well the Champion team was evaluated in those three areas, how can you claim they are an undeserving champion?
• As Dean noted, a short announcement for the Champions award is common. At the end of a long day, everyone is tired, and people are ready to go home.
• Cute, not extremely informative remarks about why a team won a particular award are also very common. Give the judges a break – they are trying to be entertaining at a time when their brains are totally fried.
• Suggesting that a team is not worthy of an award because it consists of 10 year old 5th grade girls is insulting. I’ve coached teams of 4th and 5th graders that were comprised of 9 and 10 year olds, and they were capable of competing with some of the best teams in the country.

I can understand feeling disappointed that your team didn’t win a particular award. I can understand wanting more information and transparency in the judging process. However, stating that the overall champion is not deserving of their award is not gracious and not professional.


Responding to each point

a) The judges are tired to take ..10 min more to provide a more detailed explanation is not an acceptable reason - Some of the "championship calibre teams" have spent an average of 10 hrs/week * 30 weeks = 300 hours* 10 kids = 3000 hours+ hours spent by coaches...parents.....sibblings waiting around - If the organization does not have the time.....spread the tournament over two days.....Not having an additional 10 mins or 30 mins to come up with a detailed description of what makes the selected teams a champion is not acceptable

b) I agree that that 10 year olds can be smart, I have a 10 year old and can relate to that group of kids. However there is no way a 3-10 year olds can - communicate, design a robot+ build attachments, comprehend the depth of finding an impactfull innovative solution in the area of food safety...as compared to teams with 2-3 years of experience and toppers in middle/high school. However I do understand how an entirely subjective system which is perception based, clouded by minimally trained judges, lack of a defined process, the outcome could be entirely arbitrary since it is not an objective data based decision. Here is a simple math

- There are ~24 judges at the tournament. Each team gets to meet 6 judges who gets first hand visibility into the depth of knowledge/effort that the team has demonstrated...this makes it a 1 in 4 chance. This gets diluted when random judges walking around, cast a vote with minimal knowledge about the team but by simpling observing a team member who maybe having a bad day...I was always trained to think that engineering/science is about precision, accuracy and repeatability ...not about luck or a " coin-toss"