Announcement

Collapse
No announcement yet.

A call for transparency of judging in FLL

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Re: A call for transparency of judging in FLL

    I received an email yesterday from FLL that the annual coaches survey was available. I took the survey, and it had a number of questions about judging, the rubrics, and the awards process. Looks like a good opportunity for coaches to give feedback to FLL.

    Comment


    • Re: A call for transparency of judging in FLL

      Originally posted by timdavid View Post
      I received an email yesterday from FLL that the annual coaches survey was available. I took the survey, and it had a number of questions about judging, the rubrics, and the awards process. Looks like a good opportunity for coaches to give feedback to FLL.
      I didn't receive an email about that -- I get various other FLL emails. Is this survey limited in some way?

      Comment


      • Re: A call for transparency of judging in FLL

        Originally posted by Nik Bhatt View Post
        I didn't receive an email about that -- I get various other FLL emails. Is this survey limited in some way?
        I don't know. I received the email March 29th from a person at us first.org.
        I emailed her to see if I can post a link to the survey on this forum.

        Comment


        • Re: A call for transparency of judging in FLL

          I've been reading this thread for some time, and have been thinking about how best to reply. Many people have spoken very eloquently (thank you Dean and others) about FLL judging, what it represents, how it's done, and numerous other issues. I'd like to invite any additional feedback about ways to improve FLL judging to be sent to me, Skip Gridley, Global Judge Advisor for FIRST LEGO League at flljudge@usfirst.org. We really are interested in your feedback to help make things better.

          The mission of FIRST is to inspire kids to get excited about STEM. It does it by using robotics as a hook to get kids interested, having them work with with hopefully good adult role models (coaches, mentors), and indirectly teaching them other skills like teamwork, confidence and the ability to solve problems. It also features exciting competitions to give kids a chance to be recognized for their work in ways that are familiar to them from sporting competitions.

          Almost everyone who does FLL does so via choice, as a volunteer, for free, doing the best they can in the time they can with the resources they have. Just like the teams do. Sometimes they get it wrong, but in the vast majority of cases, I would challenge reasonable people to get significantly different results.

          Yes, judging is subjective. The rubrics go a long way to minimizing it, as does the deliberations process. I welcome (notice I say welcome) anyone to sumbit more objective criteria that include all the key takeaways from FLL for each of the three areas, and are more objective than the current rubrics. I often hear that the rubrics are subjective, but receive little to no actual suggestions on how to change them.

          Accept the fact that full results of all teams will never be made public to all teams. What good does the world knowing which teams finished last do to those teams that already know they have many areas to improve? FLL and FIRST are about getting MORE kids interested in STEM, so we try to celebrate ALL teams in some fashion and focus on the positive and ways to improve. Yes there are awards. They are about Recognition. The "R" in FIRST.

          By all means, please feel free to e-mail me. Dean Hystad mentioned a discussion I had with his girls in MN several years ago. I do listen, I'm happy to share as well. In the end everyone benefits from increased understanding, if not always complete satisfaction.

          Thank you,
          Skip Gridley
          Global Judge Advisor, FIRST LEGO League
          flljudge@usfirst.org

          Comment


          • Re: A call for transparency of judging in FLL

            Originally posted by sgridley View Post
            ...It also features exciting competitions to give kids a chance to be recognized for their work in ways that are familiar to them from sporting competitions.

            The mission of FIRST is to inspire kids to get excited about STEM. It does it by using robotics as a hook to get kids interested...

            Yes, judging is subjective. The rubrics go a long way to minimizing it, as does the deliberations process. I welcome (notice I say welcome) anyone to submit more objective criteria that include all the key takeaways from FLL for each of the three areas, and are more objective than the current rubrics.
            Thanks for the first post!!!

            First, I want to highlight two points:
            1. There is a competition
            2. Hooking the kids involves robots

            Why did FLL remove the ONLY competitive and objective element, i.e., Robot Performance from the overall score considering that it is the hook that attracts kids to the program?

            I can only guess that it was done intentionally to remove any objective evaluation.

            For robot design / programming / Strategy, there are many quantitative options and here are a couple:

            # of missions attempted
            # of missions per program
            # of programs
            Score consistency (average score vs. best score or Std Deviation).
            Tony A.
            Los Angeles FLL Region OP
            www.la-fll.org
            https://www.facebook.com/LosAngelesFLL/

            Comment


            • Re: A call for transparency of judging in FLL

              Robot Performance has not been removed at all from the competition. Teams can still receive a trophy for excelling in robot performance. If teams choose to focus on only one aspect, they are unlikely to win the Champion's Award, which is for overall excellence.

              Robot performance was not removed from Champion's Award criteria with the intent to remove objective evaluation.

              FIRST's highest honor, the Chairman's Award in FRC has no robot component whatsoever.

              For every complaint we've received for "lessening" the impact of the robot, we've received complaints about the robot meaning too much because of the 40% hurdle.

              FIRST founder Dean Kamen says it better than I could in the following video:

              http://www.youtube.com/watch?v=ZuRhgX7pp_I

              Comment


              • Re: A call for transparency of judging in FLL

                Originally posted by sgridley View Post

                Robot performance was not removed from Champion's Award criteria with the intent to remove objective evaluation.
                Thank you for posting in this thread. I might send an email to you with some suggestions, time permitting.

                Can you please clarify for me:

                If a team gets the highest score in robot performance (and therefore the respective reward), is this team still eligible to get the Championship award?

                Is there an "official" rule for that?

                Thanks for your great work!
                Claus, Coach "It Should Have Worked", San Bruno 4-H

                Comment


                • Re: A call for transparency of judging in FLL

                  A team that wins the Robot Performance Award is eligible to win the Champion's Award, yes. The Robot Performance Award is the one Core Award that any team may win even if they win another Core Award.

                  Thank you for your work too. FLL needs more coaches and mentors!

                  Comment


                  • Re: A call for transparency of judging in FLL

                    Skip,

                    Thank you for your posting.

                    I started this thread after witnessing the look of disappointment on the faces of the kids in the FLL team I help to coach at the conclusion of the NorCal Capital District Championship. I told the kids to get a good night’s sleep and then write, independently, how they, individually and together as a team, had performed in the “closed door” judging sessions. Their comments raised concerns about the judging process at the Championship. On behalf of the kids, I wrote to Ms. Lee Doucette, FLL Partner Services Manager at FIRST who forwarded my e-mail to Ms. Kathy Vachon, the Partner Services Manager for the California Region. Neither Ms. Vachon nor Ms. Anna Maenhout, Director, FLL, were willing to address the concerns raised by the kids. After a long delay we received the rubrics. These unfortunately did not include any feedback and failed to provide any insight into the judging process. I told the team to write directly to Mr. Jon Dudas, President of FIRST and Mr. Dean Kamen. The timing of your response suggests that it may have been prompted by this correspondence.

                    I will write to you at the e-mail address you have listed and also send you a copy of a sheet with judge’s comments that was given to the team, by one of the judges, as we were waiting for the Awards Ceremony to start.

                    There is always room for improvement and I hope that steps can be taken to further improve the judging process. At the end of the day, if a group of motivated, self-directed, hard working kids are left disillusioned, the stated mission of FIRST “to inspire kids to get excited about STEM” has not been accomplished.

                    Comment


                    • Re: A call for transparency of judging in FLL

                      Originally posted by LOLComets View Post
                      Thanks for the first post!!!

                      First, I want to highlight two points:
                      1. There is a competition
                      2. Hooking the kids involves robots

                      Why did FLL remove the ONLY competitive and objective element, i.e., Robot Performance from the overall score considering that it is the hook that attracts kids to the program?

                      I can only guess that it was done intentionally to remove any objective evaluation.
                      My guess is that building and programming robots is least in line with overall goals of FIRST. When FLL first started most teams placed heavy emphasis on the robot game. For me it was easy to understand. The robot game is fun, and it is the most "comfortable" part of FLL. A team can stay isolated in the basement or garage and be very successful at the robot game. No need to interract with other teams, technology or science experts, or society in general. Heck, you don't even need to interract much with each other. I see way too many teams whose robot was worked on by only one or two team members.

                      Teams can keep that focus on one aspect of the program and still win an award. But once you get that carrot you see there is a bigger carrot. Winning that carrot requires you do things other than building and programming robots in relative isolation. Things that may not be in your comfort zone. Writing letters, cold calling experts, sharing hard earned knowledge with other teams, speaking in public, etc...

                      There is no doubt in my mind that FLL changed the Champion's Award evaluation for some purpose. To think that it was done to make judging less objective (and thus more difficult) is foolish in my opinion. FLL would love to have objective judging criteria for all categories of the competition. FLL would love to have judging that could be done by checkbox. I just don't see how it is possible to have completely objective judging criteria for a competition that is so ambiguous and undefined. And I don't see how you can remove ambiguity in the problem description without influencing the solution. And I definately know that FLL has no interest in limiting how teams can solve problems. That is in the best interests of no one.

                      Comment


                      • Re: A call for transparency of judging in FLL

                        Originally posted by EmGe View Post
                        Skip,
                        After a long delay we received the rubrics. These unfortunately did not include any feedback and failed to provide any insight into the judging process.
                        That is unnacceptable. As I see it the value of FLL judging is the interaction between expert judges (hopefully) and the team. We have a nice conversation about how hard it is to program robots, or what a tricky issue food safety is, or how challenging it can be view a problem not only from your individual standpoint, but as a group, or as a society. You brag a little bit, deservedly, and I offer praise and maybe a few suggestions. After the interview is done I write up a synopsis of where the team did really well and how you can improve. Once the you receive that feedback the judging has been successfully completed. That I have to choose a team for an award is secondary. I have learned to keep track of how each team compares to the rubric so I can pick an award winner, but my top goal is that the team walks out of my judging room happy, proud, and better informed.

                        If that is not how judging happens at your tournaments you are being robbed. If your judging evaluation does not provide useful feedback you are being robbed. If you don't receive the judge's evaluations you should call the police. Every FLL team should demand that their tournaments are conducted in the best way possible. However, that responsibility needs to be a two way street in FLL. I think every team should have a responsibility to provide a volunteer to help organize or run an FLL event. You can demand that FLL be better, but only if you are willing to help make it so.

                        Comment


                        • Re: A call for transparency of judging in FLL

                          Originally posted by Dean Hystad View Post
                          If that is not how judging happens at your tournaments you are being robbed. If your judging evaluation does not provide useful feedback you are being robbed. If you don't receive the judge's evaluations you should call the police.
                          What police? When I conveyed the team’s concerns regarding the judging process to the FLL administration, not only did we not receive a response to the concerns raised, but Ms. Anna Maenhout, Director, FLL wrote back stating “Perhaps the FLL program is not appropriate for your team.”

                          Reading your post, I can liken the response the team received, to concerned citizens calling the authorities to report crimes in the neighborhood and being told by the police that perhaps they should move to a different part of town!

                          Comment


                          • Re: A call for transparency of judging in FLL

                            Originally posted by Dean Hystad View Post
                            However, that responsibility needs to be a two way street in FLL. I think every team should have a responsibility to provide a volunteer to help organize or run an FLL event. You can demand that FLL be better, but only if you are willing to help make it so.
                            Yes, indeed. That was exactly the reason why I requested the parents of all 7 kids (2 of whom are siblings) in my team to volunteer. Seven parents stepped forward to help at the Qualifier and six volunteered at the NorCal Capital District Championships.

                            Comment


                            • Re: A call for transparency of judging in FLL

                              Originally posted by EmGe View Post
                              What police? When I conveyed the team’s concerns regarding the judging process to the FLL administration, not only did we not receive a response to the concerns raised, but Ms. Anna Maenhout, Director, FLL wrote back stating “Perhaps the FLL program is not appropriate for your team.”

                              Reading your post, I can liken the response the team received, to concerned citizens calling the authorities to report crimes in the neighborhood and being told by the police that perhaps they should move to a different part of town!
                              I'll word this carefully, in light of the recent thoughts about remaining gracious and professional on this forum. The process that Dean outlined is the ideal. Perhaps the area for improvement is not with the process, but that not all events are equally successful at implementing it.
                              FIRST Tech Challenge Judge: 2010, Referee: 2017
                              FIRST LEGO League Mentor, Instructor, and/or Referee/Head Referee since 2011
                              FIRST Robotics Competition judge (Chairman's Award): 2014
                              Dean says I'm an "Oompa Loompa of Science"

                              Comment


                              • Re: A call for transparency of judging in FLL

                                Originally posted by EmGe View Post
                                Yes, indeed. That was exactly the reason why I requested the parents of all 7 kids (2 of whom are siblings) in my team to volunteer. Seven parents stepped forward to help at the Qualifier and six volunteered at the NorCal Capital District Championships.
                                That's great! If every team were so generous with their time you'd see a big change in how FLL programs are run.

                                Comment

                                Working...
                                X