No announcement yet.

Testing, reliability, troubleshooting, consistency, robustness, and practicing

  • Filter
  • Time
  • Show
Clear All
new posts

  • timdavid
    I like that terminology. I realized that as a judge, I had been using "reliability" and "consistency" interchangeably in my comments, when the term I really wanted to use was "robustness".

    As a coach, my team used to occasionally record mission trials on video and watch the tests in slow motion playback. It was useful (and fun) when the kids couldn't agree on why a mission was failing.

    Leave a comment:

  • Testing, reliability, troubleshooting, consistency, robustness, and practicing

    Many of us have our opinions about testing. Some of us (me!) like the "ten times" approach. These are my thoughts and personal approach.

    1. Kid creates what he/she thinks will be a good solution for a mission.
    2. Kid runs possible solution and it fails dramatically. Go back to step 1.
    3. Eventually possible solution works. That's one out of ten.
    4. Run it again. Did it work? If you can get ten successful runs in a row, then you are probably close to being competition-ready.
    5. If it only works, say three times before it fails again, then it is time to analyze the failure. Is there anything that can be done to fix it so that failure mode never happens again? If so, make the changes and go back to step 1.
    6. Sometimes random happens, and there seems to be no solution to make the mission more reliable. Escape velocity was one such mission for us this year. We came very close to 10/10 reliability, but sometimes the spaceship would still fall down. The team discussed having some mechanism to help hold the spaceship up, but they were concerned about it not being in accordance with the rules. Ultimately, the team decided the solution "was reliable enough", even though we could rarely get the 10/10 test to pass.

    * "Testing" is what you are doing when you try something new. It's probably the first 1/10 and 2/10 runs. Does this solution "work"?
    * "Reliability" and "consistency" frequently are interchanged. Many times they can be, but there are some differences. If you think about it, "consistent" doesn't say anything about how successful something is. You can consistently miss every time, and be very consistent. "Reliability" has a goal inference. You are reliable at work if your boss can usually count on you to deliver a good product. A robot is reliable if it can solve a mission 10/10 times. A robot that solves a mission 9/10 is less reliable. Reliability is the quality of being reliable, dependable or trustworthy. We all want robots that are reliable. Document your reliability tests.
    * "Robustness" is the quality of a system to continue to be reliable, even when inputs are altered. NASA frequently won't launch rockets when the winds are too high. The launch systems have been tested for reliability up to certain wind speeds. Beyond that level, the reliability is uncertain. Similarly, if the robot can be started two cm north or south of the optimum start location and still solve the mission reliably, then you know something about the robustness of your robot. Some inputs you may have a lot of control over (robot start position), and others you may not have any control (tilted table or lighting conditions at competition). We also want very much to have robots that are robust. Note that you can (and probably should) test for robustness. Document your robustness tests.
    * "Troubleshooting" is what you are doing when you are trying to determine the failure cause and create a workaround or better solution.
    * "Practice" is what you are doing when you have humans repeat something to become more consistent. Not necessarily more reliable. Practice DOES NOT make perfect. Perfect practice makes perfect. Practice makes consistent. Document your practice sessions.

    Like I said, these are my thoughts. This is not a primer on reliability. This comes from 20+ years of doing training in the military, and six years of maintenance and reliability testing for the navy. I'd like to see what others here like to do in the name of reliability.
    Last edited by SkipMorrow; 12-31-2018, 10:05 AM.