Improve color detection for typical use cases #421
Conversation
|
Tagging @Novakasa: so this keeps your bicone distance function, but only applies it if the user provides custom colors, which is when it works best. What do you think? |
That sounds good to me. When you are working with the ideal colors, you're not really matching against a real color, but more like the abstract concept of one, so it makes sense to have a specialized matching algoirithm for that. Do I see correctly that this also modifies the mapping from raw to HSV for the specific sensors, so this can be regarded as a loosely breaking change? You might consider putting this into the changelog, so when this releases people know to recalibrate. |
|
Thanks for the input! Yes, I'm considering changing the hue correction, which was non-monotonic so skewing things a bit much. For the boost color and distance sensor, I noticed that it produces quite high saturations to begin with, so the correction we applied for Spike probably never should have been applied to the Boost sensor. This would ideally need a properly adjusted rgb to hsv mapping just like the calibrated the hsv-to-rgb conversion, but I don't know that will ever happen. |
Sets the stage for the next commits. This does not change any color detection code yet.
This didn't work great for both sensor types, so move them to their own variants. This still needs to be revisited properly, but this is less bad than before. Also make the h adjustment monotonic so it doesn't skip ahead.
The bicone mapping is highly distant dependent, which makes it suitable in limited cases, and it doesn't work great with the default colors. If only saturated or grayscale colors are in the mapping, we can generally get a much better result by looking at hue only for saturated colors and looking at value only for unsaturated colors, and use the saturation to decide which to pick. This also means we won't need the workaround of having negative V for None. The logic here is that if you do specify fine-grained, calibrated colors, then it will use the original bicone distance mapping.
|
Hello, I am a little confused by this in the changelog "Improved color detection when using default colors. Also changed the HSV calculation for the SPIKE Color Sensor and Boost Color and Distance Sensor to make them more similar. " Does this mean that as soon as I define custom colors, there is no improvement to be expected? That is the "old" method is used? |
The reason for this change is actually that previously, the performance for the default colors was more unreliable than when you calibrated the colors, due to the fact that the default colors are "ideal" colors with maxed out brightness and saturation. Because the previous approach already worked really well for calibrated colors, this change does not apply to "non-ideal" custom calibrated colors, where there likely is no need for any changes. One thing to keep in mind though is that custom calibrated colors will be more sensitive to varying distance and lighting conditions than when using the default colors. I believe you can make the method interpret your colors as "ideal" colors by specifying all colors with with So in that sense, you can choose which approach fits your use case best.
|
|
Thanks for your detailed response. So if I understand correctly, the "new" colour matching method is applied only if the color's saturation is 100% or 0%? A question: is there a way to determine a matching confidence score or index of sorts? The use case is as follows: we need to determine colors as we drive by an object. The idea is to take 5-10 readings (as fast as possible) and select the best match. So far we selected the best (most likely correct) match based on reflection value, assumign the closest the sensor is to the object, the more accurate the color matching. But I wonder if there is a better way at picking the closest match from a collection of "samples". Especially, we noticed that sometimes being to close to the object makes the color matching actually worse. I know there were plans (requests) to expose pbio_color_get_bicone_squared_distance(hsv_a, hsv_b) in the past. Regards, |
Yes,, but only if all specified custom colors have either saturation = value = 100 or saturation = value = 0, the new algorithm will be used.
Yes it sounds like you would profit from being able to access the internal color distance function. I opened this in the past, maybe someone can revive it: #217 |
|
Otherwise, you could try to recreate the color distance function yourself, it is implemented in C here: https://github.com/Novakasa/pybricks-micropython/blob/7d400e51f9d81804c48a2d6738d0f822dd1061d2/lib/pbio/src/color/util.c stuff like hsv_a->s just means you need to obtain the saturation of the color A, otherwise it should be straightforward to adapt to python. ChatGPT could probably easily adapt it to micropython as well. |
True, but what would be the performance penalty? And doesn't the firmware use some further mappings, non-linear stuff? |
|
It should work fine with the colors you can obtain yourself, pretty sure the C impl also just uses the same colors a user has access to. The performance will be worse but naively I would think it wouldn't be too bad. I would be most worried about memory when storing a lot of colors in a list (~hundreds?), but you would need to do that anyway even if the internal method is exposed. The biggest gotcha when adapting the code is that the internal implementations of sine and cosine map from degrees to the range from -10000 to 10000, so keep that in mind. |
Not really: Only the colors I want to detect, which is 5 or 6. For the colors measured, I just compute the distances for each measurement as I drive by and store only the the one with the smallest distance from the target (basically) |
The bicone mapping introduced in #104 works quite well in controlled circumstances and fixed distances, but less so in other cases.
For example, the edge of black and white may be seen as green or blue depending on the distance when it happens to be closer according to the distance function. See e.g. https://github.com/orgs/pybricks/discussions/1760, which has been bothering me for some time 😄
In many cases, we do have extra information that we can use. The bicone cost function is a measure for the distance between
HSVaandHSVb, but there is additional insight we can get from considering all candidate colors. Often we have a set of idealized color (e.g. pure red or yellow) and idealized grayscale colors (none, black, or white). One strategy we can apply then is to pick the nearest hue if the measured saturation is high, or otherwise pick the nearest value.This PR applies that strategy when suitable candidate colors are given. Otherwise it defaults to the bicone function. So people who calibrated their own colors can continue using that strategy.
This also lets us undo the workaround of using negative value to get black/none to work at all.
These pictures show the improvement in range. Left (prime) is stable firmware and right (essential) is this PR. The bricks are moved towards the sensor until the hub lights up with the detected color.