Serinda Graceland Update

ᎣᏏᏲ. ᏙᎯᏧ? Hey, welcome back!

I’ve spent the last couple of weeks working with the OAK-D. It has saved me so much time in development, where I had to figure out cameras, 3D layers, etc. The OAK-D and support libraries make using a LeapMotion obsolete and really sets up a prototype environment that I had to create. I don’t have to concern myself with stereoscopic views of cameras or writing code for hand detection and gestures; all of that is done when I plug in the OAK-D.

As well, the work I’ve done with OpenCV and AR/MR is still valid. I can still use that code. While the mixed reality test relies on AprilTags – those aren’t necessary. As I’ve stated before, a QR-code-style tag is just a proxy for an object detected. For example, if you want to display something over a lamp you detect, then you just substitute the found portion of the tag (aruco, april tag, hiro, QR code, barcode, etc) with the detected object. Now, this means that every lamp you come across would have the same display unless you train the AI to recognize only one lamp… which is entirely in the realm of working with OAK-D.

Currently, the Graceland code branch here runs on Windows 11 without the OAK-D. It will run with the camera on a laptop or USB camera but with some property changes. The only caveat to this is that the audio doesn’t work yet. The menu items work. The text commands work (type in the textbox). I’m working on the audio. I also need to test this on my Raspberry PI and Debian WSL. However, I did iron out all of the requirements. I haven’t set up a virtual environment (venv) for this yet, but I will be doing that as well, so it will enter the venv when starting up and then leave the venv when it closes. There is a lot to do, actually.

Because I spent a lot of time working on the plugin section and camera pool, I don’t have to do any more work there. The items work as expected. I am putting together a TODO list that will go on the README. However, I will list the few things I’ve got coming up that I want to accomplish sooner than later.

• Get the voice commands working.
• Wire the OAK-D as an option in the camera pool – and properties so the OAK-D could be the primary camera if a developer wanted – I do, so of course this is an option – and this will auto-detect the cameras still – but in the properties I’ll be able to define if I want OAK-D as the primary
• Set up object detection via the commands – e.g. recognize a lamp and not a QR code – and have an object display in 3D space – this is one of the examples modified to work with my base
• set up hand recognition to determine my hand location in 3D space – another example modified to work with my base
• work on the hand recognition so that it can determine that my hands are interacting with the 3D object – lots of intersection work
• use my hand to move the 3d object in 3d space

While this seems like a short list – and it is – there is a little bit of work to do to achieve these results. Once these are done, I think Graceland V3.1 would be complete. Then, it’s on to features for V3.2. I mean, accomplishing these are pretty big. I’ve been close before, but I think this will work a lot better.

In any case, right now, I’m trying to figure out how to fix the audio portion. One of the updated libraries isn’t working the way it should and is throwing an exception. Once that’s done, I’ll be able to start the list above.

Until next time. Dodadagohvi. ᏙᏓᏓᎪᎲᎢ.