Osiyo. Dohiju? Hey, welcome back.
I realize I’ve not defined the work I’m doing in terms of what it is. There is Extended Reality (XR) which encompasses Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). Virtual Reality basically means the entire view and interactions you have are generated and you do not see the “external” world. Augmented Reality is basically where you can see the real world and in that world items can be enhanced (say a 3d display of a car when you look at a brochure). The augmented portion adds to the reality. Mixed Reality is where you can view items in an augmented way as well as interact with items in that realm and the real world. An example is HoloLens or Project North Star. The real world has enhancements in view such as augmented overlays and you can interact with those overlays. So the ability to reach out and rotate a visual. Or select an item in a display. Intel has a little better definitions here.
So what is it that I hope to accomplish with my work? Mixed Reality. My goal isn’t simply to enhance the world I’m in. I have done that with a mobile app. I don’t want something simple that displays data. I have also done that with the very first hardware trial of the HUDuino (outdated source here and here). The short version of that was I took an ArduinoMega and utilized a small screen to display info from my phone via an app on my phone and BLE. I eventually used a cardboard tube, mirrors, and magnifying lenses to be able to view this set up as a HUD/HMD. That wasn’t good enough. While it accomplished the one thing I wanted, displaying speed while moving, it did not accomplish what I wanted which was an immersive experience I’ve been dreaming of for 30 years. I have a Holokit, a Vufine ARKit, and this one I found on Amazon. You can search this site for any of those and I have some reviews.
So, where is development right now? I have a successful visual test using Chrome or Mozilla XR Viewer on my phone. I have a new 5″ TFT display coming Saturday so I can put it with the Holokit and the Chinese version. I’ll probably use the Vufine as well for calibration, but I’ve got the other two sitting right with me. Back to the development progress. At this moment the functionality hasn’t changed from before only the HUD view portion will be adjusted (read: I’m moving items around so I can have the optimal viewing area). Voice commands only work, at this moment, if I have Chrome open on the host computer. They don’t work through my phone. Since this isn’t how I intended for it all to work I’m ok with that. It would’ve been cool if the commands could’ve passed through to the server and then replies come back and display on the phone. However, I can still test voice commands like /show grid/, /detect motion/, /face detect/, and /track object at 200 and 100 by 50/ where the grid location is squared and whatever is in those coordinates is tracked. As well, the text commands still work. I have 5 cameras I hook up to a LattePanda. I’m using the LattePanda since there’s a bug with Raspbian that you can’t use more than one USB camera. When that’s fixed then I’ll switch back to the Raspberry Pi 3+ and maybe get a Pi 4. Of the 5 cameras, I have one that is a zoom lens set out to a farther distance. I have one that is for closer focus. Right now, this one doesn’t have a purpose other than bragging I have 5 cameras hooked up. Eventually, this one will be for the main display. I have two cameras that are the same and they are mounted next to each other just over about where each eye is. These are for gesture tracking to make sure I have a wide enough field of view. They can also be used for 3d space so they can display together and give a stereo view.
I am very, very pleased with this work. I’ve even placed the WebXR work in an alternate file so I can come back to that later if I want.
Here are the items I want to have working. Again, waiting on the 5″ HDMI TFT to come in to fully test. I have a 7″monitor I’m using for the moment when I’m not using my phone.
- Reduce processing power when rendering 2 eyes
- Increase OpenCV speed by putting code back inline instead of via the OpenCV plugin. This one will stay for a bit longer because I’d rather focus on the rest of this list.
- Display PDF with scrolling capabilities
- Display text with scrolling capabilities
- Add accelerometer to LattePanda. I believe that this one is 6DOF. This will aid in adding AR to the display better. This is trivial with the built in Arduino portion of the board. And will also be trivial on the Raspberry Pi board.
- gestures should work even rudimentary ones
- set up two different display layouts – holokit and other one.
- See if there’s a fix for the Raspbian Buster mulitple USB cameras bug.
I still have an extensive list of features and enhancements to work through. However, when I get these items done I’ll make some videos using a phone, PC, LattePanda, and Raspberry Pi with the different viewers and approximate costs of each. Then I’ll open the code up for people to use.
I’m going to get back to working on these items so I can have them done by the time the TFT comes in.
Until next time. Dodadagohvi.
IN MY MIND BY WINK DOUBLEGUNS 