Osiyo. Dohiju? ᎣᏏᏲ. ᏙᎯᏧ? Hey, welcome back.
I want this project to be as open source and cheap as possible while giving good results. The dichotomy is that I also want to work on the software so I can build the platform I envisioned. To start with, I had the SERINDA project moved to Github. More people go to Github than bitbucket. You can checkout the repo here. There is some code clean up and documentation work done.
I want this project to range in skill levels and funds. I’d like someone with a raspberry pi, a camera, microphone, speaker/headphone, and screen to be able to use the project the same as someone who has money to buy a RealSense camera, LattePanda, LeapMotion, etc. Right now, however, the minimal entry for this project is a LattePanda. It has the speed to process the LeapMotion as well as ORB-SLAM2 (or OpenVSLAM, whatever SLAM you’re looking at). If you are just looking for augmented reality that can be done with markers and a raspberry pi and picam. If you’re wanting to be immersive mixed reality you’ll have to add more.
With all of this in mind I’ve decided to go a little different route. I am sticking with the LattePanda until I can find an SBC that’s cheaper and at least as powerful. I’ll be using Ubuntu 18.04 as the OS instead of Windows. This is purely a tools reason. Most of what I want to do I can do in Windows. However, there are some elements that are simply easier to build and take less configuration on Linux. LeapMotion and RealSense support Linux SDKs. In choosing the RealSense cameras I don’t have to work on taking the two cameras that I have mounted and combine them for stereo vision in some way. I also don’t have to integrate the IMU since that’s already there. That doesn’t mean I’m not going to continue to work on those features so I can get into cheaper cameras and IMUs for SLAM/VIO. It means that I can fully build and test my software and add to my own SDK. When I was working in game development I would find that something didn’t work so I’d look into a library then work on that library until it worked. I’d find another library lacking for what I needed and work on that. Eventually, I worked on a bunch of libraries and never worked on the game. I want to work on this game. So right now, I am at a point in my life I can spend the extra and work through details of how my software will work in its entirety while working backwards and adding libraries for simpler tech. The end result is something that people can use no matter their level of AR/MR investment. The same is true for Unity. If I hadn’t had so many years working with OpenGL in various forms and many libraries I would probably go with Unity. However, I know how to make elements in Blender that can be exported as Collada (or whatever you want, really) and then imported into an OpenGL scene, even in WebGL and then used in the display.
Final note for today. I outlined my priorities for this project in the README in the repo. Six items that will complete the majority of integration of this project. In other words, with the completion of these six items the software will be at a base state of integration that additional elements (read: plugins and filters) can build on. I’ll cover these briefly.
- OpenCV and WebGL to have the same camera perspective
- Simultaneous Location and Mapping (SLAM) and Visual Odometry (VO or VIO).
- Object Detection and Object Recognition
- Interactive HUD elements
- Dynamic loading of scenes and objects
- Audio (TTS & STT)
I think these are pretty self-explanatory of why they were chosen and how they all interact with each other. Right now, SERINDA has a good base of odd parts that come together. Developing these six items means the foundation will be more integrated so the odd parts that exist can be built up.
One of the elements I’m working on, currently, is using OpenCV to recognize features then pass that data to the browser so ThreeJS can use it. I haven’t decided if I want to use WebRTC to process this entirely in the browser. Or if I should leave it mixed. Or if I should process it with OpenGL in the back end. I’d rather process as much on the front end as I can. Plugins and filters can still be designed and commands processed on the server just triggering display elements on the front end. If I can keep the browser speed fast enough I’ll probably do as much as I can on the front end. If I move items to the back end then I’ll convert my Python code to Rust for speed. Another option is to write all plugins and filters in both Python and Rust. Then users can change a property and us which ever one. I don’t know. I’d like to keep Python as much as I can so the learning curve and use curve is low for newbies. Especially since I have a lot of features I’d like to add and Python seems to be the way I’ll do it. Rust could probably do the same things. Idk. We’ll have to see. I’ll think about it a little now, but cross that bridge when I get to it later.
The goal right now is the test driven development (TDD) approach. Red, green, refactor. Red means it’s not working. So I get all of the components attached. I get the code started. When all of the code and components work together then it’s green. At that point, I can refactor to make things faster, easier, whatever.
I am excited for what these next phases bring.
Until next time. Dodadagohvi. ᏙᏓᏓᎪᎲᎢ.
IN MY MIND BY WINK DOUBLEGUNS 