![]() ![]() Model conversion from PyTorch to ONNX to Tensorflow for use in mobile applications, Unit圓d and Androidĭevelopment of a REST API by using Flask. Trained CNNs using a custom architecture and alternatively using transfer learning with pre-trained PyTorch models Trained CNNs with PyTorch specifically for mobile applications and web applications The faces are segregated and categorized into 7 classes:Ġ=Angry, 1=Disgust, 2=Fear, 3=Happy, 4=Sad, 5=Surprise, 6=NeutralĪ total of 28,709 examples was used for training the models, which were further validated with 3,589 examples. It was prepared by Pierre-Luc Carrier and Aaron Courville and consists of grayscale facial images of size 48x48 px. We used the FER2013 dataset from Kaggle for training. It may also be used as a HR tool and support recruiters in evaluating the overall confidence level of an interviewee by measuring the change in emotions during the interviewee's responses in real time. For example: How Kaspar the robot is helping autistic students to socialise Text overlays displaying the detected facial expression: EmoAR might help people with Asperger syndrome and autism in learning about the expression of a face.Ī classifier of facial expressions (trained model) will enhance robotic projects related to therapies that help autistic people to socialize. (We took a look at the current beta version of an AR creation tool for social media platforms, but unfortunately have not yet found a function/method in the scripting API to access the CameraTexture by code in order to send these camera images via a web request to our inference tool that would return the PyTorch model prediction to the social media app where we would suggest and/or overlay specific AR filters.) In social media apps, this could be used to suggest or pre-select emojis, avatars, artistic visual content, 3D models which suit the detected facial expression, so that the user can take a selfie or videos with superimposed emojis, avatars and/ or artistic visuals as AR filter. Impact of EmoAR and its further implications Since ARcore is only supported by a small number of Android devices, we also deployed the best PyTorch model to a web app using Flask and Heroku, but without the AR feature. This virtual augmentation of a face is done with Augmented Reality (ARCore). Depending on the model prediction, different virtual content overlays the face. The facial expression of the detected face is determined in real time by using our trained model. The detected areas with a face are fed into a model that was trained on the public FER dataset (from a Kaggle competition 2013). The live AR camera stream of a mobile device (Android) is input to a segmentation tool (using tiny YOLO) that detects faces in the video frames in real time. For example: Depending on the predicted facial expression, EmoAR would superimpose or suggest different Augmented Reality "filters". What it doesĮmoAR is a mobile AR application (an Android device with ARCore support is required) that aims to recognize human facial expression in real time and to superimpose virtual content according to the recognized facial expression. Most tasks are done asynchronously, the rendering of virtual AR overlays is done by accessing the OpenGL thread. Instead of using OpenCV’s techniques, we access the AR camera stream, we use YOLO to determine a person in a video frame, we crop this AR camera image, convert it to a Bitmap and feed a copy of it as input in our custom PyTorch model to determine the facial expression of this face in real time. With OpenCV techniques, haarcascade etc, this would have been an easy task, but it would be challenging to use the camera stream for the Augmented Reality overlay. In our project we need to determine whether and where faces are located in a video frame. ![]() OpenCV and AR frameworks like Vuforia, ARKit, ARcore do not work well together, because the input video stream of the AR camera has to be shared with the other frameworks and/ or SDKs. Where to place the virtual content in the 3d coordinate system? The Augmented Reality Android app, libraries etc. Section: The Augmented Reality Android app EmoAR – Facial Expression Recognition & Augmented Realityįuture use case, a tool for people on the autism spectrumįuture use case sort AR filters in social media appsĪbout some hyper parameters and transforms
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |