Edge-AI-technology
Answering the "W"s and "H" of Edge AI
WHAT is Edge AI?
Edge AI, as the terms speaks for itself, is the AI technology integrated at the edge, i.e., closer to the user rather than hosted on the cloud.It refers to deployment and execution of AI algorithms directly on edge devices (microcontrollers, microprocessors, mobile phones, cameras, IoT sensors etc.)– without needing to communicate constantly with the cloud.
WHY Edge AI?
Traditional approaches of AI integration put forth the following limitations:
- Long Distance Cloud Communication: Cloud AI requires data to be sent over the internet to remote servers for processing, resulting in high latency.
- Privacy and Integrity risks: Transmitting sensitive data over clpoud increases the risk of data breachesand privacy violations.
- Cost: Uploding data to cloud consumes significant bandwidth and incurs high computational costs, especially at scale.
- Persistent improvement requirement: continuous changes with respect to updates and features contribute to the increased computation and maintenance costs.
- No real time insights: Real time data monitoring or insights is difficult on cloud AI due to long distance transmissions , lag and noice interferences in the signals.
These limitations are overcome by implementation of new technology called Edge AI. These are done as follows:
- Low Latency : Real – time decision-making without delays caused by cloud round trips.
- Improved privacy: Sensitive data says stays local, reducing risks of exposure.
- Reduced Bandwidth Usage : Less need to transmit raw data to the cloud hence reducing the clud costs.
- Offline Capability: works even without internet access. It has capability to work in offline state hence removing the internet dependency of the netwrok.
- Lower Operational Costs : since the reliance on cloud network is reduced, computational cost of cloud network is reduced to a great extent.
HOW does Edge AI function?
Edge Ai works in two phases:
1. Edge Devices: Edge devices include sensors, cameras, and other IoT devices present at the edge of any network.
2. AI model: AI models for different tasks like image recognition, NLP, voice recognition, etc., are trained on powerful GPUs using large datasets in the cloud or on-premise servers.
3. Hardware devices: Edge computing devices like Nvidia Jetson, Google Edge IPU, etc. are used as hardware devices to integrate AI models into hardware.
How to run AI on Microcontrollers and Microprocessors??
AI is integrated into microcontrollers and microprocessors using Edge AI and TinyML. AI tasks like gesture recognition, object detection, keyword spotting, etc., are performed using these devices.
This integration involves the following steps:
Step 1: Training model
Training deep learning models requires large datasets and powerful GPUs using frameworks like TensorFlow/PyTorch and Keras/Scikit-Learn. This training is done on models based on CNN networks, as briefly discussed here.
Step 2: Optimise model for edge devices.
Due to limited memory in MPUs and MCUs, we need to compress our deep learning models so that they’re compatible enough for the respective embedded hardware devices (edge devices). Techniques like quantisation and pruning are used to reduce the model’s size and computation demands.
Step 3: Convert model for Deployment
The model weights are converted into formats like “.pt,” “.h5,” or “.pb.” This is done to convert into lightweight model formats due to memory limitations.
Step 4: Flash Model onto the Target Hardware
Based on the device type, the models are flashed using TensorFlow Lite (for microcontrollers) and LinuxOS (for microprocessors). The model is exported with the extension “.h” and is embedded into firmware. Hence, an inference code is written using Arduino SDK.
Step 5: Inference running at Edge
Once deployed, the model can start performing inference on the device in real time. It performs AI tasks like classifying sensor readings, recognising voice commands, and detecting faces from a camera stream.
Edge AI Frameworks
LiteRT or Tensorflow Lite
TensorFlow Lite, or LiteRT, is Google’s high-performance runtime for on-device AI. It is used to run a wide variety of AI/ML tasks or convert and run TensorFlow, PyTorch, and JAX models to TFLite format using AI Edge conversion and optimisation tools.
Key Features:
- ODML (Object Detection and Machine Learning ) contraints: LiteRT addressses fie key ODML constraints: Latency, Privacy, Connectivity, size and power consumption of ODML models.
- Mult-Platform support : LiteRT is compatible wth Android, IOS, embedded linux and microcontrollers.
- Multi-framework model options : Models built using tensorflow, Pytorch, amd JAX are converted into FlatBuffers format (.tflite), enablng wide range of SOTA models on LiteRT.
- Diverse Language Support: LiteRT ncludes SDKs for Java/Kotlin, Swift, Objecte-C , C++ and Python.
- High Performance: Hardware acceleration through specialized delegates like GPU and iOS Core ML.
Edge Impulse
It is an end-to-end platform to collect data, train models, optimise, and deploy to devices. Edge Impulse is beginner-friendly and provides drag-and-drop tools and SDKs for C++, Arduino, and Python. It supports TinyML, time series classification, object detection, and keyword spotting.
OpenVINO Toolkit
OpenVINO is built by Intel for computer vision and DL inference acceleration.
It converts models from Tensorflow, ONNX, PyTorch, etc. into an optimised intermediate representation.
It is targeted for Intel CPUs. NCS(Network Control System) and FPGA(Field Programmable Gate Arays) platforms. These are bestfor real-tme vsiion on intel edge hardware.
NXP eIQ ML Software Development Environment
It is a full ML SDK (software developer kit) that supports TensorFlow Lite, Glow, Arm CMSIS-NN, and ONNX.
It is an AI software environment that enables the use of ML algorithms on NXP Edgeverse MCUs and MPUs, including i.MX RT crossover MCUs and i.MX family application processors.
An eiq software consists of:
- ML workflow toolkit (eiq toolkit).
- Inference engine.
- Neural network compilers.
- optimised libraries.
NXP’s eIQ supports a wide variety of models varying in domains from audio to vision and miscellaneous. The models performing tasks in these domains form the model zoo at eiq.
Features
- eIQ Autmotive Enablement
- eIQ genAI FLow
- MPU Vision pipelining
ONNX Runtime
Challenges with Edge AI
With a versatile interface for integrating hardware-specific libraries, ONNX Runtime is a cross-platform accelerator for machine learning models. Models from scikit-learn, TFLite, TensorFlow/Keras, PyTorch, and other frameworks can be used with ONNX Runtime.
Machine learning models are powered by ONNX Runtime Inference, Microsoft services, and products like Office, Azure, and Bing.
The following are some instances of ONNX Runtime Inferencing use cases:
1. Boost inference efficiency for a broad range of machine learning models
run on various operating systems and hardware.
2. Develop with Python but implement in a C#, C++, or Java application
3. Utilise models developed in various frameworks to train and carry out inference.
How does it work?
- Get a Model: The model can be trained on any framework that supports conversion or export to ONNX format.
- Load the model and run with ONNX Runtime.
- Tune the performance using various runtime configurations or hardware accelerators.
ONNX Runtime applies a number of graph optimizations on the model graph then partitions it into subgraphs based on available hardware-specific accelerators.
Key challenges for Edge AI are:
- Limited Resources: Edge devices have less computational and processing power. Memory resources are limited too, making it hard to run complex AI models.
- Maintenance Costs: Edge devices at local require constant updating and managing; hence, maintenance costs are increased.
- Maintenance complexity: regular updating and management of distributed devices is complex and time-consuming.
- Security Risks: More devices mean a bigger attack surface, and many lack strong cybersecurity features.
- Data Fragmentation: Local processing can create isolated data silos, making unified analysis harder.
- Integration Issues: Edge AI must work with diverse hardware and legacy systems, leading to compatibility problems.
Advantages of Edge AI
Listed following are a few of the advantages of Edge AI:
- Increased speed of signal transmission.
- Improved reliability.
- Increased scalability.
- Improved net efficiency of the network.
- Improved security.
- Bandwidth and cloud cost reduction.
- Real-time insights/monitoring.
