AI at the Far Edge
by Alexander Sack
A Smarter Edge
The concept of “edge computing” has been around since the late 90s, and typically refers to systems that process data where it is collected instead of having to both store and push it to a centralized location for off-line processing.
The aim is to move computation away from the data center in order to facilitate real-time analytics and reduce network and response latency. But some applications, particularly those that leverage deep learning, have been historically very difficult to deploy at the edge where power and compute are typically extremely limited. The problem has become particularly accute over the past few years as recent breakthroughs in deep learning have featured networks with a lot more depth and complexity, and thus require greater compute from the platforms they run on. But recent developments in the embedded hardware space have bridged that gap to a certain extent and enable AI to run fully on the edge, ushering a whole new wave of applications. And new data scientists and machine learning engineers entering the field are going to need to be prepared on how to leverage these platforms to build the next generation of truly “smart” devices.
Enter Deep IoT
Historically, edge computing devices, particualrly IoT, rely on the cloud for most of their compute – the cloud effectively becomes a device’s main processing engine. For example, Amazon’s Alexa doesn’t perform voice recognition on device but rather records voice and then sends it off to AWS for post-processing to create an actionable response. However, this comes at a trade off since real-time response isn’t possible when either the network connection is too slow to shuffle data back and forth between device and cloud or it simply doesn’t exist – both typical deployment scenarios for a lot of edge computing applications.
Yet the industry has seen an increasing demand to perform inference on the edge as many applications require applying sophisticated, highly computational deep learning and machine learning algorithms in real-time. For example, technologies like facial and voice recognizition as well as self-driving cars all require data to be processed (and by potentially several models) as it is being collected.
Use cases like these have cultivated a cottage industry of solutions, all tailored to various machine learning and deep learning workloads. In fact, many of the leaders in this space, Google, NVIDIA, and Intel to name a few, all offer complete embedded platforms in a myriad array of form factors that allow data scientists and machine learning engineers to build “smart” edge ready devices at relatively low cost.
Though each of these platforms have different trade-offs, what’s in common across all of them is each provides an embedded accelerator (GPU or VPU) that off-loads the CPU for real-time processing. Some also include one or more hardware video encoders/decoders on-chip that are directly linked to the embedded GPU enabling a complete end-to-end accelerated pipeline. For example, in computer vision applications, high-definition or even ultra high-definition video can be recorded, decoded, as well as inferred all in hardware at extremely fast frame rates.
Modeling for the Edge
Although “smart” edge computing devices offer more compute power, data scientists and machine learning engineers still need to optimize their models to make efficient use of that power. Techniques such as quantatization and model pruning as well as understanding how different metrics effect their model’s overall peformance are all key in building robust solutions on the edge. Luckily, most of these platforms support popular frameworks like Tensorflow and PyTorch as well as ship pre-optimized models that can be deployed right out of the box for rapid prototyping.
More about the author:
Alexander Sack is a machine learning engineer at Corteva Agriscience and a former TDI Fellow (Winter 2019). Before falling for the gradient descent trap, he worked as a Principal Software Engineer specializing in systems programming and operating systems. He holds a Bachelors and Masters in Computer Science from Stevens Institute of Technology with high honors. In his free time, he listens to a lot of heavy metal.
