You will find here many of the projects that I have worked on, and, am working on. For any project, I have tried to document it as I have worked on it. I have tried to capture, and link to, what I found to be the most relevant sources – for learning and knowledge, as well as code. In most cases, the code is available on my github page.

If you’d like to know more about me, click here, or just drop me a line. I look forward to hearing from you – so, how are you?

The post Nice to meet you! appeared first on Kartik Nanda, Engineering AI.

As an undergraduate student at the Indian Institute of Technology (IIT), Kanpur, I remember this sinking feeling at the end of every semester – I don’t see myself doing that for the rest of my life! Four years and something like forty courses, and at the end of it, I still had no idea what I wanted to do – every course had been a struggle.

So I did the only logical thing – enrolled for a masters. It wasn’t until the second semester of graduate school at Notre Dame that I found a course that I just “got” – Algorithms. Of course, realization has taken many more happy years to sink in.

That interest in algorithms took me to signal processing early in my career. I designed delta sigma ADCs (or sigma delta, if you must!), digital filters, DSPs. Since these don’t exist in air, what I was actually doing was building mixed signal ICs. I started to think of myself as an IC guy (really! Prof. Kar’s course!!). 

But algorithms wasn’t done with me yet – I moved on to solar, or more specifically, generating electricity from solar. From designing ICs, to coding algorithms for building irrigation pumps that run directly from a solar PV panel. Solar, though, is more than just a product for me. It has enabled me to return to India (where I was born), to rural India – what a learning experience that has been. I am now also more conscious of the environment, and of how we live our lives. 

My focus nowadays is Artificial Intelligence (AI), specifically deep learning using neural nets. Projects include looking for anomalies in audio signals, audio keyword recognition, identifying anomalies in images taken by a drone. All of these are algorithms – AI is but another tool. A powerful tool though, that makes it possible to do things we couldn’t earlier. I believe that AI complements human intelligence, not replace it. In other words, do things that humans cannot do or find too painful (expensive, tedious) to do.

So, what do I do? – I design algorithms and build products around them. That is me, in a nutshell. I call Austin, Texas home, am married and have two wonderful daughters. I like the outdoors – in the pre-Corona days you could find me on Lady Bird Lake at least three evenings every week.

OK, what do you do? Can I help you with your next thing? Do write …

The post Kartik, what do you do? appeared first on Kartik Nanda, Engineering AI.

The main goals:

• Capture images / video using a camera (sensor) connected to the Pi
• AI algo executes on the Pi
• Result is communicated to “Hub” using BT, Wifi etc. Hub could be another computer, IoT Hub, or the Cloud.

Since I have never worked with the Raspberry Pi previously, let’s start at the very beginning.

Hardware:

• Raspberry Pi 4, 4GB version (bought the Canakit Raspberry Pi 4 Starter Kit)

• OV5647 5.0MP Camera Module (bought at dlscorp.com site)
  The OV5647 module is the older version. There is a new official camera module, but I decided to go with the older version for two reasons – cost and 5MP (vs 8MP in V2) is plenty for AI applications. Important details – needs to be a CSI module, and comes with a replaceable lens mount (M12)

• Camera lens (bought different lenses from dlscorp)

• Monitor – used an old monitor I had from years ago. The Pi connects to a monitor using a micro-HDMI port. The kit includes a micro-HDMI to HDMI cable, but I needed to buy a micro-HDMI to VGA adapter for my Monitor.

• Keyboard and mouse – used an old keyboard I had laying around

Software:

• Used the standard NOOBS (preloaded on the micro SD card), and the full Raspbian install. Nothing fancy, yet.

Step 1: Connect the System

There are many excellent walk throughs – see here or here. The basics are simple enough – connect the display, keyboard and mouse, and the power, and follow the onscreen prompts. There are a couple of things I would like to point out though. One, setup SSH and VNC access to the Pi, so you can connect to it from a remote computer and don’t need the dedicated mouse, keyboard and monitor. This will be especially useful during deployment of the application. The second thing – update the software. Use “sudo apt-get update” and “sudo apt-get upgrade”.

Once the system is operational, do take the time to explore, browse the web etc. Marvel at a simple yet very complete computer.

Step 2: Python Setup

There are other languages, of course, but Python is the language of choice. Python2 and Python3 come pre-installed, with a bunch of packages. The first thing is the virtual environment setup – extremely important given the stand-alone nature of the intended application. The application should not break because of an update a couple of years out. I used virtualenv – here is a good intro.

I spent some time (a little) trying to research the “best” editor but realize that the Pi is not necessarily where I will develop the code. That can be done on a laptop or in the cloud. So, for now I am using Thonny, which came pre-installed. I have not yet written a lot of python on the Pi, will see how it goes over the next few days/weeks – might yet change my mind.

Try out a simple program, ensure that python is up and running. In the next post, the camera setup.

The post Setting up the Pi is easy appeared first on Kartik Nanda, Engineering AI.

There is not much online that compares these today, partly because its still early days. There was a panel discussion on this topic at the Texas Wireless Summit (link to the YouTube video). The excellent panelists covered some of the aspects I talk about here.

The table below summarizes some key considerations and suggests which is better – the Cloud or the Edge. Below we examine each, and present our rationale. There is not a mathematical proof, but rather intuitive arguments, with some examples. We consider two examples. First is the facial recognition on mobile devices – we can unlock our phone, the laptop just by looking at it. The device uses its camera to “see” us, and uses an AI model running on the device itself to recognize the face, and unlock. Why does this model sit on the Edge?

The second example is a drone searching for forest fires. Consider two possible implementations. Option 1 – the images are streamed into the Cloud, where the AI model looks for the fire. Option 2 – the AI sits on the drone itself, and the drone then only communicates the location of the fire.

Accuracy

Accuracy is a simple one. By “accuracy” we mean the accuracy of the AI model’s output. The Cloud has more resources, and in general, that translates into higher accuracy. What are these Resources? – compute and memory, so a bigger, more complex model. But it could also be access to other data (sensor fusion, or data from other deployments), or historical data, or other information that is not available at the Edge.

A drone with a model in the Cloud is likely better at detecting fires.

Time

Time – how long before we get the output result – is slightly more complicated. For the same computation, the Edge will be faster than the Cloud. Why? – the Edge is the data generation point. If the model sits in the Cloud, the data has to be uploaded, then inferred, and the results downloaded back to the Edge. However, if the results are not going back to the Edge and instead stay in the Cloud, then time might be the same.

For the phone unlock example, the Edge is a better place for the AI. The action – unlock the phone – is at the Edge.

Reliability

Sending data to the Cloud has a “variable” time aspect as well. Communication depends on many factors, like network availability, signal strength, data routing, traffic. Some are controllable, others harder. In a worst scenario, what if the data does not get through – will it be an issue that can be recovered from?

As an example, if the face recognition unlock on the mobile phone ran in the Cloud, it likely wouldn’t be a feature!

Power

There is a simple rule – the farther the data has to travel, the more power it needs. The most power-hungry part of an IoT (Internet of Things) device is the radio (eg, see Modeling Power Consumption for IoT devices). The energy consumed of course depends on the amount of data sent. As an example, assume a drone looking for a fire. Option one is to send a video stream and process it in the Cloud. Another would be for the AI model to sit on the drone, and for the drone to only send the location of the fire. From the power perspective, the later will be far better.

Cost

Closely tied to power is the cost. This is not obvious, because cost can include many things. However, purely from a cost-of-data angle, streaming the video is more costly than processing it at the Edge (on the drone, in the example above).

The broader question about cost is harder. One aspect is one-time costs Vs recurring costs. While the Edge device is more a one-time cost, the Cloud presents a recurring cost. That’s only one aspect though, and cost has to be evaluated on a product by product basis.

Security

There are two ways to look at security. One is the security of the data. It is most secure, and most easily secured, if it never leaves the Edge. Once the data is on the internet, or in the Cloud, it is only as secure as the encryption/protocols.

The second is securing the Intellectual Property (IP) – the AI model for instance. This is more secure if its in the Cloud.

Privacy

While “security” looks at the problem from the Provider’s perspective (company building the product), “privacy” looks at it from the Consumer’s perspective (person or entity using the product). It is a major concern, and becoming more important every day. Imagine if every time face recognition AI unlocks my phone, my image is uploaded to the Cloud. Then another AI algorithm uses it to “read” my emotional state, and sends suggestions (ads). That is a possibility (easy) with Cloud-based AI, but not as much with Edge-based AI.

The easiest way to keep our personal data personal is to keep it on our devices, and not send it to the Cloud.

So there’s the short list of considerations to keep in mind while planning your AI deployment, and figuring out where it belongs – the Cloud or the Edge. It is not a comprehensive list, and there are many other considerations like time-to-market, managing deployment, cost of building the solution, technology etc that are not covered. They make more sense though if examined within the scope of the AI project. Feel free to reach out with thoughts or if you need help to get started on your project.

The post Edge or Cloud – Where should AI live? appeared first on Kartik Nanda, Engineering AI.

Two things make this all possible – the Internet-of-Things (IoT) and Artificial Intelligence (AI). IoT is the physical interface, the senses. The internet senses the World through the IoT devices. In simpler terms, they collect the data – a camera that takes pictures or a microphone that listens. Or it could be simpler sensors like a thermometer (measuring temperature).

But merely collecting all this data is not useful. (The quantity of this data is huge and increasing.) Something has to interpret and convert this data into actionable information. Something, that can extract meaningful information from this data. That is where Artificial Intelligence (AI) steps in.

Today the AI lives in the cloud (for the most part). IoT collects and sends data to the Cloud, where the AI interprets it, and decides the outcome. Let’s see how this works with examples.

AI and IoT – a marriage made in the Cloud

In the context of IoT, the goal of AI is:

• to learn a model from the data, and,
• use the model to interpret new data and provide an actionable outcome (a decision).

We’ll consider two examples – a voice assistant, like Alexa or Siri, and a smart front door. The Voice Assistant understands human language. For the smart front door, the goal is that the door should recognize me and unlock itself, without me having to use a key. 

Example: The Voice Assisant

The Voice Assistant (VA) responds to spoken commands. It uses a microphone (the IoT sensor) to listen. The AI makes sense of the sounds – the words, sentences and context. It then provides the information or acts on the command. (For example, playing specific music or provide driving directions).

Figure shows how this works. The IoT gathers the data (1) – in the case of the Voice Assistant, the data is sounds. This data is then sent to the Cloud (2). Here the data trains the AI (3) and build a model. The model is then deployed (4).

The model for the VA is complicated. It needs to understand the entire language, spoken with many different accents. Once trained though, the same model applies to all the VAs in the network.

For the Voice Assistant, the model (AI) learns and is deployed in the cloud itself.
The AI receives and interprets all spoken words in the Cloud (Cloud-AI). Why keep it in the Cloud? Mostly because it is a large model. It is also being constantly updated, and keeping it in the Cloud means only one model needs to be updated. The main issue is the lack of privacy. Anything I say, the VA is listening to and sending to the Cloud where it may be interpreted.

Example: The Smart Front Door

The goal with the smart Front Door is it recognizes me and responds by unlocking the door for me, and only me. It needs to sense my presence, and let’s say it uses a camera (the IoT sensor). AI provides the intelligence. It learns what I look like and then uses that model to recognize me and open the door.

The Front Door system design may follow the same approach as the VA. The camera (IoT) gathers the data, which is sent to the Cloud. There the AI learns to recognize me. But here things become different. The AI has to recognize my face for my door, and my neighbor’s face for their door. That means the AI trains differently for different doors.

This is a critical difference – in the VA case, the same AI (one trained model) could serve all IoT devices in the network. But in the Front Door example, every door has a unique model.

So for the Front Door, it might make sense to instead deploy the AI in the IoT device itself (IoT-Edge), rather than the Cloud. The Front door only needs to decide Me-Vs-NotMe. So while the model may train in the Cloud (esp. the Not-Me part), it is deployed at the IoT-Edge.

There are other good reasons for the different approaches (Cloud AI Vs Edge AI). Two of the main ones are privacy, and the size of the model. Large, typically complicated models can not fit on the resource-limited IoT devices. But, I also do not want a digital representation of my face stored on the cloud.

Heterogenous IoT Networks

We looked at two examples above – the VA and front door – both using single IoT devices. In general though, IoT networks consist of many devices. They are also not always homogenous, that is, not all the attached systems are identical. They may have some common features, but some unique ones as well. Think of a factory floor or even the car – both have many different types of sensors (IoT sense devices).

This complicates the AI. It is no longer possible to train the AI on one IoT device, and map the model to all the IoT devices in the network.

To illustrate this, imagine if my front door now has a fingerprint sensor in addition to the camera. The AI for the camera cannot map to the finger print sensor. Yet the two have to work together on the same problem, namely to recognize me and unlock the door.

The figure shows a general scenario – it depicts two different IoT sensors. They both have some common features, and some unique ones. One solution is to learn a superset of all possible features in the cloud. The result is a large, complicated model. This needs far more resources in the IoT edge devices, or compromises on privacy by keeping the AI in the Cloud (figure below, left).

A Distributed AI Model (Edge-AI)

Another solution is a Distributed AI model. This implements the AI on the network (above figure, right) – which includes the Cloud and the different IoT devices. Parts of the solution – the common features – are mapped to the Cloud. The unique ones are modeled and learnt locally on the IoT sensor devices.

This provides both benefits – privacy and resource management. Since the model is local to the IoT edge device, private data is not sent or stored in the cloud. This also utilizes the available resources better. The IoT devices store only local models, and not the entire systems’ model. The result – a smaller footprint at each IoT device. Much of the AI now resides on the network devices. This reduces the computations and data requirements in the cloud.

This is a Distributed AI solution (or Edge-AI). As IoT networks become more prevalent, we will start to see more AI move to the Edge. Lets illustrate with an example next.

Example – Solar Monitoring with Distributed AI

The past few years have seen a tremendous growth in rooftop solar plants. Many residences today generate their own electricity from the Sun. Here’s a common question that comes up all too often from owners:

“I have installed a rooftop solar plant for electricity. How do I know if the energy output from my plant is actually correct, as designed?”

Solar Customer

Data there is plenty of – solar plants provide daily energy output data. The problem is this energy output number changes from day to day, based on many factors like the weather. What is lacking (and what the owner is looking for) is information. Is the output correct?

AI and Solar Monitoring 

The goal with AI, is to learn a model of the Plant, and to monitor it by comparing actual output against the predicted output.

One solution is to model the plant in the Cloud (the Cloud-AI solution). The first step is to take all the data into the cloud. The second step is to learn a model of the plant – certain plant specific details such as the shade profile. The third step is to predict the output under the given environmental conditions. The final step is comparing the actual output from the plant and the predicted output. The result is actionable information – is the plant output good or not.

This solution works, but has two issues. The first is privacy – all the data goes to the Cloud. The second is scalability. Imagine learning the models for hundreds of thousands of small residential solar plants in the Cloud. A better solution: if the models could be learnt locally in the IoT device themselves.

Edge-AI solves both of these issues. The AI solution maps on to the entire network (see figure). Certain elements that are common to all the plants – the weather for example – are modeled in the Cloud. The Plant specific modeling however, is local.

The result is better on many fronts. One, the distributed model uses resources more optimally by processing data locally. Data privacy is much better since all plant modeling is local. The network scales easily. Adding a new plant is easy since every plant learns its own model, and only its own model. 

Closing Thoughts

Products and solutions based on a network of IoT sensors are set to become prevalent. Some applications include the electricity smart-Grid, a smart-home, individualized health care. AI will be an integral part of any such product. And while today Cloud-based solutions dominate the AI landscape, such AI-IoT network products will need new distributed topologies.

Edge-AI is a distributed AI topology. It maps the AI on the entire network, not just in the Cloud. This provides two distinct advantages over AI in the Cloud. The first is privacy – with models that are learnt locally. Second is the lean implementation, an optimal use of the available resources.

Distributed AI will drive the next wave of Intelligent Products.

The post AI and IoT – Marriage made in the Cloud appeared first on Kartik Nanda, Engineering AI.