A Novel Architecture that uses the Power of SNN in Combination with Transfer Learning for Real-time Object Detection

Sanaullah1*, Shamini Koravuna2, Ulrich Rückert2, Thorsten Jungeblut1
* Corresponding Author: Sanaullah

1Industrial Internet of Things, Hochschule Bielefeld    2Cognitronics & Sensor Systems, Universität Bielefeld

Paper (under-review)

Code

Abstract


This paper presents a novel architecture that combines Spiking Neural Networks (SNN) with transfer learning for real-time human presence detection using event-based cameras. The architecture, deployed on edge computing devices (i.e. NVIDIA Jetson Xavier NX), integrates object detection, transfer learning with SNN, human recognition, localization, tracking, feature extraction, multi-core processing, and real-time analysis. It efficiently adapts pre-trained Convolutional Neural Network (CNN) weights to SNNs, allowing event-driven processing, and maintains a spike train dataset for object information. The architecture is adaptable for applications in security, surveillance, and behavioral research. Extensive real-time testing demonstrates its robustness and adaptability in dynamic environments.

Proposed Architecture INTRODUCTION

The proposed architecture seamlessly integrates detection, enhanced object recognition through transfer learning, and the capabilities of SNNs. Because it gives immediate data regarding human presence and activity in a real-time environment, this application is powerfully dynamic and adaptive. Its applicability extends across a wide range of use cases, including security, surveillance, and cutting-edge behavioral research. Therefore, this full integration of the proposed architecture is the foundation of our novel solution.

YouTube Source




Method Overview

method_overview

The execution methodology of the proposed neuromorphic computing application. The execution structure represents the transfer of learning from a CNN to a Spiking Model of the LIF Neural Model. Real-time implementation is utilized for object detection with new data, which is then stored in a Spike Train for future predictions using the Spiking Model.


Qualitative Results

Spike Vision Approach for Generating Large-Scale Spiking Dataset


input

* This 3D plot offers a visual representation of detected humans along with their corresponding spike trains.
* The right image illustrates the data of the first 25 objects along with 0-75 objects,
* And the right 3d plot provides an overview of the data from first 100 detected objects.




* This plot offers a visual representation of spike trains dataset based on the activity of multiple neurons over-time.
* Neurons are represented along the y-axis, and time is along the x-axis.
* Each spike (i.e., action potential) of a neuron is marked as a dot at its corresponding time of occurrence.


Computational Cost Efficiency


input

* Computational cost efficiency comparison of the system between object detection and no object detection.
* The system’s computational cost remains at a minimum state during periods of inactivity.
* Object detection, on the other hand, raises the computing cost, as indicated by ’Object Detected’.



Citation



	@inproceedings{sanaullah2024,
  title={A Spike Vision Approach for Multi-Object Detection and Generating Dataset Using Multi-Core Architecture on Edge Device},
  author={Sanaullah and Koravuna, Shamini and R{\"u}ckert, Ulrich and Jungeblut, Thorsten},
  booktitle={International conference on engineering applications of neural networks},
  pages={--},
  year={2024},
  organization={Springer}
}