Real-Time Edge Face Detection Using a Lightweight Multi-Task Cascaded Convolutional Neural Network with Hardware-Software Co-Design

Detection-Oriented Lightweight Redesign

3.1.

The proposed framework preserves the original P-Net/R-Net/O-Net cascade structure while redesigning each stage to reduce model complexity. In the deployed lightweight configuration, the optimized O-Net performs only final face classification and bounding-box regression, without the landmark-prediction branch of the original MTCNN. This modification is more consistent with the target application of real-time face detection and cropped-face extraction, while also reducing parameter count and inference cost.

To enable efficient edge deployment, the original cascade is redesigned through filter reduction, the replacement of selected standard convolutions with depthwise-separable convolutions, and SE-based channel recalibration. In P-Net, the first convolution is reduced from 10 to 8 filters, followed by depthwise-separable blocks with 16 and 32 output channels. In R-Net, the early convolutional widths are reduced from 28 and 48 to 16 and 32 channels, respectively, and the subsequent feature-extraction path is reorganized using depthwise-separable convolutions. In O-Net, the first convolution is reduced from 32 to 16 filters, and the subsequent feature extractor uses depthwise-separable blocks with 32, 48, and 64 output channels. SE blocks are introduced in all three subnetworks to mitigate the representational loss caused by structural compression.

Assuming identical output feature-map dimensions and neglecting bias operations and implementation overhead, the ratio of multiply-accumulate operations required by depthwise-separable convolution to those of standard convolution, with a $K \times K$ kernel, $M$ input channels, and $N$ output channels, is given by (2):

This factorization decomposes the original $K^{2}MN$ coupling into a depthwise spatial term and a pointwise channel-mixing term, thereby reducing multiply-accumulate cost while preserving the effective receptive field.Fig. 2 illustrates the lightweight P-Net architecture used in the proposed cascade. A 12×12×3 input is processed by a convolutional stem and two depthwise-separable convolution blocks, after which the feature tensor is recalibrated by an SE branch before entering the classification and bounding-box prediction heads.

Fig. 2. Proposed lightweight P-Net architecture with depthwise-separable convolution and SE channel recalibration.

SE-Based Channel Recalibration and INT8-Oriented Deployment

3.2.

SE blocks are introduced to compensate for the representational loss caused by aggressive slimming [12]. Let $U \in R^{H \times W \times C\ }$ be the output tensor of a compact feature block. The SE transformation is defined as

where $z$ is the $C$-dimensional channel descriptor obtained by global average pooling, $\delta( \cdot )$ denotes the Rectified Linear Unit (ReLU) activation function, and $\sigma( \cdot )$ represents the sigmoid activation function [12]. The matrices $W_{1}$ and $W_{2}$ correspond to the two fully connected layers of the excitation mechanism, with dimensions $(C/r) \times C$ and $C \times (C/r)$, respectively, where $r$ denotes the channel-reduction ratio. Equation (3) formalizes the role of SE in this work: it adaptively reweights channels after compression so that compact layers retain useful channel selectivity. InFig. 2, this process is represented by the squeeze path, bottleneck excitation, sigmoid gating, and channel-wise scaling preceding the output heads. In the proposed design, SE acts as an accuracy-recovery mechanism rather than as a compression method. This role is important because feature degradation in P-Net can propagate through the cascade and reduce proposal quality in later stages.

The optimized model is prepared for an INT8-oriented inference path to reduce storage cost, memory traffic, and arithmetic complexity on the embedded target. Using uniform affine quantization,

where $x$ denotes the original real-valued tensor, $x_{q}$ represents its quantized integer representation, and $\widehat{x}$ denotes the reconstructed approximation obtained through dequantization. The parameter $s_{x} > 0$ is the quantization scale, $z_{x}$ is the integer zero-point, and $q_{\min}$ and $q_{\max}$ define the permissible quantization range. The rounding operator maps values to the nearest integer, while the clipping function restricts quantized values to the supported integer range. The deployment workflow uses uniform affine quantization, where the real-valued tensor is represented by an INT8 tensor together with a scale factor and a zero-point.

In this study, post-training quantization (PTQ) is used as the default quantization strategy for deployment preparation, whereas quantization-aware training (QAT) is considered only when PTQ causes unacceptable performance degradation. Unless otherwise stated, the reported parameter counts refer to the structurally optimized lightweight MTCNN architecture. Post-training quantization (PTQ) changes the numerical representation of model weights and activations without altering the number of model parameters. The KV260 latency and throughput results presented in Section 4 correspond to the current HLS-based hardware implementation and are reported separately from the model-level quantization preparation. This distinction clearly separates model-level quantization readiness from the arithmetic mode of the current hardware implementation.

Taken together, (1)-(4) define the algorithmic basis of the proposed lightweight MTCNN. Equation (1) defines the image-pyramid scaling process, whose scaling factor controls the number of processed pyramid levels, Equation (2) lowers convolutional complexity, Equation (3) restores channel discrimination after slimming, and Equation (4) supports low-precision deployment. This combination is well suited to real-time edge face detection because it addresses the major bottlenecks jointly rather than optimizing only the network backbone.

Hardware-Software Co-Design on the AMD Kria KV260

3.3.

Fig. 3 shows the deployment of the optimized MTCNN cascade on the AMD Kria KV260, which is based on the Zynq UltraScale+ multiprocessor system-on-chip (MPSoC). Input video frames are streamed to the platform, where the customized three-stage cascade performs face detection before forwarding the detected face regions to downstream processing.

Fig. 3. System architecture of the proposed face detection framework on the Kria KV260 Vision Kit.

At the system level, the Processing System (PS) handles frame I/O, control scheduling, memory management, image resizing, image-pyramid generation, inter-stage proposal cropping and resizing, and post-processing tasks such as NMS, whereas the Programmable Logic (PL) implements P-Net, R-Net, and O-Net as dedicated Vitis HLS kernels. The PS and PL communicate through AXI4 over shared double data rate (DDR) memory, thereby keeping control-intensive tasks in software and offloading convolution-dominant computation to hardware.

This design is consistent with prior FPGA-based inference architectures, in which throughput depends on the coordinated optimization of computation and data movement. In the PL, pipelining, loop unrolling, and dataflow increase concurrency, whereas FIFO stream buffers allow adjacent stages to overlap at the cost of additional BRAM usage. Because P-Net is compact, its weights can be stored on-chip, whereas the larger R-Net and O-Net weights must be fetched from external DDR, making memory bandwidth and transfer scheduling critical to end-to-end performance.

Accordingly, the stage-level latency can be approximated by

where $T$_t,comp is the compute time, $T_t$,_mem is the memory-transfer time, and $T_t$,_ctrl is the control overhead associated with PS-side orchestration. Equation (5) shows that arithmetic reduction alone is insufficient for real-time edge deployment. Even when the network is structurally compressed, the achievable system throughput remains bounded by memory movement and control overhead if these terms are not concurrently optimized. This observation is fully aligned with the study’s target problem: real-time embedded face detection is governed jointly by algorithmic efficiency and system-level dataflow behavior.

Experimental Setup

4.1.

This section evaluates the framework at both model and system levels for real-time edge deployment. At the model level, the evaluation assesses whether the lightweight design and INT8-oriented optimization reduce computational and memory cost while preserving competitive detection accuracy. The lightweight MTCNN is also compared with the original MTCNN on the WIDER FACE validation set, a standard face detection benchmark with large variations in scale, pose, occlusion, and illumination [23]. The evaluation uses two datasets: the WIDER FACE validation set for comparison with the original MTCNN and prior detectors, and an in-house dataset representing the target edge deployment setting. The in-house dataset contains over 8,000 annotated images with more than 15,000 face instances and five-point landmark labels; however, in the present detection-oriented deployment, the lightweight model is trained and evaluated only for face classification and bounding-box regression. Training samples follow the standard MTCNN three-class strategy: intersection over union (IoU) < 0.3 for negatives, 0.4 ≤ IoU < 0.65 for part samples, and IoU ≥ 0.65 for positives. The dataset is split into 80% training, 15% validation, and 5% testing. Images are normalized as ($x$−127.5)×0.0078125, with a batch size of 32 and 4 workers. Weights are initialized with Xavier uniform, biases with 0.1, and training uses Adam. Each customized subnetwork is trained for 29 epochs, with the best checkpoint chosen from validation results. During inference, cascade thresholds are set to 0.6/0.6/0.7 for P-Net/R-Net/O-Net, and the image-pyramid scaling factor is 0.707. These values are used as the evaluation settings in this study.

At the system level, the optimized cascaded detector is deployed on the AMD Kria KV260 platform, where P-Net, R-Net, and O-Net are implemented as Vitis HLS kernels and connected through AXI-based communication between the processing system and programmable logic. For comparison, the same optimized model is also evaluated on an Intel i7 CPU and an NVIDIA GTX 1060 GPU, helping distinguish gains from model optimization and hardware acceleration. Deployment feasibility is further assessed through post-implementation hardware utilization and power analysis, including look-up table (LUT), flip-flop (FF), digital signal processing slice (DSP) and RAM usage, with power estimated using Vivado tools. Kernel-level latency is also measured for each cascade stage to show the computational distribution across P-Net, R-Net, and O-Net. Together, these results evaluate detection accuracy, model compactness, throughput, resource utilization, and power consumption, with the settings and metrics summarized inTable 1.

Model-Level Results

4.2.

Fig. 4 shows that the proposed lightweight MTCNN improves model compactness while maintaining effective detection performance. As shown inFig. 4(a), the redesigned cascade reduces the parameter count from 6,632 to 1,846 in P-Net, from 100,178 to 79,174 in R-Net, and from 389,040 to 116,182 in O-Net, with the largest reduction in O-Net. As a result, the total parameter count decreases from 495,850 to 197,202.Fig. 4(b) shows that this compression causes only a modest drop in detection performance. Although stage-wise classification accuracy decreases after optimization, for example, from 98.4% to 94.8% for O-Net, the proposed model still achieves 96.3% AP under the WIDER FACE validation protocol, compared with 97.8% AP for the original MTCNN. In addition, increasing the image-pyramid scaling factor from 0.6 to 0.707 produces a more densely sampled image pyramid and may increase the number of pyramid levels and the computational workload of P-Net, depending on the input-image dimensions and stopping criterion. Overall, these results indicate that the proposed refinements, including depthwise-separable convolutions, substantially reduce model complexity and memory overhead with only limited impact on detection capability.

Fig. 4. Comparison of the original and optimized MTCNN across cascade stages. (a) Parameter counts for P-Net, R-Net, and O-Net. (b) Stage-wise accuracy, showing a modest performance reduction after optimization.

Fig. 5 provides a qualitative comparison of the detector with and without the SE module. The SE-enabled configuration produces cleaner detections and more stable bounding-box placement across different scenes. These qualitative results suggest that SE helps preserve feature selectivity after model compression. In the non-SE configuration, shown on the left, the three-stage cascade follows the standard MTCNN progression from dense P-Net candidate proposals to R-Net refinement and the final O-Net output. As illustrated, the early stage generates multiple candidate boxes, including several redundant or off-target responses, which are only gradually suppressed in the later stages. By contrast, the SE-enabled results shown on the right exhibit cleaner detections and more stable bounding-box placement across images with different viewpoints, backgrounds, and scene compositions. This qualitative difference indicates that SE helps preserve channel-wise feature selectivity after structural slimming with depthwise-separable convolutions, thereby improving proposal quality in the early cascade stages and reducing error propagation to downstream refinement. Overall, the figure provides visual evidence that the addition of SE helps the compressed detector maintain reliable face localization while remaining suitable for real-time edge-oriented deployment.

Fig. 5. Qualitative comparison of face-detection results with and without the SE module in the proposed lightweight MTCNN accelerator. Left: non-SE cascade outputs; right: SE-enabled outputs with more stable face localization.

System-Level Results on the KV260

4.3.

While the previous results demonstrate the proposed system’s advantages at the model level, the following analysis examines its system-level behavior to evaluate deployment feasibility on the target edge platform. As shown inFig. 6(a), the accelerator is implemented in Vivado as a Zynq UltraScale+ MPSoC design, where the P-Net, R-Net, and O-Net kernels are connected to the processing system through AXI interconnect and SmartConnect.Fig. 6(b) shows that the design fits within the target device resources, although it is memory-intensive, using 126.5/144 BRAMs, 502/1,248 DSPs, 91,292/117,120 LUTs, 115,497/234,240 FFs, and 16/64 URAMs.Fig. 6(c) reports an estimated total on-chip power of 3.853 W, including 3.544 W dynamic and 0.309 W static power. Most dynamic power is consumed by the processing system at 2.419 W, while the programmable logic uses 1.125 W.

Table 2 provides the stage-wise hardware resource utilization of the optimized kernels. P-Net is the lightest stage, whereas O-Net is the most memory-intensive. Specifically, P-Net uses 24,312 LUTs, 26,764 FFs, 5 BRAMs, and 162 DSPs. R-Net uses 28,959 LUTs, 33,700 FFs, 35.5 BRAMs, and 191 DSPs. O-Net uses 25,608 LUTs, 37,178 FFs, 86 BRAMs, and 149 DSPs. The integrated PS-PL system uses 91,292 LUTs, 115,497 FFs, 126.5 BRAMs, and 502 DSPs. These system-level values are not expected to equal the simple sum of the three kernel-level values because the complete design also includes AXI interconnect, PS-PL control logic, memory interfaces, and other integration overhead. Overall, the results confirm that the proposed PS-PL integration is feasible on the KV260, although interconnect and control overhead must be considered.

Fig. 6. Integrated design and implementation results of the proposed MTCNN accelerator (a) Vivado block design showing the MPSoC, P-Net, R-Net, and O-Net kernels, and the AXI interconnect. (b) Overall hardware resource utilization of the integrated PS-PL implementation (c) Vivado-estimated on-chip power breakdown.

Beyond the model-level gains, the results further show that the proposed hardware-software co-design translates these improvements into practical runtime benefits at the system level. On the AMD Kria KV260/ARM platform, the complete implementation sustains 30 FPS as an integrated end-to-end deployment result for the cascaded detector. This figure should be interpreted as system-level throughput, including preprocessing, inter-stage data movement, kernel invocation, and post-processing, rather than as the latency of an isolated CNN kernel. For reference, the same optimized model achieves approximately 50 FPS on an Intel Core i7 CPU and about 70 FPS on an NVIDIA GTX 1060 GPU, confirming that the redesigned network remains efficient across heterogeneous platforms.

A more detailed view is provided inTable 3, which shows that the impact of the proposed hardware optimization is not uniform across the cascade but instead increases with stage complexity. For clarity, the hardware acceleration factors reported inTable 3 are measured relative to the corresponding unoptimized FPGA implementation, not relative to CPU software execution. Accordingly, the reported speedups of 1.43×, 5.00×, and 101.56× for P-Net, R-Net, and O-Net, respectively, are computed from the ratio between the original FPGA latency and the optimized FPGA latency of each stage. In particular, the 101.56× speedup for O-Net corresponds to the reduction from 358 ms to 3.525 ms on FPGA, whereas the CPU latency values are included only for runtime reference. This pattern suggests that the proposed architectural refinements and FPGA-level parallel execution strategies yield only modest benefits for lightweight stages, whereas their effect becomes much more significant as the computational burden increases.

This stage-dependent behavior is also reflected in the latency results. For P-Net, the original FPGA implementation already outperforms the CPU baseline, reducing latency from 1.2 ms to 0.050 ms, while the optimized version provides only a small further improvement to 0.035 ms. In contrast, R-Net is initially slower on the unoptimized FPGA design, with a latency of 4.047 ms versus 1.5 ms on the CPU, but optimization reduces it to 0.810 ms, allowing the FPGA to surpass the CPU. The largest improvement is seen in O-Net, where latency drops sharply from 358 ms to 3.525 ms. Although this is still slightly higher than the CPU runtime of 2.6 ms, it significantly reduces the main bottleneck in the cascade. This is especially important for end-to-end deployment, since accelerating the slowest stage has the strongest effect on real-time feasibility. However, these values represent standalone kernel latency, so total system latency will be higher once data transfer and integration overhead are included.

Contextual Comparison with Representative Detectors

4.4.

Table 4 compares representative face detectors in terms of accuracy and average throughput under the reported edge-oriented setting. Among the classical baselines, Haar Cascade achieves approximately 72% accuracy at 25 FPS on the CPU, whereas DLIB HOG improves accuracy to approximately 82% but runs more slowly at approximately 20 FPS [24], [25]. RetinaFace provides a stronger deep-learning baseline, reaching approximately 96% accuracy, although only a CPU throughput of 20 FPS is reported in this comparison [26]. In this study, the original MTCNN achieves the highest detection accuracy of 97.8%. The standalone unoptimized FPGA kernel latencies reported inTable 3 sum to 362.097 ms (0.050 + 4.047 + 358 ms). This calculated value is not a measured end-to-end frame rate, because it does not account for processing across image-pyramid levels, multiple candidate proposals, data transfers, or other system-level overhead. The original MTCNN throughput on the KV260 is therefore not reported as an end-to-end FPS measurement. The original model achieves a reported CPU throughput of 5.3 FPS. By contrast, the optimized MTCNN maintains near-baseline accuracy at 96.3% while increasing throughput to 30 FPS on the KV260, 50 FPS on the CPU, and 70 FPS on the GPU. Overall, the results reflect the expected accuracy-efficiency trade-off: classical lightweight methods are faster but less accurate, whereas deep-learning methods are generally more accurate but more computationally expensive. However, this comparison is contextual only, because the reported values were obtained using different datasets, hardware platforms, and evaluation protocols.