Case Studies

Enterprise AI Operating System

Q: How does the platform solve domestic chip adaptation issues?

We built an Abstraction Layer (HAL) with deep operator optimization for various mainstream domestic chips. Applications can migrate transparently without code changes, achieving "develop once, run anywhere".

Q: What are the advantages of Model Mesh over traditional gateways?

Traditional gateways only handle north-south traffic. Model Mesh governs east-west traffic, supporting traffic mirroring (shadow testing), weighted canary releases, and model-specific circuit breaking for complex microservices.

Q: Does the platform support public cloud LLM integration?

Yes. Via "Unified Model Proxying", the platform manages both local private models and various mainstream public cloud LLM APIs. Applications use a unified interface, while the platform handles routing to ensure "Model Neutrality".

Provincial AI Computing Power Dispatch Center

A unified AI orchestration platform for a provincial computing infrastructure, achieving centralized management of "hundreds of models" and resource pooling across heterogeneous chips.

Key Metrics

Project Results

Managed Models

Covers full-stack AI including LLM, CV, and NLP

Utilization

Significantly improved compute utilization via dynamic peak-shaving

0.9%

SLA

Enterprise-grade high availability ensuring continuous operation

Comprehensive

Unified Control

Successfully broke AI silos across the regional infrastructure

Core Technology Features

Technical Highlights

Heterogeneous Pooling

Breaks chip barriers, enabling mixed scheduling of localized and general-purpose compute

Model Service Mesh

Istio-based microservice governance for fine-grained traffic control and second-level recovery

AI Security Gateway

Built-in Prompt injection defense and data de-identification for AI-era security

Dynamic Admission

Data-driven automated model evaluation helping select appropriate models for deployment

Project Overview

Client Background

This Provincial AI Computing Power Dispatch Center coordinates regional resources for government, research, and public AI applications. With rising localization requirements, the center faced a complex mix of general GPUs and domestic AI chips. Legacy siloed architectures prevented resource pooling and cross-chip model migration, creating an urgent need for an AI Operating System to shield hardware differences.

Technology Stack

KubernetesIstiovGPUPrometheusOpenTelemetry

Transformation

From Challenges to Solutions

Before

After

Transformation

1Compute silos: Heterogeneous chips (GPU/NPU) couldn't be scheduled together, leading to expensive resources staying under 20% utilized

Developed an Heterogeneous Compute Virtualization Engine to shield chip differences, achieving unified pooling and scheduling of general and domestic chips

1Compute silos: Heterogeneous chips (GPU/NPU) couldn't be scheduled together, leading to expensive resources staying under 20% utilized

Developed an Heterogeneous Compute Virtualization Engine to shield chip differences, achieving unified pooling and scheduling of general and domestic chips

2Migration barriers: Differing drivers and frameworks across vendors made the cost of migrating models across chips extremely high

Built a Model Service Mesh providing intelligent traffic routing and enabling smooth migration and backup across localized chips

2Migration barriers: Differing drivers and frameworks across vendors made the cost of migrating models across chips extremely high

Built a Model Service Mesh providing intelligent traffic routing and enabling smooth migration and backup across localized chips

3Service governance gaps: Lack of unified traffic orchestration and circuit breaking led to poor stability for model services under peak loads

Established an AI Application Security Gateway integrating full-link monitoring and content risk plugins to provide standardized secure inference APIs

3Service governance gaps: Lack of unified traffic orchestration and circuit breaking led to poor stability for model services under peak loads

Established an AI Application Security Gateway integrating full-link monitoring and content risk plugins to provide standardized secure inference APIs

4Security admission ambiguity: Massive model access lacked unified security auditing and compliance risk control, posing data and content risks

Created an Automated Evaluation Pipeline for dynamic performance assessment of models, ensuring precise allocation of compute resources

4Security admission ambiguity: Massive model access lacked unified security auditing and compliance risk control, posing data and content risks

Created an Automated Evaluation Pipeline for dynamic performance assessment of models, ensuring precise allocation of compute resources

Technical Architecture

System Architecture Design

Layer 1

Resource Abstraction Layer

Abstracting underlying chip differences (General/Domestic Chips) for unified resource pooling and scheduling

Heterogeneous MgmtPoolingAuto-scalingvGPU

Layer 2

Model Service Mesh

Service Mesh based model traffic governance supporting A/B testing, canary release, and circuit breaking

RoutingCircuit BreakingCanaryOrchestration

Layer 3

AI Application Gateway

Enterprise-grade unified API access with auth, rate limiting, billing, and full-link observability

Unified APIAuthObservabilityBilling

异构算力调度→模型服务网格→全链路观测→资源配额控制

Implementation Timeline

Phased Implementation

Phase 1

Infrastructure Pooling

Completed unified access and virtualization of GPU/NPU resources; established heterogeneous scheduling foundation

Phase 2

Service Governance Launch

Deployed Model Mesh to take over regional AI traffic, achieving multi-tenant isolation and dynamic rate limiting

Phase 3

Ecosystem Opening

Launched the AI App Gateway and Developer Center to support one-stop AI capability invocation for all provincial agencies

Phase 1

Infrastructure Pooling

Completed unified access and virtualization of GPU/NPU resources; established heterogeneous scheduling foundation

Phase 2

Service Governance Launch

Deployed Model Mesh to take over regional AI traffic, achieving multi-tenant isolation and dynamic rate limiting

Phase 3

Ecosystem Opening

Launched the AI App Gateway and Developer Center to support one-stop AI capability invocation for all provincial agencies

Testimonial

Client Testimonial

“This platform solved our urgent need of "having compute but failing to schedule". It not only shielded the complexity of different chips but also nearly doubled our resource utilization, truly achieving centralized regional management.”
Center Chief Engineer
Provincial Digital Transformation Expert

FAQ

Frequently Asked Questions

How does the platform solve domestic chip adaptation issues?

What are the advantages of Model Mesh over traditional gateways?

Does the platform support public cloud LLM integration?

Related Cases

Explore More Cases

Clinical Decision Support & Precision Care System

Large Tertiary Hospital

A clinical decision support system for large medical institutions, merging EMR/LIS/PACS multi-modal data to achieve assisted diagnosis and whole-course disease management.

View Case Study

Enterprise CLM & Compliance Platform

Large Group Enterprise

A full-lifecycle contract management and risk control platform for a large enterprise group, replacing SaaS with private deployment to ensure data sovereignty and cost efficiency.

View Case Study

Medical Policy Intelligence & Compliance Platform

Large Research-based Tertiary Hospital Group

A policy and compliance base for multi-campus medical groups, providing deep AI interpretation of medical insurance (DRG/DIP), health, and drug regulations to achieve closed-loop management from policy issuance to clinical execution.

View Case Study

Back to Case Studies

Want Similar Results?

Let's discuss how we can achieve similar success for your organization.