Just because a prototype AI application works fine on a test server doesn’t mean it will behave the same once it goes live. Once launched as a production service, not only will it need predictable availability, effective security and monitoring; it will also require enough capacity to handle simultaneous requests, stored files and background tasks. In this post, we explain why the infrastructure requirements for AI application hosting depend on how the application uses models and APIs, as well as its expected traffic and workload.
Contents
Start with the AI workload
When assessing an application’s hosting requirements, businesses and developers first need to establish where the AI processing will take place. If an application calls a third-party AI service via an API, it will usually need hosting for its own code, database and supporting tasks. The resources for generating responses, however, will be supplied by the third party that runs the model.
If a model runs locally, the hosting requirements can change significantly. In this scenario, the server will need enough processing power, memory and storage to run the model and handle all the requests. This can also have implications for the type of hardware needed, as some workloads may require GPUs.
For this reason, hardware and software compatibility will need checking before a hosting plan is chosen. To better determine which environment is needed, testing should use realistic file sizes, a credible number of simultaneous users and acceptable response times. These criteria are more important than headline server specifications.
VPS, cloud or dedicated hosting?
VPS, cloud hosting and dedicated servers are all potential options for hosting AI applications. A VPS offers a predictable and controlled environment that can be suitable for many application and API workloads. However, developers should test capacity requirements, including the resources used by databases and background tasks, before choosing a platform.
Cloud hosting can be useful for businesses expecting variations in demand or when different parts of the application need to grow independently. However, cloud hosting for AI apps does not automatically provide scaling or resilience. Whether either is possible depends on the application’s design, service configuration and management arrangements.
For AI applications with consistently high resource demands or specialist configuration requirements, a dedicated server can be an appropriate fit. However, businesses should carry out a hardware assessment, as some specialist model workloads may require a suitable GPU and not all standard dedicated servers include one.
It is important to note that even if a business runs both the model and application, they can be hosted on separate infrastructure and communicate via an internal API. For instance, they could run on separate cloud instances, or the application and database could be hosted on a VPS, with the AI model running on a dedicated server with more resources, a suitable GPU or both.
Production requirements that are easy to overlook
Monitoring is an essential production requirement, as it helps businesses establish whether their AI service is working effectively. This should include monitoring response times, failed requests, database performance and jobs waiting to run. To enable a rapid response to issues, alerts should be directed to those who can investigate.
For more information, read our article Application Monitoring and Its Benefits for Businesses
Another area needing consistent attention is storage. Uploaded documents, logs, and generated files can quickly accumulate, while sluggish database queries can delay responses even when CPU capacity is available. To reduce storage issues, businesses should set effective retention rules and conduct regular reviews.
When it comes to security, it is vital that businesses know who owns the various responsibilities. This includes internal staff and developers, as well as third-party AI providers and the hosting provider. These responsibilities should cover operating system patches, application dependencies and access to customer data. Businesses should also ensure that API keys are kept in protected server settings or a secrets management service, and that access is strictly limited to those who need it.
Backups are another area that shouldn’t be overlooked. Besides covering the data and configuration needed to restore the service, the restore process itself should be tested. Furthermore, businesses should plan for failures involving external dependencies, as an unavailable API or network connection can cause major disruption even if the application’s own hosting is healthy.
Scaling AI applications safely
Before adding more resources, businesses should first establish and measure the actual bottleneck. Is the constraint due to processing power, memory, database performance, background jobs or an external service? Only when this is known can the right fix be applied.
There are two potential scaling options. Vertical scaling adds resources to an existing server, while horizontal scaling adds more instances. With the latter, the application must be designed to work across those instances. In addition, businesses can separate the application, database and background processing, which can be helpful when one workload regularly holds up another.
Unexpected traffic bursts can prevent an application from processing all the requests at once. To help avoid issues, there should be limits on how much work can run simultaneously and practical rules for retrying delayed or unsuccessful requests.
Moreover, if an external API sets rate limits on how many requests it accepts within a given timeframe, work may need to be queued and retried later. When doing this, any changes should be tested with realistic levels of demand, and their impact on performance and cost should be reviewed.
Who manages the infrastructure?
While managed infrastructure can reduce the admin burden when hosting AI applications, the responsibilities need to be agreed before launch. For instance, the hosting provider may maintain the operating system, monitor server services and investigate infrastructure faults, while the development team remains responsible for the application.
External AI or API providers are responsible for operating their service within their specific terms and limits. Prior to launch, businesses therefore need to agree how any problems will be escalated between the hosting provider, development team and AI or API provider.
Aside from recording these arrangements, businesses should ensure the support plan identifies who will investigate first, who can authorise changes and how any unresolved problems move between the various parties.
AI application hosting: production checklist
Before launching, the business and its technical team should confirm the following:
Conclusion
Businesses considering AI application hosting should prioritise the demands of the application, its users and the model services it relies on. To help ensure the right infrastructure is chosen, companies should test workloads, agree clear maintenance responsibilities and ensure a workable recovery plan is in place.
To discuss the most suitable setup for your AI application, speak to the eukhost team about our managed VPS hosting, cloud hosting or dedicated servers.
