Open Telekom Cloud

Ollama

Gpu

Openvpn

Remote Development

Securely Expose Remote Ollama Endpoints to your Development Machine

Expose Ollama endpoints directly from Open Telekom Cloud to your local development machine.

Exposing Ollama endpoints directly from your Open Telekom Cloud tenant environment to your local development machine can be highly beneficial, especially when it comes to optimising the use of expensive resources like GPUs and integrating them with local cost-effective development hardware.

The benefits of integrating remote Ollama endpoints to local development workflows are multi-faceted but not limited to:

• Enhanced Development Workflow : By securely exposing Ollama endpoints locally, you can streamline the development process. This setup allows you to test changes in real-time without deploying them in a public or external environment or requiring expensive development machines; for that matter accelerating iteration and debugging and reducing delivery costs.
• Cost Efficiency : Reusing GPU cloud resources for every test or development cycle among developers, can lead to significant cost savings on expensive individual local development devices.
• Customized Environment Testing : Exposing endpoints locally allows you to create a controlled environment that mirrors your production setup. This capability ensures that any AI model behavior is consistent with expectations before deploying broadly.
• Compliance and Regulation Adherence : Certain industries have stringent compliance requirements for data processing and storage. Running AI models in a controlled, isolated, end-to-end encrypted environment helps adhere to these regulations by ensuring that data does not leave your secure environment without proper safeguards even on development phase.

By securely exposing Ollama endpoints, you can achieve a balance between operational efficiency, security, and compliance while facilitating a robust development process for AI models on your machine.

Be aware that this blueprint may incur additional costs related to ingress and egress data.

Prerequisites

For this Blueprint, we are going to need:

1. an ECS server (Ubuntu 22.04): a GPU-accelerated instance (8 vCPUs, 32 GiB) pi2.2xlarge.4 will suffice.
2. a Point-to-Site VPN connection: We need to establish a connection between our development machine and the VPC hosting the Ollama VM.
3. an Ollama instance: Ollama must be installed on the ECS server above.

If you don’t currently have an Ollama setup, please refer to the official guide for both manual and automated installation options available at this link.

Make sure you add Ollama as a startup service.

Creating a Point-to-Site VPN

Exposing an Ollama endpoint by assigning an Elastic IP (EIP) directly to a Virtual Machine or using Destination Network Address Translation (DNAT) through a NAT Gateway is considered dangerous due to several key concerns. First, assigning an EIP makes the VM accessible from the internet, significantly increasing its exposure and risk of unauthorized access. This direct accessibility enlarges the attack surface, leaving it vulnerable to potential breaches.

Additionally, having such an endpoint exposed can make it a target for Distributed Denial of Service (DDoS) attacks aimed at overwhelming your service with excessive traffic, potentially causing downtime or degraded performance. Moreover, using DNAT via a NAT Gateway does not eliminate these risks entirely; it still requires meticulous configuration and management of security groups and firewall rules to ensure only legitimate traffic is allowed. Misconfigurations in such setups can easily result in unintentional exposure.

From a compliance perspective, direct internet exposure might violate certain regulatory requirements that mandate strict data protection and access controls, depending on the industry or region.

On the other hand, using a Point-to-Site (P2S) VPN to connect to the VPC where the Ollama VM resides is often viewed as a secure solution and aligns with best practices for several reasons. A P2S VPN establishes an encrypted tunnel between your local machine(s) and the respective Open Telekom Cloud VPC, ensuring that transmitted data remains secure from eavesdropping.

Moreover, a P2S VPN provides flexibility for multiple users to securely connect from different locations without needing complex infrastructure changes. This approach also facilitates compliance with data protection regulations since access can be restricted to authorized users only and connections can be monitored and logged for auditing and incident response purposes.

You can find instruction on how to create and establish a P2S VPN connection here: Establish a Point-to-Site VPN Connection between your Development Machine and a VPC.

Changing Ollama’s Listening Address

Let’s inspect the network sockets associated with Ollama service:

sudo ss -tupln | grep ollama

tcp   LISTEN   0   4096   127.0.0.1:11434   0.0.0.0:*   users:(("ollama",pid=2728253,fd=3))

We instantly notice that although Ollama is bound to all available network interfaces (0.0.0.0) and listens on any port (*), in practice, it restricts incoming connections to 127.0.0.1:11434. 127.0.0.1 is the loopback IP address (localhost), meaning it's accessible only from the same machine. The port 11434 is where connections should be made to communicate with Ollama. That will naturally prohibit us accessing the Ollama endpoint from our development machine, or any other machine for that matter. What we need to do is to instruct Ollama service to allow incoming connection to any (or even better to a specific interface). For that matter we need to:

1. Stop temporarily the Ollama service:

sudo systemctl stop ollama

2. Change the systemd service unit for Ollama to accept incoming connections to all interfaces:

sudo nano /etc/systemd/system/ollama.service

add to it an additional environment variable: OLLAMA_HOST=0.0.0.0:11434

[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/local/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"
Environment="OLLAMA_HOST=0.0.0.0:11434"

[Install]
WantedBy=default.target

3. Reload systemd configuration files:

sudo systemctl daemon-reload

4. Start the Ollama service on your system

sudo systemctl start ollama

Creating Security Group

1. Go to Network Console -> Security Groups, and click Create Security Group. Add two new Inbound Rules:

• Protocol: TCP
• Port: 11434
• Source: 192.168.10.0/24

Although we configured Ollama service to listen to any interface, we want to restrict access to the Ollama endpoint only to resources that reside inside the same VPC and not to anyone that might potentially reach this VM, 0.0.0.0/0.

The VPC CIDR for this lab was 192.168.10.0/24, this might differ in your environment given your individual configuration of the VPC and its Subnets so adjust the Source of the Inbound Rule accordingly.

Once the P2S VPN connection is established, your development machine will technically be part of this VPC and will be able to access the Ollama endpoint.

2. Add the new Security Group to the Security Groups of the ECS Server.

Validation

Verify connectivity from your development machine using cURL:

Ensure that the VPN connection is already established.192.168.10.183 is the private IP address assigned by the DHCP server of our VPC to the ECS Server that Ollama is installed on. Remember from the step before that the VPC CIDR for this lab was 192.168.10.0/24, this might differ in your environment given your individual configuration of the VPC and its Subnets.

curl http://192.168.10.183:11434/api/tags

If you have already pulled some models, the response should look similar to this:

{
  "models": [
    {
      "name": "nomic-embed-text:latest",
      "model": "nomic-embed-text:latest",
      "modified_at": "2025-01-20T23:46:07.861519801Z",
      "size": 274302450,
      "digest": "0a109f422b47e3a30ba2b10eca18548e944e8a23073ee3f3e947efcf3c45e59f",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "nomic-bert",
        "families": [
          "nomic-bert"
        ],
        "parameter_size": "137M",
        "quantization_level": "F16"
      }
    },
    {
      "name": "llama3.3:latest",
      "model": "llama3.3:latest",
      "modified_at": "2025-01-17T09:17:50.765928869Z",
      "size": 42520413916,
      "digest": "a6eb4748fd2990ad2952b2335a95a7f952d1a06119a0aa6a2df6cd052a93a3fa",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "llama",
        "families": [
          "llama"
        ],
        "parameter_size": "70.6B",
        "quantization_level": "Q4_K_M"
      }
    },
    {
      "name": "phi4:latest",
      "model": "phi4:latest",
      "modified_at": "2025-01-14T09:16:25.911933283Z",
      "size": 9053116391,
      "digest": "ac896e5b8b34a1f4efa7b14d7520725140d5512484457fab45d2a4ea14c69dba",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "phi3",
        "families": [
          "phi3"
        ],
        "parameter_size": "14.7B",
        "quantization_level": "Q4_K_M"
      }
    },
    {
      "name": "deepseek-coder-v2:latest",
      "model": "deepseek-coder-v2:latest",
      "modified_at": "2024-11-09T08:07:18.67583696Z",
      "size": 8905126121,
      "digest": "63fb193b3a9b4322a18e8c6b250ca2e70a5ff531e962dbf95ba089b2566f2fa5",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "deepseek2",
        "families": [
          "deepseek2"
        ],
        "parameter_size": "15.7B",
        "quantization_level": "Q4_0"
      }
    },
  ]
}

This article was originally posted as a blueprint in Open Telekom Cloud Architecture Center: Securely Expose Remote Ollama Endpoints to your Development Machine.