Issue with transparent cache.nixos.org proxy

secana · 4 months ago

Issue with transparent cache.nixos.org proxy

letThemPlay@lemmy.one · edit-2 4 months ago

Might be misunderstanding this myself, but I thought a transparent proxy intercepts the requests to the destination.

Now if this is the case, I wouldn’t expect any changes to your nix config, as it should be transparently intercepting the requests to cache.nixos.org. Now how this would work with nginx in this context I haven’t quite gotten my head around.

Unless your set your gateway to be your nginx box; to handle all requests and forward them on as necessary.

Might give this a go on my build server if I get a chance.

onlinepersona · 4 months ago

/usr/share/nginx/html/nix-cache-info

That indicates your nginx is probably still using the default config.

This is the default config in nginx.conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

Reading the dockerhub page for your image it states

Customize configuration

You can mount your configuration file, or build a new image with it.

If you wish to adapt the default configuration, use something like the following to get it from a running nginx container:

$ docker run --rm --entrypoint=cat nginx /etc/nginx/nginx.conf > /host/path/nginx.conf

And then edit /host/path/nginx.conf in your host file system.

Basically

volumeMounts:
            - name: nix-cache-storage
              mountPath: /data
            - name: nix-cache-config
              mountPath: /etc/nginx/sites-available/default # this should be /etc/nginx/nginx.conf

You might want to take some configuration options from the default config like the user, worker connections, etc.

Good luck.

Anti Commercial-AI license

secana · 4 months ago

Thanks for the response. You are right, the config was at the wrong path. Unfortunately, the config itself does not work, too. After a bit of testing around this worked for me:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nix-cache-volume
spec:
  capacity:
    storage: 500Gi
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/k8s/nix-cache" # Needs exists before PV is created!
  persistentVolumeReclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nix-cache-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: manual
  resources:
    requests:
      storage: 500Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nix-cache
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nix-cache
  template:
    metadata:
      labels:
        app: nix-cache
        name: nix-cache
    spec:
      volumes:
        - name: nix-cache-storage
          persistentVolumeClaim:
            claimName: nix-cache-pvc
        - name: nix-cache-config
          configMap:
            name: nix-cache-config
      containers:
        - name: nix-cache
          image: nginx:1.27.0 
          ports:
            - containerPort: 80
          volumeMounts:
            - name: nix-cache-storage
              mountPath: /data
            - name: nix-cache-config
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf
              readOnly: true
          resources:
            limits:
              memory: "512Mi"
              cpu: "300m"
            requests:
              memory: "256Mi"
              cpu: "200m"
---
apiVersion: v1
kind: Service
metadata:
  name: nix-cache
spec:
  selector:
    app: nix-cache
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nix-cache-ingress
  annotations:
    traefik.ingress.kubernetes.io/router.tls: "true"
spec:
  rules:
    - host: "nix-cache.raspi.home"
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: nix-cache
                port:
                  number: 80
  tls:
    - secretName: nix-cache-raspi-home-tls
      hosts:
        - "nix-cache.raspi.home"
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: nix-cache.raspi.home
spec:
  commonName: nix-cache.raspi.home
  dnsNames:
    - "nix-cache.raspi.home"
  secretName: nix-cache-raspi-home-tls
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nix-cache-config
data:
  # See: https://www.channable.com/tech/setting-up-a-private-nix-cache-for-fun-and-profit
  nginx.conf: |
    events {
        worker_connections 1024;
    }
    http {
      proxy_cache_path /data/nginx/cache max_size=500G keys_zone=cache_zone:50m inactive=365d;
      proxy_cache cache_zone;
      proxy_cache_valid 200 365d;
      proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_504 http_403 http_404 http_429;
      proxy_ignore_headers X-Accel-Expires Expires Cache-Control Set-Cookie;
      proxy_cache_lock on;

      server {
          listen 80;

          server_name nix-cache.raspi.home;

          location /nix-cache-info {
              return 200 "StoreDir: /nix/store\nWantMassQuery: 1\nPriority: 41\n";
          }

          location / {
              proxy_set_header Host $proxy_host;
              proxy_pass https://cache.nixos.org;
          }
      }
    }

The config is an adaption from this blog post: https://www.channable.com/tech/setting-up-a-private-nix-cache-for-fun-and-profit

onlinepersona · 4 months ago

Nice. Glad you got it working 👍

Anti Commercial-AI license