Monday, December 21, 2015

Use squid redirect_program to correct bandersnatch directory structure for easy_install


After you mirror pypi using bandersnatch, the directory structure is usable with easy_install/pip if you install only web server to host from bandersnatch directory, so you need some program to rewrite the url path.

******** information  ************
bandersnatch direcotory location: /srv/pypi
host name for our repository: www.pypi.repo

1- install apache and squid
$ sudo apt-get install squid apache2 bind9

2- squid configuration file /etc/squid3/squid.conf

acl SSL_ports port 443
acl Safe_ports port 80        # http
acl Safe_ports port 21        # ftp
acl Safe_ports port 443        # https
acl Safe_ports port 70        # gopher
acl Safe_ports port 210        # wais
acl Safe_ports port 1025-65535    # unregistered ports
acl Safe_ports port 280        # http-mgmt
acl Safe_ports port 488        # gss-http
acl Safe_ports port 591        # filemaker
acl Safe_ports port 777        # multiling http
acl CONNECT method CONNECT
acl localhost src 127.0.0.1

http_access allow localhost
http_access deny !Safe_ports

http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access deny all
http_port 3128

dns_nameservers 127.0.0.1acl SSL_ports port 443
acl Safe_ports port 80        # http
acl Safe_ports port 21        # ftp
acl Safe_ports port 443        # https
acl Safe_ports port 70        # gopher
acl Safe_ports port 210        # wais
acl Safe_ports port 1025-65535    # unregistered ports
acl Safe_ports port 280        # http-mgmt
acl Safe_ports port 488        # gss-http
acl Safe_ports port 591        # filemaker
acl Safe_ports port 777        # multiling http
acl CONNECT method CONNECT
acl localhost src 127.0.0.1

http_access allow localhost
http_access deny !Safe_ports

http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access deny all
http_port 3128
#dns_nameservers 8.8.8.8
dns_nameservers 127.0.0.1

acl pypi dstdomain .pypi.repo

#redirect_program /usr/bin/python /etc/squid3/helper.py
#url_rewrite_program /usr/bin/python /etc/squid3/helper.py
#url_rewrite_access allow pypi
redirect_program /usr/bin/python /etc/squid3/helper.py
#url_rewrite_access allow pypi
#redirect_access allow pypi
redirect_children 5

coredump_dir /var/spool/squid3
refresh_pattern ^ftp:        1440    20%    10080
refresh_pattern ^gopher:    1440    0%    1440
refresh_pattern -i (/cgi-bin/|\?) 0    0%    0
refresh_pattern (Release|Packages(.gz)*)$      0       20%     2880
refresh_pattern .        0    20%    4320


acl pypi dstdomain .pypi.repo
redirect_program /usr/bin/python /etc/squid3/helper.py
redirect_children 5

coredump_dir /var/spool/squid3
refresh_pattern ^ftp:        1440    20%    10080
refresh_pattern ^gopher:    1440    0%    1440
refresh_pattern -i (/cgi-bin/|\?) 0    0%    0
refresh_pattern (Release|Packages(.gz)*)$      0       20%     2880
refresh_pattern .        0    20%    4320


3- helper.py content

#!/usr/bin/python
import sys
import re
def modify_url(line):
    if line == "":
        return line
    ulist = line.split()
    old_url = ulist[0]
    old_url1 = ulist[0]
    new_url = "\n"
    old_url = re.sub(r"/$","",old_url)
        n = old_url.split("/")
    if len(n) < 4:
        return old_url1

    if re.search(r"pypi.repo/./.*", old_url):
        return old_url1

    else:
        old_url = re.sub(r"/$","",old_url)
                n = old_url.split("/")
        startLetter = n[3][0]
        new_urls = "/".join(n[0:3]) + "/" + startLetter + "/" + "/".join(n[3:])

    return new_urls


while True:
   
    line = sys.stdin.readline().strip()
    new_url = modify_url(line)
    logrewrite  = open("/var/log/squid3/url.log", "a")
    print new_url
    logrewrite.write(new_url + "\n")
    logrewrite.close()
    sys.stdout.flush()



4- bind9 configuration
after install bind9 create zone name pypi.repo add record www and allow-query from 127.0.0.1

5- apache2 configuration
allow root access to directory /srv/pypi create virtual host name www.pypi.repo with root directory to /srv/pypi/web/packages/sources


6- restart all services that we have modified configuration file, web, dns and proxy

7- client side using easy_install
$ export http_proxy=http://127.0.0.1:3128
$ sudo easy_install -i http://www.pypi.repo dexy

8- if client does not use easy_install, but use pip
$ mkdir ~/.pip
$ echo -e "[global]\nindex-url = http://pypi.repo\n" > ~/.pip/pip.conf
$ export http_proxy=http://127.0.0.1:3128
$ sudo pip install da


+++++++Note+++++++
pip have problem with case sensitivity, so some packages or dependency start with capital letter, it can not be installed.


No comments:

Post a Comment