2018 Scrapy Environment Enhance(4)Docker Service for Scrapyd and Tor Network

2018 Scrapy Environment Enhance(4)Docker Service for Scrapyd and Tor Network

Docker Service for Scrapyd

http://sillycat.iteye.com/blog/2422861

Docker Service for Tor Network

There are 3 ports and 3 different function services.

Tor 9051 Port reset IP

Tor 9050 Port to proxy

Privoxy 8118 to proxy to 9050

I put them into one Docker App, but we should be able to split them into 2 apps.

Dockerfile

#Run a Tor Network Server

#Prepare the OS

FROM ubuntu:16.04

MAINTAINER Carl Luo

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -qq update

RUN apt-get -qqy dist-upgrade

#Prepare the denpendencies

RUN apt-get install -qy tor

RUN apt-get install -qy netcat

RUN apt-get install -qy curl

RUN apt-get install -qy privoxy

#set up tor configuration

RUN echo “ControlPort 9051” >> /etc/tor/torrc

RUN echo “ControlListenAddress 0.0.0.0” >> /etc/tor/torrc

RUN echo HashedControlPassword $(tor –hash-password “xxxxxxxx” | tail -n 1) >> /etc/tor/torrc

#set up forward configuration

RUN echo “forward-socks5t / 127.0.0.1:9050 .”>> /etc/privoxy/config

RUN echo “listen-address 0.0.0.0:8118”>> /etc/privoxy/config

#set up the app

EXPOSE 9051 8118

RUN mkdir -p /app/

ADD start.sh /app/

WORKDIR /app/

CMD [ “./start.sh” ]

The command line to start, start.sh

#!/bin/sh -ex

#start the service

service tor start

service privoxy start

tail -f /dev/null

Makefile to open the PORT number

IMAGE=sillycat/public

TAG=ubuntu-tornetwork-1.0

NAME=ubuntu-tornetwork-1.0

docker-context:

build: docker-context

docker build -t $(IMAGE):$(TAG) .

run:

docker run -d -p 9051:9051 -p 8118:8118 –name $(NAME) $(IMAGE):$(TAG)

debug:

docker run -p 9051:9051 -p 8118:8118 –name $(NAME) -ti $(IMAGE):$(TAG) /bin/bash

clean:

docker stop ${NAME}

docker rm ${NAME}

logs:

docker logs ${NAME}

publish:

docker push ${IMAGE}

How to change the IP

> echo -e ‘AUTHENTICATE “xxxxxxxxx”\r\nsignal NEWNYM\r\nQUIT’ | nc localhost 9051

Check the IP

> curl -x localhost:8118 http://icanhazip.com/

How to change the IP from remote

> echo -e ‘AUTHENTICATE “xxxxxxxxx”\r\nsignal NEWNYM\r\nQUIT’ | nc ubuntu-master 9051

Check the IP from Remote

> curl -x ubuntu-master:8118 http://icanhazip.com/

References:

http://sillycat.iteye.com/blog/2418229

http://sillycat.iteye.com/blog/2418353

http://sillycat.iteye.com/blog/2422861

https://github.com/torproject/tor/blob/master/src/config/torrc.sample.in

https://github.com/osminogin/docker-tor-simple

https://github.com/dperson/torproxy

https://program-think.blogspot.com/2014/12/gfw-privoxy.html

http://www.cnblogs.com/another-wheel/archive/2011/11/16/setup-http-proxy-via-socks-and-privoxy.html

https://blog.phpgao.com/privoxy-shadowsocks.html

https://blog.csdn.net/north_eagle/article/details/8683403

https://stackoverflow.com/questions/15459170/trouble-authenticating-tor-with-python

https://gist.github.com/DusanMadar/c1155329cf6a71e4346cae271a2eafd3

https://www.torproject.org/docs/faq.html.en

https://stackoverflow.com/questions/45901892/how-to-connect-to-tor-control-port-9051-from-a-remote-host

https://stackoverflow.com/questions/25775266/how-to-keep-docker-container-running-after-starting-services/36872226