https://linux.cn/article-10469-1.html
https://arp242.net/weblog/dont-need-k8s.html
EExcel 丞燕快速查詢2
EExcel 丞燕快速查詢2 https://sandk.ffbizs.com/
[轉]用 docker 改善開發流程筆記
https://kkc.github.io/2018/04/28/docker-note/
multistage build
Optimization && Debugging skill
Development environment revolution
multistage build
Optimization && Debugging skill
Development environment revolution
ethereum etl ELK docker docker-compose .env startblock endblock
ethereum-etl export data every time write over file. Filebeat always reload file then ELK receive repeat records...
So
source get file that path need to be careful
So
.env
STARTBLOCK=01205866
ENDBLOCK=01205888
startetl.sh
IP_PORT is go-ethereum-node1 outside ip & port. here is docker-machinesource get file that path need to be careful
#!/bin/bash
IP_PORT=192.168.99.100:18545
ETH_METHOD=eth_blockNumber
BLOCKNUMBER_JSON_HEX=$(curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"'$ETH_METHOD'","params":[],"id":1}' $IP_PORT | jq '.result' | tr -d '"')
BLOCKNUMBER_DEX=$(printf "%08d\n" $BLOCKNUMBER_JSON_HEX)
printf "\n===== Now Geth BlockerNumber =====\n"
printf "HEX: %s\n" $BLOCKNUMBER_JSON_HEX
printf "DEC: %s\n" $BLOCKNUMBER_DEX
source .env
#source ~/ethereum_etl/.env
printf "\n===== .env BLOCK VARS =====\n"
printf "STARTBLOCK: %s\n" $STARTBLOCK
printf "ENDBLOCK: %s\n" $ENDBLOCK
if [[ $BLOCKNUMBER_DEX =~ ^?[0-9]+$ ]] || [ -z $BLOCKNUMBER_DEX ]; then
printf "NOT Number Geth BLOCKNUMBER, check Geth\n"
exit
fi
if [[ $STARTBLOCK =~ ^?[0-9]+$ ]] || [ -z $STARTBLOCK ]; then
printf "NOT Number .env STARTBLOCK, check .env\n"
exit
fi
if [[ $ENDBLOCK =~ ^?[0-9]+$ ]] || [ -z $ENDBLOCK ]; then
printf "NOT Number .env ENDBLOCK, check .env\n"
exit
fi
#printf "\n%s\n" $BLOCKNUMBER_DEX
#printf "%s" $STARTBLOCK
#printf "\n%s\n" $((10#$BLOCKNUMBER_DEX - 10#$STARTBLOCK))
if (( (( 10#$BLOCKNUMBER_DEX - 10#$STARTBLOCK -1 )) > 0 )); then
printf "\n===== Change .env =====\n"
printf "startblock: %s => %s\n" $STARTBLOCK $((10#$ENDBLOCK+1))
printf "endblock: %s => %s\n" $ENDBLOCK $BLOCKNUMBER_DEX
STARTBLOCK=$(printf "%08d\n" $((10#$ENDBLOCK+1)) )
echo STARTBLOCK=$STARTBLOCK > ~/ethereum_etl/.env
echo ENDBLOCK=$BLOCKNUMBER_DEX >> .env
#echo STARTBLOCK=$STARTBLOCK > ~/ethereum_etl/.env
#echo ENDBLOCK=$BLOCKNUMBER_DEX >> ~ethereum_etl/.env
fi
docker-compose up --build ethereum_etl
docker-compose.yml
version: '3.3'
services:
go-ethereum-node1:
build:
context: go-ethereum-node1/
volumes:
#- ./go-ethereum/keystore:/root/.ethereum/devchain/keystore:rw
- ./go-ethereum-node1/genesis/poa_for_dev.json:/root/genesis/poa_for_dev.json:ro
- alldata:/root/.ethereum/devchain
- /etc/localtime:/etc/localtime:ro
#- ./go-ethereum/log:/root/log:rw
entrypoint: /root/start.sh
ports:
- "18545:8545"
- "30313:30303"
- "30313:30303/udp"
networks:
- etl
ethereum_etl:
build:
context: ethereum-etl/
env_file: .env
volumes:
- alldata:/ethereum-etl/output:rw
#- /root/go/src/github.com/ethereum/go-ethereum/build/bin/data:/ethereum-etl/ipc
#restart: unless-stopped
networks:
- etl
volumes:
alldata:
networks:
etl:
driver: bridge
ethereum_etl DOCKERFILE
FROM python:3.6-alpine
MAINTAINER Eric Lim
ENV PROJECT_DIR=ethereum-etl
RUN apk add unzip
RUN wget https://github.com/blockchain-etl/ethereum-etl/archive/develop.zip \
&& unzip develop.zip && rm develop.zip
RUN mv ethereum-etl-develop /$PROJECT_DIR
WORKDIR /$PROJECT_DIR
RUN apk add --no-cache gcc musl-dev #for C libraries:
RUN pip install --upgrade pip && pip install -e /$PROJECT_DIR/
#CMD ["export_all", "-s", "01990000", "-e", "99999999", "-p", "http://xxx.xxx.xxx.xxx:8545", "-o", "output"]
#CMD ["sh","-c", "echo startblock=$STARTBLOCK endblock=$ENDBLOCK"]
CMD ["sh","-c","python ethereumetl export_all -s $STARTBLOCK -e $ENDBLOCK -p http://xxx.xxx.xxx.xxx:8545 -o output"]
go-ethereum-node1
see https://sueboy.blogspot.com/search?q=poaelk oss kibana error docker
{"type":"log","@timestamp":"2019-01-21T08:57:51Z","tags":["status","plugin:elasticsearch@6.5.2","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Request Timeout after 3000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
1. First use oss
FROM docker.elastic.co/elasticsearch/elasticsearch-oss:6.5.2
FROM docker.elastic.co/kibana/kibana-oss:6.5.2
So Not X-pack problem
2. Truly Problem is connect to elasticsearch failed
Even login kibana docker then ping elasticsearch docker that get response. But kibana logs always get this error message.
kibana website message "Kibana server is not ready yet"
Try to get docker elasticsearch ip, change docker kibana.yml like example：
#elasticsearch.url: http://elasticsearch:9200
elasticsearch.url: http://172.19.0.2:9200
docker-compose up --build # sure image rebuild
docker iptables part 3
https://morphyhu.szitcare.com/wordpress/?p=1314
New docker use DOCKER-USER
host route have 172.17.0.0, 172.18.0.0, 172.19.0.0, 172.20.0.0
docker insdie route use 172.18.0.0
iptables MASQUERADE use 172.17.0.0
Docker Internet is failed.
So add iptables -t nat -A POSTROUTING -s 172.18.0.0/16 ! -o docker0 -j MASQUERADE
sudo iptable
iptables-save
iptables -S
New docker use DOCKER-USER
Important!!
check host route & login docker container check route. and iptables -L -t nat POSTROUTING MASQUERADE 172.17.0.0/16 must same submask. EX:host route have 172.17.0.0, 172.18.0.0, 172.19.0.0, 172.20.0.0
docker insdie route use 172.18.0.0
iptables MASQUERADE use 172.17.0.0
Docker Internet is failed.
So add iptables -t nat -A POSTROUTING -s 172.18.0.0/16 ! -o docker0 -j MASQUERADE
#启动后默认增加的规则
iptables -N DOCKER
iptables -N DOCKER-ISOLATION-STAGE-1
iptables -N DOCKER-ISOLATION-STAGE-2
iptables -N DOCKER-USER
iptables -t nat -N DOCKER
iptables -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
iptables -t nat -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
iptables -t nat -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
iptables -t nat -A DOCKER -i docker0 -j RETURN
iptables -A FORWARD -j DOCKER-USER
iptables -A FORWARD -j DOCKER-ISOLATION-STAGE-1
iptables -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -o docker0 -j DOCKER
iptables -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
iptables -A FORWARD -i docker0 -o docker0 -j ACCEPT
iptables -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
iptables -A DOCKER-ISOLATION-STAGE-1 -j RETURN
iptables -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
iptables -A DOCKER-ISOLATION-STAGE-2 -j RETURN
iptables -A DOCKER-USER -j RETURN
#docker run --name smokeping -d -p 82:80 -e PUID=1000 -e PGID=1000 -e TZ=Asia/Shanghai -v /data/smokeping/data:/data -v /data/smokeping/config:/config linuxserver/smokeping
#启动上述镜像后默认增加的规则
iptables -t nat -A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
iptables -t nat -A DOCKER ! -i docker0 -p tcp -m tcp --dport 82 -j DNAT --to-destination 172.17.0.2:80
iptables -A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
sudo iptable
#启动后默认增加的规则
sudo iptables -N DOCKER
sudo iptables -N DOCKER-ISOLATION-STAGE-1
sudo iptables -N DOCKER-ISOLATION-STAGE-2
sudo iptables -N DOCKER-USER
sudo iptables -t nat -N DOCKER
sudo iptables -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
sudo iptables -t nat -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
sudo iptables -t nat -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
sudo iptables -t nat -A DOCKER -i docker0 -j RETURN
sudo iptables -A FORWARD -j DOCKER-USER
sudo iptables -A FORWARD -j DOCKER-ISOLATION-STAGE-1
sudo iptables -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A FORWARD -o docker0 -j DOCKER
sudo iptables -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
sudo iptables -A FORWARD -i docker0 -o docker0 -j ACCEPT
sudo iptables -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
sudo iptables -A DOCKER-ISOLATION-STAGE-1 -j RETURN
sudo iptables -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
sudo iptables -A DOCKER-ISOLATION-STAGE-2 -j RETURN
sudo iptables -A DOCKER-USER -j RETURN
#docker run --name smokeping -d -p 82:80 -e PUID=1000 -e PGID=1000 -e TZ=Asia/Shanghai -v /data/smokeping/data:/data -v /data/smokeping/config:/config linuxserver/smokeping
#启动上述镜像后默认增加的规则
iptables -t nat -A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
iptables -t nat -A DOCKER ! -i docker0 -p tcp -m tcp --dport 82 -j DNAT --to-destination 172.17.0.2:80
iptables -A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
iptables-save
# Generated by iptables-save v1.4.21 on Mon Jan 21 14:26:28 2019
*nat
:PREROUTING ACCEPT [76:8149]
:INPUT ACCEPT [19:1447]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Mon Jan 21 14:26:28 2019
# Generated by iptables-save v1.4.21 on Mon Jan 21 14:26:28 2019
*filter
:INPUT ACCEPT [70:5722]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [42:4460]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Mon Jan 21 14:26:28 2019
iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
elasticsearch dump ethereum-etl
https://github.com/taskrabbit/elasticsearch-dump
Output elasticsearch->json
docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-logs --output=/tmp/etl-logs.json --type=mapping
docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-transactions --output=/tmp/etl-transactions.json --type=mapping
docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-receipts --output=/tmp/etl-receipts.json --type=mapping
docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-contracts --output=/tmp/etl-contracts.json --type=mapping
docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-blocks --output=/tmp/etl-blocks.json --type=mapping
docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-logs --output=/tmp/etl-logs-data.json --type=data
docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-transactions --output=/tmp/etl-transactions-data.json --type=data
docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-receipts --output=/tmp/etl-receipts-data.json --type=data
docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-contracts --output=/tmp/etl-contracts-data.json --type=data
docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-blocks --output=/tmp/etl-blocks-data.json --type=data
Innput json->elasticsearch
docker run --rm -ti -u root -v data:/tmp taskrabbit/elasticsearch-dump --input=/tmp/etl-contracts.json --output=http://xxx.xxx.xxx.xxx:9200/filebeat-6.5.4-etl-contracts --type=mapping
docker run --rm -ti -u root -v data:/tmp taskrabbit/elasticsearch-dump --input=/tmp/etl-contracts-data.json --output=http://xxx.xxx.xxx.xxx:9200/filebeat-6.5.4-etl-contracts --type=data
docker run --rm -ti -u root -v data:/tmp taskrabbit/elasticsearch-dump --input=/tmp/etl-receipts.json --output=http://xxx.xxx.xxx.xxx:9200/filebeat-6.5.4-etl-receipts --type=mapping
docker run --rm -ti -u root -v data:/tmp taskrabbit/elasticsearch-dump --input=/tmp/etl-receipts-data.json --output=http://xxx.xxx.xxx.xxx:9200/filebeat-6.5.4-etl-receipts --type=data
docker run --rm -ti -u root -v data:/tmp taskrabbit/elasticsearch-dump --input=/tmp/etl-transactions.json --output=http://xxx.xxx.xxx.xxx:9200/filebeat-6.5.4-etl-transactions --type=mapping
docker run --rm -ti -u root -v data:/tmp taskrabbit/elasticsearch-dump --input=/tmp/etl-transactions-data.json --output=http://xxx.xxx.xxx.xxx:9200/filebeat-6.5.4-etl-transactions --type=data
docker run --rm -ti -u root -v data:/tmp taskrabbit/elasticsearch-dump --input=/tmp/etl-blocks.json --output=http://xxx.xxx.xxx.xxx:9200/filebeat-6.5.4-etl-blocks --type=mapping
docker run --rm -ti -u root -v data:/tmp taskrabbit/elasticsearch-dump --input=/tmp/etl-blocks-data.json --output=http://xxx.xxx.xxx.xxx:9200/filebeat-6.5.4-etl-blocks --type=_data
clover 4458 mac hardware testing
https://www.tonymacx86.com/threads/solved-clover-boot-way-too-long.245889/
https://themacadmin.com/2012/02/15/mounting-the-efi-boot-partition-on-mac-os-x/
https://themacadmin.com/2012/02/15/mounting-the-efi-boot-partition-on-mac-os-x/
diskutil ls EFI EFI 209.7 MB disk0s1 sudo mount -t msdos /dev/disk0s1 /Volumes/efi
訂閱：
文章 (Atom)