EExcel 丞燕快速查詢2

EExcel 丞燕快速查詢2
EExcel 丞燕快速查詢2 https://sandk.ffbizs.com/

gitlab old version backup restore upgrade

check version   http://xxx.xxx.xxx.xxx/help

1. https://packages.gitlab.com/gitlab/gitlab-ce




2. example https://packages.gitlab.com/gitlab/gitlab-ce/packages/el/7/gitlab-ce-9.5.4-ce.0.el7.x86_64.rpm


curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.rpm.sh | sudo bash
sudo yum install gitlab-ce-9.5.4-ce.0.el7.x86_64

ps: https://docs.gitlab.com/omnibus/manual_install.html



3. http://sueboy.blogspot.com/2018/12/gitlab.html

sudo nano /etc/gitlab/gitlab.rb

可以在裡面找到external_url

sudo gitlab-ctl reconfigure
sudo gitlab-ctl status


4. open firewall

firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --list-all --permanent


5. backup restore
https://docs.gitlab.com/ee/raketasks/backup_restore.html
https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/raketasks/backup_restore.md


6. upgrade
https://docs.gitlab.com/omnibus/update/

gitlab Maintenance Policy   Upgrade recommendations
https://docs.gitlab.com/ee/policy/maintenance.html#upgrade-recommendations

9.5 to 10
https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/9.5-to-10.0.md



================================

目前已經在測試環境模擬過,安全可行方式:
  1. 建立一台新的centos
  2. 灌gitlab 9.5.4,還原 目前9.5.4,確認是否能正常執行gitlab
  3. 開始升級 9.5.4 -> 11.5.5 會失敗,必需 9.5.4 -> 9.5.10 -> 10.8.7 -> 11.3.4 -> 11.5.5,必需按照這個步驟升級4次
  4. gitlab 11.5.5升級完後,請人員操作測試是否有問題,沒問題後,再把舊的關閉,切換ip
個人帳號操作看起來沒有問題,需要其他人測試才能更清楚是否有其他問題

事先準備:
  • 升級過程大概需要4小時以上的時間
  • 原本gitlab匯出後,不可以再新增資料


[轉]【Flutter踩雷心得】楊維中 (R1-Day2) MOPCON 2018


A script to backup GitLab repositories.

https://gist.github.com/avence12/70fc35963444d096d1bdb558b252327a
https://www.lightblue.asia/gitlab-backup-script/?doing_wp_cron=1545378263.6541728973388671875000

Change:

1. http token: kzy3x2TSeukztpvzBiYA token from gitlab Personal Access Token, Don't use Account Private Tokens. Need to product Personal Access Token


GLAB_GIT_CLONE_CMD="git clone --quiet --mirror git@${GLAB_GITHOST}:"


GLAB_GIT_CLONE_CMD="git clone --quiet --mirror http://oauth2:${GLAB_TOKEN}@${GLAB_GITHOST}/"

PS:GLAB_TOKEN use in GLAB_PROJ_API, Personal Access Token or Account Private Tokens both is ok

2. REPO have bug:REPO have " must remove, so add line under
https://stackoverflow.com/questions/13570327/how-to-delete-a-substring-using-shell-script

for REPO in $REPOLIST; do


for REPO in $REPOLIST; do
  REPO=${REPO%\"}


3.
GLAB_GITHOST、GLAB_API have gitlab.com change => ip:port or domain


ALL

#!/bin/bash
# A script to backup GitLab repositories.

GLAB_BACKUP_DIR=${GLAB_BACKUP_DIR-"gitlab_backup"}                   # where to place the backup files
GLAB_TOKEN=${GLAB_TOKEN-"YOUR_TOKEN"}        #kzy3x2TSeukztpvzBiYA   # the access token of the account
GLAB_GITHOST=${GLAB_GITHOST-"gitlab.com"}                            # the GitLab hostname
GLAB_PRUNE_OLD=${GLAB_PRUNE_OLD-true}                                # when `true`, old backups will be deleted
GLAB_PRUNE_AFTER_N_DAYS=${GLAB_PRUNE_AFTER_N_DAYS-7}                 # the min age (in days) of backup files to delete
GLAB_SILENT=${GLAB_SILENT-false}                                     # when `true`, only show error messages
GLAB_API=${GLAB_API-"https://gitlab.com/api/v3"}                     # base URI for the GitLab API
GLAB_GIT_CLONE_CMD="git clone --quiet --mirror git@${GLAB_GITHOST}:" # base command to use to clone GitLab repos

TSTAMP=`date "+%Y%m%d"`

# The function `check` will exit the script if the given command fails.
function check {
  "$@"
  status=$?
  if [ $status -ne 0 ]; then
    echo "ERROR: Encountered error (${status}) while running the following:" >&2
    echo "           $@"  >&2
    echo "       (at line ${BASH_LINENO[0]} of file $0.)"  >&2
    echo "       Aborting." >&2
    exit $status
  fi
}

# The function `tgz` will create a gzipped tar archive of the specified file ($1) and then remove the original
function tgz {
   check tar zcf $1.tar.gz $1 && check rm -rf $1
}

$GLAB_SILENT || (echo "" && echo "=== INITIALIZING ===" && echo "")

$GLAB_SILENT || echo "Using backup directory $GLAB_BACKUP_DIR"
check mkdir -p $GLAB_BACKUP_DIR

$GLAB_SILENT || echo -n "Fetching list of repositories ..."
GLAB_PROJ_API="${GLAB_API}/projects?private_token=${GLAB_TOKEN}&per_page=100&simple=true"
echo ${GLAB_PROJ_API}

REPOLIST=`check curl --silent ${GLAB_PROJ_API} | check perl -p -e "s/,/\n/g" | check grep "\"path_with_namespace\"" | check awk -F':"' '{print $2}' | check sed -e 's/"}//g'`

$GLAB_SILENT || echo "found `echo $REPOLIST | wc -w` repositories."

$GLAB_SILENT || (echo "" && echo "=== BACKING UP ===" && echo "")

for REPO in $REPOLIST; do
   REPO=${REPO%\"}
   $GLAB_SILENT || echo "Backing up ${REPO}"
   check ${GLAB_GIT_CLONE_CMD}${REPO}.git ${GLAB_BACKUP_DIR}/${GLAB_ORG}-${REPO}-${TSTAMP}.git && tgz ${GLAB_BACKUP_DIR}/${GLAB_ORG}-${REPO}-${TSTAMP}.git
done

if $GLAB_PRUNE_OLD; then
  $GLAB_SILENT || (echo "" && echo "=== PRUNING ===" && echo "")
  $GLAB_SILENT || echo "Pruning backup files ${GLAB_PRUNE_AFTER_N_DAYS} days old or older."
  $GLAB_SILENT || echo "Found `find $GLAB_BACKUP_DIR -name '*.tar.gz' -mtime +$GLAB_PRUNE_AFTER_N_DAYS | wc -l` files to prune."
  find $GLAB_BACKUP_DIR -name '*.tar.gz' -mtime +$GLAB_PRUNE_AFTER_N_DAYS -exec rm -fv {} > /dev/null \;
fi

$GLAB_SILENT || (echo "" && echo "=== DONE ===" && echo "")
$GLAB_SILENT || (echo "GitLab backup completed." && echo "")




新版gitlab安裝

https://xenby.com/b/131-%E6%8E%A8%E8%96%A6-gitlab%E5%AE%89%E8%A3%9D%E8%88%87%E5%9F%BA%E6%9C%AC%E8%A8%AD%E5%AE%9A%E6%95%99%E5%AD%B8


https://gitlab.com/gitlab-org/gitlab-ce/#installation





sudo nano /etc/gitlab/gitlab.rb

可以在裡面找到external_url

sudo gitlab-ctl reconfigure

過年

最近外資群準備要過年了,內資及集團開始作帳,成交量不大,但穩定的上漲,內資或外資群不作帳,第四季的績效就難看,可能這一年過完後就會被開除,所以意思意思還是得作一下帳。

[轉]Bye bye Mongo, Hello Postgres

https://www.theguardian.com/info/2018/nov/30/bye-bye-mongo-hello-postgres

[轉]我花了一个五一终于搞懂了OpenLDAP

https://juejin.im/entry/5aec6ac46fb9a07ac3635884

如果你搜索OpenLDAP的安装指南,很不幸地告诉你,网上不管中文的英文的,90%都是错的,它们都还活在上个世纪,它们会告诉你要去修改一个叫做slapd.conf的文件,基本上看到这里,你就不用往下看了,这个文件早就被抛弃,新版的OpenLDAP里根本就没有这个文件!取而代之的是slapd.d的文件夹,然后另一部分教程会告诉你,让你修改这个文件夹下的某一个ldif文件,看到这里,你也不用往下看了,你又看到了伪教程,因为这个文件夹下的所有文件的第一行都明确地写着:『这是一个自动生成的文件,不要修改它!』你修改了它之后,它的md5校验值会匹配不上,造成更多的问题。你应该用ldapmodify来修改这个文件,而关于ldapmodify的教程,可以说几乎就没有!我一开始不知道面临这样荒谬的处境,很多运维人员是怎么活下来的,不过等我自己配通了以后,真的是累到连写教程的精力都没有了,好吧,我已经配通了,你们各人自求

[轉]Huge performance hit on a simple Go server with Docker

https://www.reddit.com/r/golang/comments/8ojjoj/huge_performance_hit_on_a_simple_go_server_with/

Docker's disk I/O was the bottleneck!

Solved: Disk I/O was the problem. Even though I'm on Debian 9.4, changing to an in-memory sqlite allowed Docker to push 2.9k reqs/sec. Credits to /u/Willson50 for the amazing find!


With file=dco.sqlite3?mode=memory, Docker can push ~2.9k req/sec.



Other way:

PRAGMA journal_mode=WAL;
PRAGMA synchronous=NORMAL;

https://www.sqlite.org/wal.html

xgo Go CGO cross compile sqlite

https://github.com/karalabe/xgo

ethereum poa docker-compose docker


https://medium.com/taipei-ethereum-meetup/%E4%BD%BF%E7%94%A8-go-ethereum-1-6-clique-poa-consensus-%E5%BB%BA%E7%AB%8B-private-chain-1-4d359f28feff

node 1
node 2
signer 1
signer 2

all private key & address create by https://vanity-eth.tk/
check again private key & address by https://www.myetherwallet.com/#view-wallet-info

 node1
private key:138cbbfb21686ddc3b5ffeb2cfc83491175af68319977acb81d0ae93392c626c
account address:e79d33e93bd888b35e055f1a12d876354729037b

 node2
private key:df0c39faccd3c9d6d10091932f2214b9d42e92bd07b64a1552d13d39a6a84122
account address:Bf95EAacf10BEA53f0a73955a7eFCE2fb4A4Ef1d

signer1
nodekeyhex:fcc406a7344690f66c757cc2c9987e3e78bb01e33229f9877d48a7fecc2d6657
node address:256330933851d6d3c5f7326b01021553415a33cd5485e23bfbe35f6321e6e2f8373bb1c94933fdb3283a1a8b2b737587dd99c555029e65e173a3094daa39277c

private key:d05bd152f3d71ff5f91830f3ccc1090fb670c7026ebf8c2136d4e5090d59398d
account address:5921a4C1B13afbD4b61d63e9c7BD47741C47B176

signer2
private key:df504d175ae63abf209bad9dda965310d99559620550e74521a6798a41215f46
account address:8Cc5A1a0802DB41DB826C2FcB72423744338DcB0


docker-compose


version: '3.3'

services:
  go-ethereum-node1:
    build:
      context: go-ethereum-node1/
    volumes:
      #- ./go-ethereum/keystore:/root/.ethereum/devchain/keystore:rw
      - ./go-ethereum-node1/genesis/poa_for_dev.json:/root/genesis/poa_for_dev.json:ro
      - /etc/localtime:/etc/localtime:ro
      #- ./go-ethereum/log:/root/log:rw
    entrypoint: /root/start.sh
    ports:
      - "18545:8545"
      - "30313:30303"
      - "30313:30303/udp"
    networks:
      - fastdev

  go-ethereum-node2:
    build:
      context: go-ethereum-node2/
    volumes:
      - ./go-ethereum-node2/genesis/poa_for_dev.json:/root/genesis/poa_for_dev.json:ro
      - /etc/localtime:/etc/localtime:ro
    entrypoint: /root/start.sh
    ports:
      - "28545:8545"
      - "30323:30303"
      - "30323:30303/udp"
    networks:
      - fastdev

  go-ethereum-signer1:
    build:
      context: go-ethereum-signer1/
    volumes:
      - ./go-ethereum-signer1/genesis/poa_for_dev.json:/root/genesis/poa_for_dev.json:ro
      - /etc/localtime:/etc/localtime:ro
    entrypoint: /root/start.sh
    ports:
      - "38545:8545"
      - "30333:30303"
      - "30333:30303/udp"
    networks:
      - fastdev

  go-ethereum-signer2:
    build:
      context: go-ethereum-signer2/
    volumes:
      - ./go-ethereum-signer2/genesis/poa_for_dev.json:/root/genesis/poa_for_dev.json:ro
      - /etc/localtime:/etc/localtime:ro
    entrypoint: /root/start.sh
    ports:
      - "48545:8545"
      - "30343:30303"
      - "30343:30303/udp"
    networks:
      - fastdev

  mongo:
    image: mongo
    #restart: always
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: example
    volumes:
      - alldata:/data/db
    networks:
      - fastdev

  mongo-express:
    image: mongo-express
    #restart: always
    links:
      - mongo:mongo
    ports:
      - 8081:8081
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: root
      ME_CONFIG_MONGODB_ADMINPASSWORD: example
    networks:
      - fastdev

  postgres:
    image: postgres
    #restart: always
    environment:
      PGDATA: /var/lib/postgresql/data/pgdata
      POSTGRES_USER: root
      POSTGRES_PASSWORD: example
    volumes:
      - alldata:/var/lib/postgresql/data
    networks:
      - fastdev

  mariadb:
    image: mariadb
    #restart: always
    environment:
      MYSQL_ROOT_PASSWORD: example
    volumes:
      - alldata:/var/lib/mysql
    networks:
      - fastdev

  adminer:
    image: adminer
    #restart: always
    ports:
      - 8082:8080
    networks:
      - fastdev
      
  openldap:
    build:
      context: openldap/
    command: [--copy-service,  --loglevel, debug]
    environment:
      LDAP_ORGANISATION: "das-labor"
      LDAP_DOMAIN: "das-labor.org"
      LDAP_BASE_DN: "dc=das-labor,dc=org"
      LDAP_ADMIN_PASSWORD: "abc123"
      LDAP_CONFIG_PASSWORD: "abc123"
    volumes:
      - ./openldap/data/ldap/environment:/container/environment/01-custom
      - ./openldap/data/slapd/database:/var/lib/ldap
      - ./openldap/data/slapd/config:/etc/ldap/slapd.d
      #- ./openldap/certs:/container/service/slapd/assets/certs
    ports:
      - "389:389"
      - "689:689"
    networks:
      - fastdev

  phpldapadmin:
    image: osixia/phpldapadmin
    command: [--loglevel, debug]
    environment:
      PHPLDAPADMIN_LDAP_HOSTS: "openldap"
    ports:
      - "6443:443"
    networks:
      - fastdev

volumes:  
  alldata: 

networks:
  fastdev:
    driver: bridge


geth Dockerfile all same


FROM ethereum/client-go:alltools-latest

RUN \
    echo '@edge http://dl-cdn.alpinelinux.org/alpine/edge/main' >> /etc/apk/repositories && \
    echo '@edge http://dl-cdn.alpinelinux.org/alpine/edge/community' >> /etc/apk/repositories && \
    echo '@edge http://dl-cdn.alpinelinux.org/alpine/edge/testing' >> /etc/apk/repositories && \
    apk --no-cache upgrade

RUN apk update &&\
    apk add git nodejs bash perl

ADD ./genesis/poa_for_dev.json /root/genesis/poa_for_dev.json
#RUN geth init /root/genesis/genesis.json --datadir "~/.ethereum/devchain"


#RUN geth --dev --rpcapi "db,personal,eth,net,web3" --rpc --rpcaddr "0.0.0.0" --rpccorsdomain "*" --mine 
#geth --syncmode "light" --cache=2048
#./geth --datadir /root/.ethereum --password /root/geth.password --unlock "0" --port 30310 --rpc --rpcaddr "0.0.0.0" -rpcapi "eth,web3" --rpcport 8110 --networkid 4567890 --dev --lightkdf --nodiscover --maxpeers 0 --verbosity 6 --pprof --pprofport 6110 $@ 2> /tmp/geth.log

COPY start.sh /root/start.sh
RUN chmod +x /root/start.sh

ENTRYPOINT /root/start.sh


genesis use puppeth see

https://medium.com/taipei-ethereum-meetup/%E4%BD%BF%E7%94%A8-go-ethereum-1-6-clique-poa-consensus-%E5%BB%BA%E7%AB%8B-private-chain-1-4d359f28feff



start.sh all different

node1


#!/bin/bash
set -e
datadir="~/.ethereum/devchain"
#ip=$(echo `awk 'END{print $1}' /etc/hosts`)
ip=$(ping -c1 go-ethereum-signer1 | sed -nE 's/^PING[^(]+\(([^)]+)\).*/\1/p')

echo "======== geth ========"
echo "======== init ========"
geth --datadir=$datadir init "/root/genesis/poa_for_dev.json"

sleep 2
echo "======== bootnode ========"
#bootkey="fcc406a7344690f66c757cc2c9987e3e78bb01e33229f9877d48a7fecc2d6657"
#bootnode -nodekeyhex $bootkey -verbosity 4 &
#bootnodeid=$(bootnode --nodekeyhex=$bootkey -writeaddress) #node address: 256330933851d6d3c5f7326b01021553415a33cd5485e23bfbe35f6321e6e2f8373bb1c94933fdb3283a1a8b2b737587dd99c555029e65e173a3094daa39277c

#bootnode --genkey=boot.key
#bootnode --nodekey=boot.key -verbosity 4 &
#bootnode --nodekey=boot.key  -writeaddress > bootnodeid.txt
#bootnodeid=$(bootnode --nodekey=boot.key  -writeaddress)

sleep 2
echo "======== account ========"
privatekey="138cbbfb21686ddc3b5ffeb2cfc83491175af68319977acb81d0ae93392c626c"
address="e79d33e93bd888b35e055f1a12d876354729037b"

echo "======== check account exist ========"
isInFile=$(echo `geth account list --datadir=$datadir | grep -c $address`)
if [ $isInFile -eq 0 ]; then
    echo "======== acoount no exist! Starting import! ========"
    echo "" > ~/.accountpassword
    echo $privatekey > ~/.privatekey
    geth account import --datadir=$datadir --password ~/.accountpassword  ~/.privatekey
else
   echo "======== account exist ========"
fi

sleep 2
echo "======== mine ========"
nodeaddress="256330933851d6d3c5f7326b01021553415a33cd5485e23bfbe35f6321e6e2f8373bb1c94933fdb3283a1a8b2b737587dd99c555029e65e173a3094daa39277c"

#no use bootnode, fix nodekeyhex
#geth --datadir=$datadir --nodekeyhex "fcc406a7344690f66c757cc2c9987e3e78bb01e33229f9877d48a7fecc2d6657" --networkid 53809 --port 30303 --rpcapi "db,admin,debug,miner,personal,txpool,eth,net,web3" --rpc --rpcaddr "0.0.0.0" --rpccorsdomain "*" --rpcvhosts=* --password ~/.accountpassword --unlock $address --mine --minerthreads=1  2> /root/geth.log

#use bootnode
bootnodeId="enode://"$nodeaddress"@"$ip":30303"
#bootnodeId="enode://"$bootnodeid"@"$ip":30303"
until echo | nc -z -v go-ethereum-signer1 30303; do
    echo "Waiting go-ethereum-signer1 to start..."
    sleep 2
done
geth --datadir=$datadir --bootnodes $bootnodeId --networkid 11559 --port 30303 --rpcapi "db,admin,debug,miner,personal,txpool,eth,net,web3" --rpc --rpcaddr "0.0.0.0" --rpccorsdomain "*" --rpcvhosts=* --password ~/.accountpassword --etherbase $address  2> /root/geth.log

node2


#!/bin/bash
set -e
datadir="~/.ethereum/devchain"
#ip=$(echo `awk 'END{print $1}' /etc/hosts`)
ip=$(ping -c1 go-ethereum-signer1 | sed -nE 's/^PING[^(]+\(([^)]+)\).*/\1/p')

echo "======== geth ========"
echo "======== init ========"
geth --datadir=$datadir init "/root/genesis/poa_for_dev.json"

sleep 2
echo "======== bootnode ========"
#bootkey="fcc406a7344690f66c757cc2c9987e3e78bb01e33229f9877d48a7fecc2d6657"
#bootnode -nodekeyhex $bootkey -verbosity 4 &
#bootnodeid=$(bootnode --nodekeyhex=$bootkey -writeaddress) #node address: 256330933851d6d3c5f7326b01021553415a33cd5485e23bfbe35f6321e6e2f8373bb1c94933fdb3283a1a8b2b737587dd99c555029e65e173a3094daa39277c

#bootnode --genkey=boot.key
#bootnode --nodekey=boot.key -verbosity 4 &
#bootnode --nodekey=boot.key  -writeaddress > bootnodeid.txt
#bootnodeid=$(bootnode --nodekey=boot.key  -writeaddress)

sleep 2
echo "======== account ========"
privatekey="df0c39faccd3c9d6d10091932f2214b9d42e92bd07b64a1552d13d39a6a84122"
address="bf95eaacf10bea53f0a73955a7efce2fb4a4ef1d"

echo "======== check account exist ========"
isInFile=$(echo `geth account list --datadir=$datadir | grep -c $address`)
if [ $isInFile -eq 0 ]; then
    echo "======== acoount no exist! Starting import! ========"
    echo "" > ~/.accountpassword
    echo $privatekey > ~/.privatekey
    geth account import --datadir=$datadir --password ~/.accountpassword  ~/.privatekey
else
   echo "======== account exist ========"
fi

sleep 2
echo "======== mine ========"
nodeaddress="256330933851d6d3c5f7326b01021553415a33cd5485e23bfbe35f6321e6e2f8373bb1c94933fdb3283a1a8b2b737587dd99c555029e65e173a3094daa39277c"

#no use bootnode, fix nodekeyhex
#geth --datadir=$datadir --nodekeyhex "fcc406a7344690f66c757cc2c9987e3e78bb01e33229f9877d48a7fecc2d6657" --networkid 53809 --port 30303 --rpcapi "db,admin,debug,miner,personal,txpool,eth,net,web3" --rpc --rpcaddr "0.0.0.0" --rpccorsdomain "*" --rpcvhosts=* --password ~/.accountpassword --unlock $address --mine --minerthreads=1  2> /root/geth.log

#use bootnode
bootnodeId="enode://"$nodeaddress"@"$ip":30303"
#bootnodeId="enode://"$bootnodeid"@"$ip":30303"
until echo | nc -z -v go-ethereum-signer1 30303; do
    echo "Waiting go-ethereum-signer1 to start..."
    sleep 2
done
geth --datadir=$datadir --bootnodes $bootnodeId --networkid 11559 --port 30303 --rpcapi "db,admin,debug,miner,personal,txpool,eth,net,web3" --rpc --rpcaddr "0.0.0.0" --rpccorsdomain "*" --rpcvhosts=* --password ~/.accountpassword --etherbase $address  2> /root/geth.log


signer1


#!/bin/bash
set -e
datadir="~/.ethereum/devchain"
ip=$(echo `awk 'END{print $1}' /etc/hosts`)
#ip=$(ping -c1 go-ethereum-signer1 | sed -nE 's/^PING[^(]+\(([^)]+)\).*/\1/p')

echo "======== geth ========"
echo "======== init ========"
geth --datadir=$datadir init "/root/genesis/poa_for_dev.json"

sleep 2
echo "======== bootnode ========"
#bootkey="fcc406a7344690f66c757cc2c9987e3e78bb01e33229f9877d48a7fecc2d6657"
#bootnode -nodekeyhex $bootkey -verbosity 4 &
#bootnodeid=$(bootnode --nodekeyhex=$bootkey -writeaddress) #node address: 256330933851d6d3c5f7326b01021553415a33cd5485e23bfbe35f6321e6e2f8373bb1c94933fdb3283a1a8b2b737587dd99c555029e65e173a3094daa39277c

#bootnode --genkey=boot.key
#bootnode --nodekey=boot.key -verbosity 4 &
#bootnode --nodekey=boot.key  -writeaddress > bootnodeid.txt
#bootnodeid=$(bootnode --nodekey=boot.key  -writeaddress)

sleep 2
echo "======== account ========"
privatekey="d05bd152f3d71ff5f91830f3ccc1090fb670c7026ebf8c2136d4e5090d59398d"
address="5921a4c1b13afbd4b61d63e9c7bd47741c47b176"

echo "======== check account exist ========"
isInFile=$(echo `geth account list --datadir=$datadir | grep -c $address`)
if [ $isInFile -eq 0 ]; then
    echo "======== acoount no exist! Starting import! ========"
    echo "" > ~/.accountpassword
    echo $privatekey > ~/.privatekey
    geth account import --datadir=$datadir --password ~/.accountpassword  ~/.privatekey
else
   echo "======== account exist ========"
fi

sleep 2
echo "======== mine ========"
#nodeaddress="256330933851d6d3c5f7326b01021553415a33cd5485e23bfbe35f6321e6e2f8373bb1c94933fdb3283a1a8b2b737587dd99c555029e65e173a3094daa39277c"

#no use bootnode, fix nodekeyhex
geth --datadir=$datadir --nodekeyhex "fcc406a7344690f66c757cc2c9987e3e78bb01e33229f9877d48a7fecc2d6657" --networkid 11559 --port 30303 --rpcapi "db,admin,debug,miner,personal,txpool,eth,net,web3" --rpc --rpcaddr "0.0.0.0" --rpccorsdomain "*" --rpcvhosts=* --password ~/.accountpassword --unlock $address --mine --minerthreads=1  2> /root/geth.log

#use bootnode
#bootnodeId="enode://"$nodeaddress"@"$ip":30303"
#bootnodeId="enode://"$bootnodeid"@"$ip":30303"
#until echo | nc -z -v go-ethereum-signer1 30303; do
#    echo "Waiting go-ethereum-signer1 to start..."
#    sleep 2
#done
#geth --datadir=$datadir --bootnodes $bootnodeId --networkid 11559 --port 30303 --rpcapi "db,admin,debug,miner,personal,txpool,eth,net,web3" --rpc --rpcaddr "0.0.0.0" --rpccorsdomain "*" --rpcvhosts=* --password ~/.accountpassword --etherbase $address  2> /root/geth.log


signer2


#!/bin/bash
set -e
datadir="~/.ethereum/devchain"
#ip=$(echo `awk 'END{print $1}' /etc/hosts`)
ip=$(ping -c1 go-ethereum-signer1 | sed -nE 's/^PING[^(]+\(([^)]+)\).*/\1/p')

echo "======== geth ========"
echo "======== init ========"
geth --datadir=$datadir init "/root/genesis/poa_for_dev.json"

sleep 2
echo "======== bootnode ========"
#bootkey="fcc406a7344690f66c757cc2c9987e3e78bb01e33229f9877d48a7fecc2d6657"
#bootnode -nodekeyhex $bootkey -verbosity 4 &
#bootnodeid=$(bootnode --nodekeyhex=$bootkey -writeaddress) #node address: 256330933851d6d3c5f7326b01021553415a33cd5485e23bfbe35f6321e6e2f8373bb1c94933fdb3283a1a8b2b737587dd99c555029e65e173a3094daa39277c

#bootnode --genkey=boot.key
#bootnode --nodekey=boot.key -verbosity 4 &
#bootnode --nodekey=boot.key  -writeaddress > bootnodeid.txt
#bootnodeid=$(bootnode --nodekey=boot.key  -writeaddress)

sleep 2
echo "======== account ========"
privatekey="df504d175ae63abf209bad9dda965310d99559620550e74521a6798a41215f46"
address="8cc5a1a0802db41db826c2fcb72423744338dcb0"

echo "======== check account exist ========"
isInFile=$(echo `geth account list --datadir=$datadir | grep -c $address`)
if [ $isInFile -eq 0 ]; then
    echo "======== acoount no exist! Starting import! ========"
    echo "" > ~/.accountpassword
    echo $privatekey > ~/.privatekey
    geth account import --datadir=$datadir --password ~/.accountpassword  ~/.privatekey
else
   echo "======== account exist ========"
fi


sleep 2
echo "======== mine ========"
nodeaddress="256330933851d6d3c5f7326b01021553415a33cd5485e23bfbe35f6321e6e2f8373bb1c94933fdb3283a1a8b2b737587dd99c555029e65e173a3094daa39277c"

#no use bootnode, fix nodekeyhex
bootnodeId="enode://"$nodeaddress"@"$ip":30303"
until echo | nc -z -v go-ethereum-signer1 30303; do
    echo "Waiting go-ethereum-signer1 to start..."
    sleep 2
done
geth --datadir=$datadir --bootnodes $bootnodeId --networkid 11559 --port 30303 --rpcapi "db,admin,debug,miner,personal,txpool,eth,net,web3" --rpc --rpcaddr "0.0.0.0" --rpccorsdomain "*" --rpcvhosts=* --password ~/.accountpassword --unlock $address --mine --minerthreads=1  2> /root/geth.log

#use bootnode
#bootnodeId="enode://"$nodeaddress"@"$ip":30303"
#bootnodeId="enode://"$bootnodeid"@"$ip":30303"
#until echo | nc -z -v go-ethereum-signer1 30303; do
#    echo "Waiting go-ethereum-signer1 to start..."
#    sleep 2
#done
#geth --datadir=$datadir --bootnodes $bootnodeId --networkid 11559 --port 30303 --rpcapi "db,admin,debug,miner,personal,txpool,eth,net,web3" --rpc --rpcaddr "0.0.0.0" --rpccorsdomain "*" --rpcvhosts=* --password ~/.accountpassword --etherbase $address  2> /root/geth.log



Best Important is

1. private key & address must right.
2. signer1 must start, so node1 node2 signer2 wait signer1
3. start.sh file must utf-8 & LE
4. docker inside port different outside port(expose port)



Final check:

login docker signer1


docker ps -a |grep signer1    Get docker signer1 id
docker exec -it xxxid /bin/bash

into docker


>find / -name geth.ipc     Get geth.ipc path
>get attach ipc:/xxxx/xxxxx/xxxxx/xxxxx/geth.ipc

into geth console


>admin.peers    Can see three node (node1 node2 signer2)
>personal.listWallets   Check unlock 
>eth.mining   Check true (mining start mining)



PS:keep bootnode code for if you want wake up bootnode and not use signer1 be connect. This can make code more easy and smae, only some key word ( private key, address..etc) different.


PS:If create finish, change volumn path that maybe get “can't find .accountpassword “. fix way:just add again at account exist

   echo "" > ~/.accountpassword

full demo like this

else
   echo "======== account exist ========"
   echo "" > ~/.accountpassword
fi

Flutter ionic framwork7

有寫一支離線APP(不需要網路),app有上架android和ios,andorid有開廣告,而ios賣出去一些。

一開始開發使用framework 7 (純javascrip、html、css),開發速度是最快的,照著官網教學做,顏色不大改、版型照用,真的很快,然後用codova包一包,就上架了,操作起來也順,介面看起來都很像ios,唯一的問題是要花時間了解fromework 7架構上,第一次使用當然很痛苦,痛點在於framework,優點在於 html、css、javascript自由。
整體開發時程:一周內肯定解決。

後來改版了,花一些時間了解ionic,發現ionic跟Framework7用法差不多,但卻可以使用ionic native跟底層溝通(不過實際上這支APP根本不用到native),硬生生加入一個底層開啟圖片的功能;ionic開發起來也是很快,因為還是html為主的方式,另外也有花時間了解react,和react不同是,一開始的jsx就一堆問題,不過這是在很早期react會發生的問題(現在已經沒有這些問題);除react外,Angular也同樣有花時間了解,那時 Angular 1 基本上也是放棄,因為太複雜、反人類… Angular 2在寶哥展示後,發現“真的“不反人類了,就決定可以用ionic;ionic開發快在Angular 2+的底層架構,實際開發中的痛苦點反而比Framework7少,畢竟有Framework7可以對照開發,只需要進行語法上的轉換,不用花心思在架構上,Angular也是可以用javascript,Framework7上的array、object可以直接照搬來使用,唯一花時間在 上廣告 和 圖片呼叫底層,有痛到。
一周內肯定解決。

ionic 2 -> ionic 3 ,基本上 印象中 1天內解決吧,處理一些不相容問題而已。

ionic 2或3 順暢度,覺得差不多,唯一碰到的是原生的元件,在中文顯示小標題時,有一點被裁到,這問題在網頁上看時是正常的,在app上看時是不正常,這令人很困擾,但也懶得再用css調整了,就是這問題成為往flutter開發前進的關鍵原因。

Flutter官方展示對我最重要的點:修改後即時更新顯示 和 Material的元件。
改版撰寫時期發現 即時更新顯示 正常情況下是沒問題,但會有失效的情況,最終還是要重新complie app,但這功能只要運作正常時,是非常好用,可以慢慢調整畫面,真的太好用了!
之前開發都是跟web相關,即使在web broswer看是正常的,套入app後還是要把所有的情況再點擊測試一次(會有網頁正常、APP不正常情況),撰寫過程中發現問題進行修正後,需要再度complie app,然後再執行再調整,重覆web模式,和原生android開發也差不多,app每次complie完再操作,再調整,再重來一次complie執行…一直循環

能即時更新這點……真的節省非常多的時間!

但!!! 這次使flutter開發卻是有始以來花費時間最久的,Dart…Flutter Framework,Widget.......開發總時間已經逼進三星期以上,現在也只不過開發80%而己。

1、圖片 要能兩指縮放,最終網上有大神有做出plug,但為了套到去,就花了一星期以上,先參考官方demo app,移值失敗,用大神才成功。好處:終於離開底層的縮放,也不是css或javascript假縮放。

2、搜尋 看了一堆demo後,官方有比較推的方式,照做,照完後,發現是有BUG的,可能官網的做法,最後的搜尋完的點選,是在同一頁,但實際肯定會nav push到新頁,回到搜尋頁後,就不能搜尋了,debug查了整整一天,試了各種方式後,放棄! 改找網路上土砲法,自己做搜尋在header,然後再參考官方控制操作,這樣終於改完了。這也是一周以上的時間。

3、文章式的顯示:多塊區域組合,這裡也是花一堆時間,ListView怎麼處理widget [] 和chrild、chrildren等等,這裡正式踢鐵版,卻更了解上述一些架構上的問題。

改版到這裡就花了三星期以上的時間,這樣開發不是一次三星期,而是下班後斷斷續續的開發,到現在還是沒開發完畢,以之前的開發經驗,接下來肯定有

1、admob
2、androdi ios 實際創建app問題 android應該最少,ios最多

真的開發的話,有時間壓力的話,ionic還是最快的,但為了app畫面精緻度,也許Flutter………

ethereum-etl ethereumetl elk logstash kibana


all output columns with logstash



filter {
  if [srctype] == "etl" { #[fields][srctype]
    csv {
      columns => [
        "number", "hash",  "parent_hash",  "nonce",  "sha3_uncles",  "logs_bloom",  "transactions_root",
        "state_root",  "receipts_root",  "timestamp",  "extra_data",  "transaction_count",  "gas_limit",
        "size",  "total_difficulty",  "difficulty",  "miner",  "block_hash",  "block_number",
        "transaction_index",  "from_address",  "to_address",  "value",  "gas",  "gas_price",  "input",
        "address",  "bytecode",  "function_sighashes",  "is_erc20",  "is_erc721",  "log_index",
        "transaction_hash",  "data",  "topics",  "cumulative_gas_used",  "gas_used",  "contract_address",
        "root,status"
      ]
      separator => ","
      remove_field => ["message"]
      #autodetect_column_names => true   #have problems
      #autogenerate_column_names => true #have problems
      skip_empty_columns => true
      skip_empty_rows => true
    }
  }

logstash fileds if



https://sueboy.blogspot.com/2018/11/elk60filebeatdocumenttype.html


filebeat.yml


- type: log
  paths:
    - /var/log/geth.log
  exclude_files: ['.gz$']

  fields:
    srctype: "geth"



pipleline logstah.conf

if [fields][srctype] == "geth" {


BUT fields_under_root: true 


- type: log
  paths:
    - /var/log/geth.log
  exclude_files: ['.gz$']

  fields:
    srctype: "geth"
  fields_under_root: true

if [srctype] == "geth" {

ethereum-etl ethereumetl docker



docker-compose.yml


version: '3.3'

services:
  ethereum_etl:
    build:
      context: .
    volumes:
      - ./var/log/:/ethereum-etl/output:rw
      #- /root/go/src/github.com/ethereum/go-ethereum/build/bin/data:/ethereum-etl/ipc #failed ipc is interface not data
    networks:
      - etl

networks:
  etl:
    driver: bridge


DOCKERFILE


FROM python:3.6-alpine
MAINTAINER Eric Lim 
ENV PROJECT_DIR=ethereum-etl

RUN mkdir /$PROJECT_DIR
WORKDIR /$PROJECT_DIR
RUN apk add --no-cache gcc musl-dev  #for C libraries:  
RUN pip install --upgrade pip && pip install ethereum-etl

#ENTRYPOINT ["python", "ethereumetl"]
ENTRYPOINT ["ethereumetl"]
CMD ["export_all", "-s", "0", "-e", "500000", "-p", "http://xxx.xxx.xxx.xxx:8545", "-o", "output"]


-s -e -p -o see
https://github.com/blockchain-etl/ethereum-etl

Import:-p  must give port. http 80, https 443 so custom geth port usually 8545


--rpcaddr "0.0.0.0" or special ip


geth --datadir ./gethdata --networkid xxxxxx --rpcapi "db,eth,net,web3,personal" --rpc --rpccorsdomain "*" --rpcaddr "0.0.0.0" --port 30303 console