https://kubernetes.github.io/ingress-nginx/examples/docker-registry/
https://sueboy.blogspot.com/2019/11/kubernetes-nodeport.html
PS：Close basic auth just remark configmap.yml auth:
#auth:
# htpasswd:
# realm: basic-realm
# path: /auth/htpasswd
README.md
# Docker Registry K8s
Domain：docker-registry
Port:30500
/etc/hosts add 192.168.99.118 docker-registry ## ip is k8s host eth2's ip
Finish command： docker login -u username -p password docker-registry:30500
https://docs.docker.com/registry/insecure/
Deploy a plain HTTP registry
```bash
/etc/docker/daemon.json
{
"insecure-registries" : ["docker-registry:30500"]
}
Restart Docker for the changes to take effect
```
https://blog.csdn.net/zsd498537806/article/details/79290732
https://serverfault.com/questions/611120/failed-tls-handshake-does-not-contain-any-ip-sans
http://dockone.io/article/684
#-subj "/C=/ST=/L=/O=/OU=/CN=192.168.99.118"
-subj "/C=/ST=/L=/O=/OU=/CN=docker-registry"
## Generate private key & certificate
```bash
openssl req -nodes -newkey rsa:4096 -keyout certs/domain.key -out certs/domain.csr -subj "/C=/ST=/L=/O=/OU=/CN=docker-registry"
openssl x509 -req -sha256 -days 365 -in certs/domain.csr -signkey certs/domain.key -out certs/domain.crt
#> openssl req \
# -newkey rsa:4096 -nodes -keyout certs/domain.key \
# -x509 -days 365 -out certs/domain.crt
```
## login ID/PWD
```bash
docker run --rm --entrypoint htpasswd registry:2 -Bbn username password > htpasswd
```
## K8s Secret
```bash
cat domain.crt | base64
cat domain.key | base64
```
No bash64, just online
### domain.crt base64
```bash
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVzRENDQXBnQ0NRRHpteitOQUZCMGRUQU5C
Z2txaGtpRzl3MEJBUXNGQURBYU1SZ3dGZ1lEVlFRRERBOWsKYjJOclpYSXRjbVZuYVhOMGNua3dI
aGNOTVRrd01qSTJNVEF3TlRBMldoY05NakF3TWpJMk1UQXdOVEEyV2pBYQpNUmd3RmdZRFZRUURE
QTlrYjJOclpYSXRjbVZuYVhOMGNua3dnZ0lpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElDCkR3QXdn
Z0lLQW9JQ0FRRFBKcHhZK0RVQ05aMVh4Tkc0bUJlaFp2ZXZ4R21XMFdZandhRklzcFQ0L09ETWgw
ZVgKRXRmT0xpaUZibXFzMi93L01CeWlUSWlZczltMFdhdG5uVDc5bm5wZUVHdFNtQUJHeXMyU1hx
dDZ3SzJubHJSLwowVllaVEpJQ2t2amE5WExtQ2w2eEgrbFFxWVFnNHlHSWxuZ2ZXUFBCTzkvVFhn
S2Jmc2xBTk9uTXY0ckVTMlNFCkZLQzNuSTRqY1JWc1BQSUNkNDNJVlU1akNiRllublZJS2l2eTdq
Y0l0Q2JoelZsTHRuOHFIRE5YcElHSlpTTlYKUkdSdUVhN3ZFbUloei90blJZdFJRTmpSNGhQNkM5
eG1QMFpBZFhjQWZYMTk0aVVKNDRFYnZUZ0NKK0I4Yi9jdwpEZ29IRWxCbGtJaUtmSm9wL0RhR3B5
OUY0VjRhbHZIcnNMdFdYYTZYWmQzeGxmWWx6S2xMZ0IzTDJVQ2VCUWljCk0yUDlXMEZZZGFIR3Nj
UkFyNGQvbEtiZE1PYXNDV2dXaHlGekRDMUFYQkcxdU9SSXFBbi84OTJPQnk1UVE2SDQKSk9yOXNR
RGhRejVEdWZReFlGNDZUZjZUNk9MOGp6TkUrUk11YmR2RmVVS1ZwYzJXazgrVWY3UE9OdW9iRC9X
ZgpTRFVCYjNsSCtGVTFZalVhR0Y5VzNwSE14NXZzTU8xOFJ3UTIzU2ZFT1dMZFFseU8rYXg3STZK
Vm5CbG1xaStFCk16QjFHMzVTbmlMMWcyVDRkbFlIWThIbllaU1A3RFdYR3JrZ2l6SloweUUveE55
dkJuLy9GeWtIYTlIdjBhUk0KNE45OUhHRkthTmNzdkFDR2NSU2QvYUdHUDdqWHNEb3R2R1U5VjU4
ems2S21oVGRjN3V0cE9Hb0Nud0lEQVFBQgpNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUNBUUM4U1or
N0V3U0liY0tDT0xoV3NvbDEzNUVIYVdiRTEzOFRRdHBVCk55aExqZXM3THVQSjg3YVdjK0NWT3VM
WEdIY2FtTiszQWhKL1k3L1U5NlVCWS8yTDREUnd6b0RDYzhhQ3ZQY0oKZDhWcGxvUXJuRzRueUgx
VTJUV243SXo1UDNrYjVZeE82UHlVdlNZMTJTaXBxWmVuQU1STzVQSDRlUldwbDkwbAp3cHJDVU1T
NGhaL3FGVE1ORG1HSzBGS243VHZRekZrSVo1eEhUK0VncTNzcERVY1BCenV4WVdGTG9pT2ZGUFRZ
CnZkSDc5Vis5K1p6T3ZBVlF3ZHhHSVlOaGlpNGd1OTg4N3Jld3RKbEZUQ1pXT05DWnNMcUw0RHBt
WmJhbUlxRi8KakF5ZUR4T2kwN2VKcTlUREFLUlErZzg1NzMvR2JhVU5MVHVndzM0Z1pNZUttUnNz
UjBEZWc5ckFHeXRRM2pqRApqeWJLYmo1SE8vT25xNTlGSk9rRi94eklZYWtXcjVPem1kRzJDdVhN
OXhSN3BYSm5IM0RKcWUwc2FQd2ZMV2xPClF3L1RRM1lpMHg4ajhwOTRpeHZWbnRPRFRIUXBmeUFD
MDloaFNmVG1sMnJXT3FqTzJ4d1dzUkhEOXBUQ0FpMG0KTHlZQS9UTVVXRWdMdklmMEZjOXRZZGkr
ck9HQVJuRWdXdjdhRjhKL0pRUlZ2ZjBVL2hGTmo4bFRmY0RuOU95WQprVEtWdHNRVllqZXNBU1lI
dzYyRmZoUkRIZmZRbUlscTN3emljU3MvbnBQMzZraExCZGt0TnZzakVNaE40a0JLCjR5NlJWekdZ
R20rakpVQk8zV3ZQbHF0T0pwVUFCbHVCWFZOK3hhMnZOODNGYjZtMFlJTmVWbXNFQmxnVTVGT1oK
a2xFVTJ3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
```
### domain.key base64
```bash
LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUpRd0lCQURBTkJna3Foa2lHOXcwQkFRRUZB
QVNDQ1Mwd2dna3BBZ0VBQW9JQ0FRRFBKcHhZK0RVQ05aMVgKeE5HNG1CZWhadmV2eEdtVzBXWWp3
YUZJc3BUNC9PRE1oMGVYRXRmT0xpaUZibXFzMi93L01CeWlUSWlZczltMApXYXRublQ3OW5ucGVF
R3RTbUFCR3lzMlNYcXQ2d0sybmxyUi8wVllaVEpJQ2t2amE5WExtQ2w2eEgrbFFxWVFnCjR5R0ls
bmdmV1BQQk85L1RYZ0tiZnNsQU5Pbk12NHJFUzJTRUZLQzNuSTRqY1JWc1BQSUNkNDNJVlU1akNi
RlkKbm5WSUtpdnk3amNJdENiaHpWbEx0bjhxSEROWHBJR0paU05WUkdSdUVhN3ZFbUloei90blJZ
dFJRTmpSNGhQNgpDOXhtUDBaQWRYY0FmWDE5NGlVSjQ0RWJ2VGdDSitCOGIvY3dEZ29IRWxCbGtJ
aUtmSm9wL0RhR3B5OUY0VjRhCmx2SHJzTHRXWGE2WFpkM3hsZllsektsTGdCM0wyVUNlQlFpY00y
UDlXMEZZZGFIR3NjUkFyNGQvbEtiZE1PYXMKQ1dnV2h5RnpEQzFBWEJHMXVPUklxQW4vODkyT0J5
NVFRNkg0Sk9yOXNRRGhRejVEdWZReFlGNDZUZjZUNk9MOApqek5FK1JNdWJkdkZlVUtWcGMyV2s4
K1VmN1BPTnVvYkQvV2ZTRFVCYjNsSCtGVTFZalVhR0Y5VzNwSE14NXZzCk1PMThSd1EyM1NmRU9X
TGRRbHlPK2F4N0k2SlZuQmxtcWkrRU16QjFHMzVTbmlMMWcyVDRkbFlIWThIbllaU1AKN0RXWEdy
a2dpekpaMHlFL3hOeXZCbi8vRnlrSGE5SHYwYVJNNE45OUhHRkthTmNzdkFDR2NSU2QvYUdHUDdq
WApzRG90dkdVOVY1OHprNkttaFRkYzd1dHBPR29DbndJREFRQUJBb0lDQVFDb0FkZXliUFNKS3pU
R3Q1R2tsbTRyCjhKZ1lUVU9hRjVFd2xOMmJXWU5yZ2FWTkhMYmFkMlRpQ3h2QVJISnVhaWpta3Iz
V2dCNFZRMkRoWGd5Zmg3Qk0KQUZtY3F0SndkV2xjQVNYRGZkYTBRMHRVYmUxTThWcFVRSDczYVg0
K0QrNDFWdG0vN0hMTmdmN3dzV1pFSWsyZwpsd056aUgwYkw5Y0xpRVB2T0RzYlhZcE5kQnc4NTRJ
R0NRcGJBbDl1bnV0bFJ0T09WR216OTJ1SHhWUzN4OFJIClRoOENoeUkxNXRkeWlid1BTam4xN2kx
Rll1cktRQ0w0Qm8yTHhxd2lZazJ6VDNzVDFuVkRuSW01Tmg5MktnRnYKVEJYTkp2bzl1OGx3Z0x6
NDQzbG1mak1hbWJDd24rSlFJL0Zpdmx1UGZrbUduZEVYV3NjRGRTcGVoR2dIVVJwTwpGWURKU29j
aTlGVG5NRTF4a3c4TUR5V1JjSzdvSjJrRzd0RXh4aDBML1NjK1dmY2RKVFJyRW9BR3VxNzNYdWZG
Ck1CdVRoK3JCVU9ITVpSTVJMTXN6aGVTekp1SXBZU1BsNUF1VUY5cHVNd21vYkFpbXBzdmpQVEpr
RlR3bytLdmQKYUVlNW5IMFlmTlRNS3JSU1YybjBaTmp0dHB3VU9VZ2lXVE9VdnJXOGNiMlVJZjRp
RVk3VjJBc3ArKzVvK25QdQpZRkdiZWd0d1U3dURyNTN2YjNPRE9ZRVUrUlZmYTlScWxXMUswYlpm
eGZBcCtzdVNQTHczclBiNk4xZ0UybWhFCnNUcy9FYmQ5dW5Vdm1XZ0tzdGtBeDBuRVM4Z1VqNW9Z
c1F5Um1wM1NaaUJXUmlXYnZoSzRaQ3k0V3NaZUZWZEoKMEtuNDZCeXB1Sndtelozd2hRMDBZUUtD
QVFFQS9XODlsa0c1eW9UVXlEZnI3M0hqMHNyZ0o3amlLdlNoZ29PVwpCck1ndkxxeEMzUVRaZjg2
VHpObnhoVDFwSS9sUnNRN2tiYjJhclpHZ3phRHlNemV2QXpYMUJiNnRkakxnVVIvCi9ZMTR1QW9V
VzhYNEhrY0Rhb2xtVkFjOXdsRmpJcHVjY25YdEZJZ3NacnVydEJSQW9mSU91aUV4bzJCQkc3aUEK
MW1qN2RqZlZwSTQzWWd4UHBiZndmVjV0bC9CTENMQis0RzZkclhvaExBdTlqSWlKNWw4MEEydG1q
R3Z3S3lVTApHc3RGamc1MVcxTjQyTC8rY0FSQ213a3pUUTkwMFZaS1BaOUdtZTE2NndZdjl4eXds
ZjN6L09udGJsZjRLaXE4ClVVQXR3RFF3bWFwN3o1OUhpckp1TEczMDk3QWRlNTZjQ1JOMk53c0Yz
YTJQM09VeFVRS0NBUUVBMFQ5dHlzK24KVHdkd0JNK0lRcVBQUWt0ZjhodENSUW1Wamw0WWVZc1pt
R0F2TXRqN00wYVhjcnVLT0k2VjBmc0k2T0RrbWRDSQpnZWYwUk15WXR3czdMcGtXWmIyQzlFdzk0
K3VpL2tEQVlxUUJXczBVaTQwbTdNeDgwN2pKdGZONVJad3d1MGxNCm1kR1ZSa3hkbWVtM0V6Mmhs
R2ltZUMvcnB5REtiMkY1SU9Mdjk1cUdlWHBlOGNwUDhVMDd6dTdqdmdDNVRSeDcKV0pYYVVFSEt4
bjVHVHU2UUluMXQrZmw5MnRnNXprNmNjaklUeStvUG4yNWU1ZnR3ZlNaVzg1Tno2dFQzRE04bwpM
Z0h4aFBNS0RzcjZySnVRSExObHI1MnY4TDlwVlZlWlpOanlmbDNxUmhqOE5zanN4OEhCczV1Z1ZF
YU9MWkJTClFrU1pCYzNkYldONDd3S0NBUUFMWVhaemkrb0s5U0lWQjZYWnI5U2FQN0FIT0o4M2JK
cmdJSWFaTUd3VTdaRkwKd1YrcG9lZDliUUtJKzZINFRObnY3cG42Tml3a3FLREwvYU8zWGJQVzRB
REZMbTc5d1ltSitZaUIyMXVEY0RodgowcGtNeXdXVlFDeFBNN0FMVE5wVjBJVWJBcjZ0d0Z1MkZW
RmxtYzg1Tm9TQXlVTFFZbDhxSG9QeW5PNUY1Y1g3ClNNT2w3SVdiSElKNUxzZGZIbFJRUkpTK0FW
bWxVNm12cjZ0SVowMUNBaDUwMU5yYkRYSFFqaTVNZXJwWjRiNVYKWm84enRKTjVka2VEYk10WDk0
eTRaZGFabzlCbzB5aEU2NWMxMHB6L2IxTUlsWHlyQ3RwREVuNDRhOWhpSERYZQpnTEJrcDNOd0FB
bEFoNE9McCtYN0NpbFpvdjBaQmE2eXdMdUlJT2F4QW9JQkFILytaSkppWXQwY2w1RFp3V2ZCCis4
MTVmYjhDblk3TlJnWkgySURBYlcrZnBqOWNLb3c2NmZqc2JxZ1p6OXNaeTJGbjcyMkcrWGs5QVZY
ME1KWUcKOExTWVAvTUw4VWF1UEUxbW5kVVlscFcvN0hRMmJoTE96TGljZk42bGNmMHlhVVFEUG12
UmNtSEVxYUxPbzVLNgpPNW94WU56L0JwVU4rdi9JTnd2YWU5VjUrMzlocmxBd1lsTlVoWUg4MUZE
SUg5YmhhbmVZckhxTGIzY1lSNGtUClJMNlRmdWNhQW1PZUVUWUVmWmVNem5XcjNDQW5sRmJabElM
cFF1M0t2ZlZRaE45b1ZaWW9RVk9qTExFK1dUQ2oKU0tLOHBYSm82aEo2U2x6aE9QVlRHQ2x2NnlV
UkhYU3pZOU8vSTljczFlZWZrbnI2alkyaGNpL0c5ak51c2J0dgoxVkVDZ2dFQkFJVUdEbEcxQmpP
dTVhcVZ1eDZzTzUwbHB4eVpienFvMm1TRVMwZHNYNDcvclN6ZGRoblE4QnVVCjlURHF3aDBlM0pZ
dGlUczM4Z21RT1pVdndEM2FnNjdDU0t1T3FNUWM3aU00S01rbmdrakxOMnhIM2luYjZxUjMKc0ZL
YVluWDlMdTBZeVVZU013cGwyNjM4ZHoxanY3a05wRE8xd2p0bTh5TjdxQk00d3IrU3JNV0EyWHlh
SG1hZgpBZE5hNzdxdHppL0JIcXdVTFZsbDFTSitIU1dqdHU2QmRNQTlLandzMllieWxUTm1UbmZW
TzdLeVMzTFAwY2xCCm1MaEVhYS93d2J1Mm5sTWRxdDJpeTQzcE5QYmhRZ0lOck95dDR6cXhNZEVX
dWtSeHAwSGFieXdFcldFWXJFQ1cKdS90ZDhKZ291QmZqZ0JDQ3hnNjNvWlJVS1ZnU1FQST0KLS0t
LS1FTkQgUFJJVkFURSBLRVktLS0tLQo=
```
### htpasswd base64
```bash
dXNlcm5hbWU6JDJ5JDA1JGprbU5CSm1DVTYyV0pKc0tqcWx3RC5BQmpLcUswc2R0SU9lZ29kb3VrQzFpR0owc05ENjhD
```
### K8s Namespace namespace.json
```json
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "docker-registry",
"labels": {
"name": "test-dev"
}
}
}
```
### K8s Secret secret.yml
```YAML
apiVersion: v1
kind: Secret
metadata:
name: registry-tls
namespace: docker-registry
type: Opaque
data:
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVzRENDQXBnQ0NRRHpteitOQUZCMGRUQU5CZ2txaGtpRzl3MEJBUXNGQURBYU1SZ3dGZ1lEVlFRRERBOWsKYjJOclpYSXRjbVZuYVhOMGNua3dIaGNOTVRrd01qSTJNVEF3TlRBMldoY05NakF3TWpJMk1UQXdOVEEyV2pBYQpNUmd3RmdZRFZRUUREQTlrYjJOclpYSXRjbVZuYVhOMGNua3dnZ0lpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElDCkR3QXdnZ0lLQW9JQ0FRRFBKcHhZK0RVQ05aMVh4Tkc0bUJlaFp2ZXZ4R21XMFdZandhRklzcFQ0L09ETWgwZVgKRXRmT0xpaUZibXFzMi93L01CeWlUSWlZczltMFdhdG5uVDc5bm5wZUVHdFNtQUJHeXMyU1hxdDZ3SzJubHJSLwowVllaVEpJQ2t2amE5WExtQ2w2eEgrbFFxWVFnNHlHSWxuZ2ZXUFBCTzkvVFhnS2Jmc2xBTk9uTXY0ckVTMlNFCkZLQzNuSTRqY1JWc1BQSUNkNDNJVlU1akNiRllublZJS2l2eTdqY0l0Q2JoelZsTHRuOHFIRE5YcElHSlpTTlYKUkdSdUVhN3ZFbUloei90blJZdFJRTmpSNGhQNkM5eG1QMFpBZFhjQWZYMTk0aVVKNDRFYnZUZ0NKK0I4Yi9jdwpEZ29IRWxCbGtJaUtmSm9wL0RhR3B5OUY0VjRhbHZIcnNMdFdYYTZYWmQzeGxmWWx6S2xMZ0IzTDJVQ2VCUWljCk0yUDlXMEZZZGFIR3NjUkFyNGQvbEtiZE1PYXNDV2dXaHlGekRDMUFYQkcxdU9SSXFBbi84OTJPQnk1UVE2SDQKSk9yOXNRRGhRejVEdWZReFlGNDZUZjZUNk9MOGp6TkUrUk11YmR2RmVVS1ZwYzJXazgrVWY3UE9OdW9iRC9XZgpTRFVCYjNsSCtGVTFZalVhR0Y5VzNwSE14NXZzTU8xOFJ3UTIzU2ZFT1dMZFFseU8rYXg3STZKVm5CbG1xaStFCk16QjFHMzVTbmlMMWcyVDRkbFlIWThIbllaU1A3RFdYR3JrZ2l6SloweUUveE55dkJuLy9GeWtIYTlIdjBhUk0KNE45OUhHRkthTmNzdkFDR2NSU2QvYUdHUDdqWHNEb3R2R1U5VjU4ems2S21oVGRjN3V0cE9Hb0Nud0lEQVFBQgpNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUNBUUM4U1orN0V3U0liY0tDT0xoV3NvbDEzNUVIYVdiRTEzOFRRdHBVCk55aExqZXM3THVQSjg3YVdjK0NWT3VMWEdIY2FtTiszQWhKL1k3L1U5NlVCWS8yTDREUnd6b0RDYzhhQ3ZQY0oKZDhWcGxvUXJuRzRueUgxVTJUV243SXo1UDNrYjVZeE82UHlVdlNZMTJTaXBxWmVuQU1STzVQSDRlUldwbDkwbAp3cHJDVU1TNGhaL3FGVE1ORG1HSzBGS243VHZRekZrSVo1eEhUK0VncTNzcERVY1BCenV4WVdGTG9pT2ZGUFRZCnZkSDc5Vis5K1p6T3ZBVlF3ZHhHSVlOaGlpNGd1OTg4N3Jld3RKbEZUQ1pXT05DWnNMcUw0RHBtWmJhbUlxRi8KakF5ZUR4T2kwN2VKcTlUREFLUlErZzg1NzMvR2JhVU5MVHVndzM0Z1pNZUttUnNzUjBEZWc5ckFHeXRRM2pqRApqeWJLYmo1SE8vT25xNTlGSk9rRi94eklZYWtXcjVPem1kRzJDdVhNOXhSN3BYSm5IM0RKcWUwc2FQd2ZMV2xPClF3L1RRM1lpMHg4ajhwOTRpeHZWbnRPRFRIUXBmeUFDMDloaFNmVG1sMnJXT3FqTzJ4d1dzUkhEOXBUQ0FpMG0KTHlZQS9UTVVXRWdMdklmMEZjOXRZZGkrck9HQVJuRWdXdjdhRjhKL0pRUlZ2ZjBVL2hGTmo4bFRmY0RuOU95WQprVEtWdHNRVllqZXNBU1lIdzYyRmZoUkRIZmZRbUlscTN3emljU3MvbnBQMzZraExCZGt0TnZzakVNaE40a0JLCjR5NlJWekdZR20rakpVQk8zV3ZQbHF0T0pwVUFCbHVCWFZOK3hhMnZOODNGYjZtMFlJTmVWbXNFQmxnVTVGT1oKa2xFVTJ3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUpRd0lCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQ1Mwd2dna3BBZ0VBQW9JQ0FRRFBKcHhZK0RVQ05aMVgKeE5HNG1CZWhadmV2eEdtVzBXWWp3YUZJc3BUNC9PRE1oMGVYRXRmT0xpaUZibXFzMi93L01CeWlUSWlZczltMApXYXRublQ3OW5ucGVFR3RTbUFCR3lzMlNYcXQ2d0sybmxyUi8wVllaVEpJQ2t2amE5WExtQ2w2eEgrbFFxWVFnCjR5R0lsbmdmV1BQQk85L1RYZ0tiZnNsQU5Pbk12NHJFUzJTRUZLQzNuSTRqY1JWc1BQSUNkNDNJVlU1akNiRlkKbm5WSUtpdnk3amNJdENiaHpWbEx0bjhxSEROWHBJR0paU05WUkdSdUVhN3ZFbUloei90blJZdFJRTmpSNGhQNgpDOXhtUDBaQWRYY0FmWDE5NGlVSjQ0RWJ2VGdDSitCOGIvY3dEZ29IRWxCbGtJaUtmSm9wL0RhR3B5OUY0VjRhCmx2SHJzTHRXWGE2WFpkM3hsZllsektsTGdCM0wyVUNlQlFpY00yUDlXMEZZZGFIR3NjUkFyNGQvbEtiZE1PYXMKQ1dnV2h5RnpEQzFBWEJHMXVPUklxQW4vODkyT0J5NVFRNkg0Sk9yOXNRRGhRejVEdWZReFlGNDZUZjZUNk9MOApqek5FK1JNdWJkdkZlVUtWcGMyV2s4K1VmN1BPTnVvYkQvV2ZTRFVCYjNsSCtGVTFZalVhR0Y5VzNwSE14NXZzCk1PMThSd1EyM1NmRU9XTGRRbHlPK2F4N0k2SlZuQmxtcWkrRU16QjFHMzVTbmlMMWcyVDRkbFlIWThIbllaU1AKN0RXWEdya2dpekpaMHlFL3hOeXZCbi8vRnlrSGE5SHYwYVJNNE45OUhHRkthTmNzdkFDR2NSU2QvYUdHUDdqWApzRG90dkdVOVY1OHprNkttaFRkYzd1dHBPR29DbndJREFRQUJBb0lDQVFDb0FkZXliUFNKS3pUR3Q1R2tsbTRyCjhKZ1lUVU9hRjVFd2xOMmJXWU5yZ2FWTkhMYmFkMlRpQ3h2QVJISnVhaWpta3IzV2dCNFZRMkRoWGd5Zmg3Qk0KQUZtY3F0SndkV2xjQVNYRGZkYTBRMHRVYmUxTThWcFVRSDczYVg0K0QrNDFWdG0vN0hMTmdmN3dzV1pFSWsyZwpsd056aUgwYkw5Y0xpRVB2T0RzYlhZcE5kQnc4NTRJR0NRcGJBbDl1bnV0bFJ0T09WR216OTJ1SHhWUzN4OFJIClRoOENoeUkxNXRkeWlid1BTam4xN2kxRll1cktRQ0w0Qm8yTHhxd2lZazJ6VDNzVDFuVkRuSW01Tmg5MktnRnYKVEJYTkp2bzl1OGx3Z0x6NDQzbG1mak1hbWJDd24rSlFJL0Zpdmx1UGZrbUduZEVYV3NjRGRTcGVoR2dIVVJwTwpGWURKU29jaTlGVG5NRTF4a3c4TUR5V1JjSzdvSjJrRzd0RXh4aDBML1NjK1dmY2RKVFJyRW9BR3VxNzNYdWZGCk1CdVRoK3JCVU9ITVpSTVJMTXN6aGVTekp1SXBZU1BsNUF1VUY5cHVNd21vYkFpbXBzdmpQVEprRlR3bytLdmQKYUVlNW5IMFlmTlRNS3JSU1YybjBaTmp0dHB3VU9VZ2lXVE9VdnJXOGNiMlVJZjRpRVk3VjJBc3ArKzVvK25QdQpZRkdiZWd0d1U3dURyNTN2YjNPRE9ZRVUrUlZmYTlScWxXMUswYlpmeGZBcCtzdVNQTHczclBiNk4xZ0UybWhFCnNUcy9FYmQ5dW5Vdm1XZ0tzdGtBeDBuRVM4Z1VqNW9Zc1F5Um1wM1NaaUJXUmlXYnZoSzRaQ3k0V3NaZUZWZEoKMEtuNDZCeXB1Sndtelozd2hRMDBZUUtDQVFFQS9XODlsa0c1eW9UVXlEZnI3M0hqMHNyZ0o3amlLdlNoZ29PVwpCck1ndkxxeEMzUVRaZjg2VHpObnhoVDFwSS9sUnNRN2tiYjJhclpHZ3phRHlNemV2QXpYMUJiNnRkakxnVVIvCi9ZMTR1QW9VVzhYNEhrY0Rhb2xtVkFjOXdsRmpJcHVjY25YdEZJZ3NacnVydEJSQW9mSU91aUV4bzJCQkc3aUEKMW1qN2RqZlZwSTQzWWd4UHBiZndmVjV0bC9CTENMQis0RzZkclhvaExBdTlqSWlKNWw4MEEydG1qR3Z3S3lVTApHc3RGamc1MVcxTjQyTC8rY0FSQ213a3pUUTkwMFZaS1BaOUdtZTE2NndZdjl4eXdsZjN6L09udGJsZjRLaXE4ClVVQXR3RFF3bWFwN3o1OUhpckp1TEczMDk3QWRlNTZjQ1JOMk53c0YzYTJQM09VeFVRS0NBUUVBMFQ5dHlzK24KVHdkd0JNK0lRcVBQUWt0ZjhodENSUW1Wamw0WWVZc1ptR0F2TXRqN00wYVhjcnVLT0k2VjBmc0k2T0RrbWRDSQpnZWYwUk15WXR3czdMcGtXWmIyQzlFdzk0K3VpL2tEQVlxUUJXczBVaTQwbTdNeDgwN2pKdGZONVJad3d1MGxNCm1kR1ZSa3hkbWVtM0V6MmhsR2ltZUMvcnB5REtiMkY1SU9Mdjk1cUdlWHBlOGNwUDhVMDd6dTdqdmdDNVRSeDcKV0pYYVVFSEt4bjVHVHU2UUluMXQrZmw5MnRnNXprNmNjaklUeStvUG4yNWU1ZnR3ZlNaVzg1Tno2dFQzRE04bwpMZ0h4aFBNS0RzcjZySnVRSExObHI1MnY4TDlwVlZlWlpOanlmbDNxUmhqOE5zanN4OEhCczV1Z1ZFYU9MWkJTClFrU1pCYzNkYldONDd3S0NBUUFMWVhaemkrb0s5U0lWQjZYWnI5U2FQN0FIT0o4M2JKcmdJSWFaTUd3VTdaRkwKd1YrcG9lZDliUUtJKzZINFRObnY3cG42Tml3a3FLREwvYU8zWGJQVzRBREZMbTc5d1ltSitZaUIyMXVEY0RodgowcGtNeXdXVlFDeFBNN0FMVE5wVjBJVWJBcjZ0d0Z1MkZWRmxtYzg1Tm9TQXlVTFFZbDhxSG9QeW5PNUY1Y1g3ClNNT2w3SVdiSElKNUxzZGZIbFJRUkpTK0FWbWxVNm12cjZ0SVowMUNBaDUwMU5yYkRYSFFqaTVNZXJwWjRiNVYKWm84enRKTjVka2VEYk10WDk0eTRaZGFabzlCbzB5aEU2NWMxMHB6L2IxTUlsWHlyQ3RwREVuNDRhOWhpSERYZQpnTEJrcDNOd0FBbEFoNE9McCtYN0NpbFpvdjBaQmE2eXdMdUlJT2F4QW9JQkFILytaSkppWXQwY2w1RFp3V2ZCCis4MTVmYjhDblk3TlJnWkgySURBYlcrZnBqOWNLb3c2NmZqc2JxZ1p6OXNaeTJGbjcyMkcrWGs5QVZYME1KWUcKOExTWVAvTUw4VWF1UEUxbW5kVVlscFcvN0hRMmJoTE96TGljZk42bGNmMHlhVVFEUG12UmNtSEVxYUxPbzVLNgpPNW94WU56L0JwVU4rdi9JTnd2YWU5VjUrMzlocmxBd1lsTlVoWUg4MUZESUg5YmhhbmVZckhxTGIzY1lSNGtUClJMNlRmdWNhQW1PZUVUWUVmWmVNem5XcjNDQW5sRmJabElMcFF1M0t2ZlZRaE45b1ZaWW9RVk9qTExFK1dUQ2oKU0tLOHBYSm82aEo2U2x6aE9QVlRHQ2x2NnlVUkhYU3pZOU8vSTljczFlZWZrbnI2alkyaGNpL0c5ak51c2J0dgoxVkVDZ2dFQkFJVUdEbEcxQmpPdTVhcVZ1eDZzTzUwbHB4eVpienFvMm1TRVMwZHNYNDcvclN6ZGRoblE4QnVVCjlURHF3aDBlM0pZdGlUczM4Z21RT1pVdndEM2FnNjdDU0t1T3FNUWM3aU00S01rbmdrakxOMnhIM2luYjZxUjMKc0ZLYVluWDlMdTBZeVVZU013cGwyNjM4ZHoxanY3a05wRE8xd2p0bTh5TjdxQk00d3IrU3JNV0EyWHlhSG1hZgpBZE5hNzdxdHppL0JIcXdVTFZsbDFTSitIU1dqdHU2QmRNQTlLandzMllieWxUTm1UbmZWTzdLeVMzTFAwY2xCCm1MaEVhYS93d2J1Mm5sTWRxdDJpeTQzcE5QYmhRZ0lOck95dDR6cXhNZEVXdWtSeHAwSGFieXdFcldFWXJFQ1cKdS90ZDhKZ291QmZqZ0JDQ3hnNjNvWlJVS1ZnU1FQST0KLS0tLS1FTkQgUFJJVkFURSBLRVktLS0tLQ==
HTPASSWD: dXNlcm5hbWU6JDJ5JDA1JGprbU5CSm1DVTYyV0pKc0tqcWx3RC5BQmpLcUswc2R0SU9lZ29kb3VrQzFpR0owc05ENjhD
```
## K8s ConfigMap configmap.yml
```YAML
apiVersion: v1
kind: ConfigMap
metadata:
name: docker-registry
namespace: docker-registry
data:
registry-config.yml: |
version: 0.1
log:
fields:
service: registry
storage:
delete:
enabled: true
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/lib/registry
http:
addr: :5000
headers:
X-Content-Type-Options: [nosniff]
Access-Control-Allow-Origin: ['*']
Access-Control-Allow-Methods: ['HEAD', 'GET', 'OPTIONS', 'DELETE']
Access-Control-Allow-Headers: ['Authorization']
Access-Control-Max-Age: [1728000]
Access-Control-Allow-Credentials: [true]
Access-Control-Expose-Headers: ['Docker-Content-Digest']
auth:
htpasswd:
realm: basic-realm
path: /auth/htpasswd
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
```
## K8s Deployment deployment.yml
```YAML
apiVersion: apps/v1
kind: Deployment
metadata:
name: registry
namespace: docker-registry
spec:
replicas: 2
selector:
matchLabels:
app: registry
template:
metadata:
labels:
app: registry
spec:
#restartPolicy: Always
volumes:
- name: config
configMap:
name: docker-registry
items:
- key: registry-config.yml
path: config.yml
- name: htpasswd
secret:
secretName: registry-tls
items:
- key: HTPASSWD
path: htpasswd
- name: registry-data
emptyDir: {}
containers:
- name: registry
image: registry:2.7.1
ports:
- name: http
containerPort: 5000
protocol: TCP
volumeMounts:
- name: config
mountPath: /etc/docker/registry
readOnly: true
- name: htpasswd
mountPath: /auth
readOnly: true
- name: registry-data
mountPath: /var/lib/registry
```
## K8s Service service.yml
```YAML
apiVersion: v1
kind: Service
metadata:
name: docker-registry
namespace: docker-registry
spec:
type: ClusterIP
ports:
- name: http
protocol: TCP
port: 5000
targetPort: 5000
selector:
app: registry
```
## K8s Ingress ingress.yml
```YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: docker-registry
namespace: docker-registry
annotations:
#kubernetes.io/ingress.class: nginx
#ingress.kubernetes.io/rewrite-target: /
#nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
#certmanager.k8s.io/issuer: acme-issuer
spec:
#tls:
#- hosts:
# - tt.t
# secretName: registry-tls
rules:
- host: tt.t
http:
paths:
- backend:
serviceName: docker-registry
servicePort: 5000
path: /
#- path: /foo
# backend:
# serviceName: service1
# servicePort: 4200
```
yml
kubectl create -f xxx.yml
kubectl edit xxx ooo
example：
kubectl create -f secret.yml
kubectl edit secrets --namespace=docker-registry
kubectl edit secrets registry-tls --namespace=docker-registry
kubectl create -f configmap.yml
kubectl edit configmap --namespace=docker-registry
kubectl edit configmap docker-registry --namespace=docker-registry
PS
containers.volumeMounts mount path must different path. Same path get error: .mountPath: Invalid value: ooxxooxx : must be unique
So configmap
auth.htpasswd.path /etc/docker/registry/htpasswd -> /auth/htpasswd
Vaild
1. check Pod realy get config.yml & htpasswd is ok or not? Pod ip use minikube dashboard website more easy.
> minikube ssh
> curl -v -XGET -u username:password http://172.17.xxx.xxx:5000/v2/_catalog
get 200 OK.
2. Check Service is ok or not ? Service ip use minikube dashboard website more easy.
> minikube ssh
> curl -v -XGET -u username:password http://10.99.xxx.xxx:5000/v2/_catalog
get 200 OK.
3. 2. Check Ingress is ok or not ? Ingress ip use minikube dashboard website more easy.
> minikube ssh
> curl -v -XGET -u username:password http://10.0.xxx.xxx:5000/v2/_catalog
get 200 OK.
========
Ingress
have problems with windows minikube. So now check list：
1、Use it is ingress-nginx
https://kubernetes.github.io/ingress-nginx/deploy/
2、On ingress-nginx deploy minikube
Don't USE
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml
USE
minikube addons enable ingress
Because nginx-control already be create by "minikube addons enable ingress". Check use
kubectl get pods --all-namespaces -l app.kubernetes.io/name=nginx-ingress-controller
PS：Use mandatory.yaml be create ingress-nginx namespace. Use minikube addons be put in kube-system namespace.
https://medium.com/@Oskarr3/setting-up-ingress-on-minikube-6ae825e98f82
Enabling the add-on provisions the following:
．a configMap for the Nginx loadbalancer
．the Nginx ingress controller
．a service that exposes a default Nginx backend pod for handling unmapped requests.
Best Important!!
Use nginx-ingress that let everything pass nginx-ingress. Thinking Way now is "path" or "domain" to "k8s service"
Here use "domain" to "k8s service docker-registry"
EX： tt.t -> nginx-ingress -> service(docker-registry)
Windows use notepad(administrator mode) change C:\Windows\System32\drivers\etc\hosts
192.168.99.xxx tt.t
Run curl
curl -v -XGET -u username:password http://tt.t/
get 200. ok
Best Important!!
Use TLS just remove mark TLS insdie ingress.yml.
Run curl
curl -v -XGET -u username:password https://tt.t/
But maybe get cert can't correct.
Best Important!!
ingress Only domain or path, So http & https. Can't custom port.
https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0
Maybe
you try to edit Deployments -> nginx-ingress-controller YAML
add custom Port
"ports": [
...,
{
"hostPort": 5000,
"containerPort": 5000,
"protocol": "TCP"
}
]
Yes, This edit get No error, but ingress
can't
accept custom port
rules:
- http:
port: 5000
paths:
- backend:
serviceName: docker-registry
servicePort: 5000
path: /
Or
maybe you think Deployments -> nginx-ingress-controller YAML add name for Port. EX:registryhttp 5000
But ingress
can't
accept registryport
rules:
- registryhttp:
paths:
- backend:
serviceName: docker-registry
servicePort: 5000
path: /
Other way is LoadBalancer
But only nodePort
apiVersion: v1
kind: Service
metadata:
name: docker-registry-lb
namespace: docker-registry
spec:
type: LoadBalancer
ports:
- name: registryhttp
port: 5000
targetPort: 5000
nodePort: 30500
selector:
app: registry
apiVersion: v1
kind: Service
metadata:
name: docker-registry
namespace: docker-registry
spec:
type: NodePort
ports:
- name: registryhttp
port: 5000
targetPort: 5000
nodePort: 30500
selector:
app: registry
Port 5000 still
can't use
Only Port 30500 can use
maybe you think
Use "selector" to focus nginx-ingress-controller
Answer is can't, because ingress only http & https
OK, some demo add port on tcp-services (configmap)
https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/
https://yq.aliyun.com/articles/603225
apiVersion: v1
data:
"9000": docker-registry/docker-registry:5000
kind: ConfigMap
metadata:
creationTimestamp: "2019-01-30T05:24:36Z"
labels:
addonmanager.kubernetes.io/mode: EnsureExists
name: tcp-services
namespace: kube-system
resourceVersion: "256144"
selfLink: /api/v1/namespaces/kube-system/configmaps/tcp-services
uid: 552a9b32-244f-11e9-b70e-080027bbc6ea
.........................................................N o ..... U s e
Default backend Test
https://kubernetes.github.io/ingress-nginx/user-guide/default-backend/
．/healthz that returns 200
．/ that returns 404
