投稿
  1. 首页
  2. 随笔

Containerd客户端工具（CLI）介绍ctr,nerdctl,crictl，podman以及docker

防结块剂•2022-12-17•随笔•阅读22

Containerd客户端工具（CLI）介绍ctr,nerdctl,crictl，podman以及docker,第1张

Containerd客户端工具（CLI）介绍ctr,nerdctl,crictl，podman以及docker

下面设计到私有镜像库私有镜像库 http和https两种方式

 

在 containerd v1.5 中为ctr 客户端（管理员/开发人员的容器工具）、容器镜像服务客户端和 CRI 客户端（例如kubectl和crictl).

ctr,crictl,nerdctl：都是containerd客户端，但containerd的客户端工具ctr及crictl使用不方便，containerd 全新的一个客户端工具nerdctl  Releases · containerd/nerdctl · GitHub  #文章 Title | #网站标题 

crictl,nerdctl：可以使用它来检查和调试 k8s 节点上的容器运行时和应用程序。

接下来就是crictl的的常见命令，其中能完全替代docker命令的参照下列表格

*** 作crictldocker查看运行容器crictl psdocker ps查看镜像crictl imagesdocker images查看容器日志crictl logsdocker logs登陆容器控制台crictl execdocker execpull镜像crictl pulldocker pull容器启动/停止crictl start/stopdocker start/stop容器资源情况crictl statsdocker stats

可以看到crictl对容器生命周期的管理基本已经覆盖，不过在crictl我们不能完成 *** 作也比较多，比如对镜像的管理就不属于它的管理范围。这部分还得依靠ctr来实现， *** 作方式同样可以参照下表

*** 作ctrdocker查看镜像ctr images lsdocker images镜像导入/导出ctr images import/exporterdocker load/save镜像拉取/推送ctr images pull/pushdocker pull/push镜像tagctr images tagdocker tag 私有仓库的使用

1. crictl使用：k8s/k3s集群在node通过kubelet节点下载镜像也是用的这个而不是ctr

1.1 第一种情况私有仓库https的且自签名

1.1.1)生成配置文件，然后修改文件

[root@jettoloader containerd]# containerd config default > /etc/containerd/config.toml

[root@jettoloader ~]# cat /etc/containerd/config.toml
disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
version = 2

[cgroup]
  path = ""

[debug]
  address = ""
  format = ""
  gid = 0
  level = ""
  uid = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

[metrics]
  address = ""
  grpc_histogram = false

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = true
    disable_proc_mount = false
    disable_tcp_service = true
    enable_selinux = false
    enable_tls_streaming = false
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "k8s.gcr.io/pause:3.5"
    selinux_category_range = 1024
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    systemd_cgroup = false
    tolerate_missing_hugetlb_controller = true
    unset_seccomp_profile = ""

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
      max_conf_num = 1

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
      no_pivot = false
      snapshotter = "overlayfs"

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          base_runtime_spec = ""
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
            IoUid = 0
            NonewKeyring = false
            NoPivotRoot = false
            Root = ""
            ShimCgroup = ""
            SystemdCgroup = false

      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."nginx.cclinux.cn:5043"]
          endpoint = ["https://nginx.cclinux.cn:5043"]
        #[plugins."io.containerd.grpc.v1.cri".registry.mirrors."nginx.cclinux.cn:5000"]
        #  endpoint = ["http://nginx.cclinux.cn:5000"]
      [plugins."io.containerd.grpc.v1.cri".registry.headers]
      [plugins."io.containerd.grpc.v1.cri".registry.auths]
        [plugins."io.containerd.grpc.v1.cri".registry.auths."https://nginx.cclinux.cn:5043"]
 	  #auth = "XXXXXXXXXXX"
          username = "admin"
          password = "123456aA"
        #[plugins."io.containerd.grpc.v1.cri".registry.auths."http://nginx.cclinux.cn:5000"]
 	#  #auth = "XXXXXXXXXXX"
        #  username = "admin"
        #  password = "123456aA"
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".tls]
          ca_file = "/opt/auth/ca.crt"
  	  #cert_file = "/opt/ssl/harbor.jettech.com.crt"
  	  #key_file = "/opt/ssl/harbor.jettech.com.key"
          #insecure_skip_verify = true
        #[plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".auth]
        #  username = "admin"
        #  password = "123456aA"
        #[plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5000".auth]
        #  username = "admin"
        #  password = "123456aA"
   

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

  [plugins."io.containerd.internal.v1.opt"]
    path = "/opt/containerd"

  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"

  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"

  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false

  [plugins."io.containerd.runtime.v1.linux"]
    no_shim = false
    runtime = "runc"
    runtime_root = ""
    shim = "containerd-shim"
    shim_debug = false

  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["linux/amd64"]

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]

  [plugins."io.containerd.snapshotter.v1.aufs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.btrfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.devmapper"]
    async_remove = false
    base_image_size = ""
    pool_name = ""
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.native"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.overlayfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.zfs"]
    root_path = ""

[proxy_plugins]

[stream_processors]

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_ConFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar"

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_ConFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0
1.1.2 需改的地方： 
    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."nginx.cclinux.cn:5043"]
          endpoint = ["https://nginx.cclinux.cn:5043"]
      [plugins."io.containerd.grpc.v1.cri".registry.headers]
      [plugins."io.containerd.grpc.v1.cri".registry.auths]
                [plugins."io.containerd.grpc.v1.cri".registry.auths."https://nginx.cclinux.cn:5043"]
 	      #auth = "XXXXXXXXXXX"
          username = "admin"
          password = "123456aA"

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".tls]
          ca_file = "/opt/auth/ca.crt"   #根证书 用来生成cert_file 和key_file  
          #cert_file = "/opt/ssl/harbor.jettech.com.crt"  #harbor的crt
          #key_file = "/opt/ssl/harbor.jettech.com.key"  #harbor的key
          #insecure_skip_verify = true  #两者选其一
        [plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".auth]
          username = "admin"
          password = "123456aA"

经过测试：         

         #auth = "XXXXXXXXXXX"
          username = "admin"
          password = "123456aA"

这段认证配置有两个地方可以配置：

1） 在[plugins."io.containerd.grpc.v1.cri".registry.auths]下面

      [plugins."io.containerd.grpc.v1.cri".registry.auths]
          [plugins."io.containerd.grpc.v1.cri".registry.auths."https://nginx.cclinux.cn:5043"]
            username = "admin"
            password = "123456aA"

 2） 在[plugins."io.containerd.grpc.v1.cri".registry.configs]下面

     [plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".auth]
          username = "admin"
          password = "123456aA"

insecure_skip_verify = true 意为跳过证书认证。 

然后测试：

[root@jettoloader containerd]# crictl pull  nginx.cclinux.cn:5043/library/registry:latest
Image is up to date for sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2

注意：ctr是不可以用这个配置文件的，ctr 不使用 CRI，因此它不读取plugins."io.containerd.grpc.v1.cri"配置issues 6285  issues 5407  这个很重要，困惑我很久。

[root@jettoloader containerd]# ctr images pull nginx.cclinux.cn:5043/library/registry:latest
INFO[0000] trying next host                              error="failed to do request: Head "https://nginx.cclinux.cn:5043/v2/library/registry/manifests/latest": x509: certificate signed by unknown authority" host="nginx.cclinux.cn:5043"
ctr: failed to resolve reference "nginx.cclinux.cn:5043/library/registry:latest": failed to do request: Head "https://nginx.cclinux.cn:5043/v2/library/registry/manifests/latest": x509: certificate signed by unknown authorit
1.1.3 如果想用ctr下载上面的镜像方法有两个：

1）参数化：

[root@jettoloader containerd]#  ctr  images pull --user admin:123456aA  --tlscacert=/opt/auth/ca.crt   nginx.cclinux.cn:5043/library/registry:latest
nginx.cclinux.cn:5043/library/registry:latest:                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4: exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0d96da54f60b86a4d869d44b44cfca69d71c4776b81d361bc057d6666ec0d878:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3790aef225b922bc97aaba099fe762f7b115aec55a0083824b548a6a1e610719:    exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5b27040df4a23c90c3837d926f633fb327fb3af9ac4fa5d5bc3520ad578acb10:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:79e9f2f55bf5465a02ee6a6170e66005b20c7aa6b115af6fcd04fad706ea651a:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e2ead8259a04d39492c25c9548078200c5ec429f628dcf7b7535137954cc2df0:    exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.3 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4...
done: 33.847854ms

我应该在其他地方指定这个配置吗？我不想-u user:password --tlscacert每次必须使用 ctr pull 

但是cri没有配置可以配置ca证书？，也没有配置可以设置skip-verify， 这也太弱了吧，觉得不可能阿，google上看到文档里说可以配置:

2） hosts方式：          hosts.toml   详细参考章节2        

1.2 第二种情况 私有仓库http方式的

1.2.1)生成配置文件，然后修改文件

[root@jettoloader ~]# cat /etc/containerd/config.toml
disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
version = 2

[cgroup]
  path = ""

[debug]
  address = ""
  format = ""
  gid = 0
  level = ""
  uid = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

[metrics]
  address = ""
  grpc_histogram = false

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = true
    disable_proc_mount = false
    disable_tcp_service = true
    enable_selinux = false
    enable_tls_streaming = false
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "k8s.gcr.io/pause:3.5"
    selinux_category_range = 1024
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    systemd_cgroup = false
    tolerate_missing_hugetlb_controller = true
    unset_seccomp_profile = ""

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
      max_conf_num = 1

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
      no_pivot = false
      snapshotter = "overlayfs"

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          base_runtime_spec = ""
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
            IoUid = 0
            NonewKeyring = false
            NoPivotRoot = false
            Root = ""
            ShimCgroup = ""
            SystemdCgroup = false

      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        #[plugins."io.containerd.grpc.v1.cri".registry.mirrors."nginx.cclinux.cn:5043"]
        #  endpoint = ["https://nginx.cclinux.cn:5043"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."nginx.cclinux.cn:5000"]
          endpoint = ["http://nginx.cclinux.cn:5000"]
      [plugins."io.containerd.grpc.v1.cri".registry.headers]
      [plugins."io.containerd.grpc.v1.cri".registry.auths]
        #[plugins."io.containerd.grpc.v1.cri".registry.auths."https://nginx.cclinux.cn:5043"]
 	#  #auth = "XXXXXXXXXXX"
        #  username = "admin"
        #  password = "123456aA"
        [plugins."io.containerd.grpc.v1.cri".registry.auths."http://nginx.cclinux.cn:5000"]
 	  #auth = "XXXXXXXXXXX"
          username = "admin"
          password = "123456aA"
           
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        #[plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".tls]
        #  ca_file = "/opt/auth/ca.crt"
      	#  #cert_file = "/opt/auth/harbor.jettech.com.crt"
       	#  #key_file = "/opt/auth/harbor.jettech.com.key"
        #  #insecure_skip_verify = true
        #[plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".auth]
        #  username = "admin"
        #  password = "123456aA"
        #[plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5000".auth]
        #  username = "admin"
        #  password = "123456aA"
   

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

  [plugins."io.containerd.internal.v1.opt"]
    path = "/opt/containerd"

  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"

  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"

  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false

  [plugins."io.containerd.runtime.v1.linux"]
    no_shim = false
    runtime = "runc"
    runtime_root = ""
    shim = "containerd-shim"
    shim_debug = false

  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["linux/amd64"]

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]

  [plugins."io.containerd.snapshotter.v1.aufs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.btrfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.devmapper"]
    async_remove = false
    base_image_size = ""
    pool_name = ""
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.native"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.overlayfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.zfs"]
    root_path = ""

[proxy_plugins]

[stream_processors]

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_ConFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar"

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_ConFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0

1.2.2 需改的地方：

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        #[plugins."io.containerd.grpc.v1.cri".registry.mirrors."nginx.cclinux.cn:5043"]
        #  endpoint = ["https://nginx.cclinux.cn:5043"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."nginx.cclinux.cn:5000"]
          endpoint = ["http://nginx.cclinux.cn:5000"]
      [plugins."io.containerd.grpc.v1.cri".registry.headers]
      [plugins."io.containerd.grpc.v1.cri".registry.auths]
        #[plugins."io.containerd.grpc.v1.cri".registry.auths."https://nginx.cclinux.cn:5043"]
 	#  #auth = "XXXXXXXXXXX"
        #  username = "admin"
        #  password = "123456aA"
        [plugins."io.containerd.grpc.v1.cri".registry.auths."http://nginx.cclinux.cn:5000"]
 	  #auth = "XXXXXXXXXXX"
          username = "admin"
          password = "123456aA"
           
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        #[plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".tls]
        #  ca_file = "/opt/auth/ca.crt"
      	#  #cert_file = "/opt/auth/harbor.jettech.com.crt"
       	#  #key_file = "/opt/auth/harbor.jettech.com.key"
        #  #insecure_skip_verify = true
        #[plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".auth]
        #  username = "admin"
        #  password = "123456aA"
        #[plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5000".auth]
        #  username = "admin"
        #  password = "123456aA"

然后测试：

[root@jettoloader ~]# crictl pull  nginx.cclinux.cn:5000/library/registry:latest
Image is up to date for sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2

注意：ctr是不可以用这个配置文件的，ctr 不使用 CRI，因此它不读取plugins."io.containerd.grpc.v1.cri"配置issues 6285  issues 5407  这个很重要，困惑我很久。

[root@jettoloader ~]# ctr images pull nginx.cclinux.cn:5000/library/registry:latest
INFO[0000] trying next host                              error="failed to do request: Head "https://nginx.cclinux.cn:5000/v2/library/registry/manifests/latest": http: server gave HTTP response to HTTPS client" host="nginx.cclinux.cn:5000"
ctr: failed to resolve reference "nginx.cclinux.cn:5000/library/registry:latest": failed to do request: Head "https://nginx.cclinux.cn:5000/v2/library/registry/manifests/latest": http: server gave HTTP response to HTTPS client

1.2.3 如果想用ctr下载上面的镜像方法有两个：

1）参数化：

[root@jettoloader ~]#  ctr --address=/run/containerd/containerd.sock images pull --skip-verify  --user admin:123456aA  --plain-http  nginx.cclinux.cn:5000/library/registry:latest
nginx.cclinux.cn:5000/library/registry:latest:                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4: exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3790aef225b922bc97aaba099fe762f7b115aec55a0083824b548a6a1e610719:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0d96da54f60b86a4d869d44b44cfca69d71c4776b81d361bc057d6666ec0d878:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5b27040df4a23c90c3837d926f633fb327fb3af9ac4fa5d5bc3520ad578acb10:    exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:79e9f2f55bf5465a02ee6a6170e66005b20c7aa6b115af6fcd04fad706ea651a:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e2ead8259a04d39492c25c9548078200c5ec429f628dcf7b7535137954cc2df0:    exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.3 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4...
done: 74.140227ms

我应该在其他地方指定这个配置吗？我不想-u user:password 每次必须使用 ctr pull 

--user user:password

--plain-http:allow connections using plain HTTP

2） hosts方式：          hosts.toml   详细参考 2. ctr 工具介绍

2. ctr 工具介绍

      github hosts.toml

     https://github.com/containerd/containerd/blob/main/docs/hosts.md

指定配置目录 将主机命名空间配置与 CTR 结合使用

通过ctr使用--hosts-dir选项来拉取容器映像时，告诉ctr 查找并使用位于指定路径中的主机配置文件：

ctr images pull --hosts-dir "/etc/containerd/certs.d" myregistry.io:5000/image_name:tag
CRI

用于指定 registry.mirrors 和 registry.configs 的旧 CRI 配置模式已弃用。您现在应该将注册表指向文件config_path所在的路径 hosts.toml。

修改您的config.toml（默认位置/etc/containerd/config.toml：）如下：

Modify your config.toml (default location: /etc/containerd/config.toml) as follows:

[plugins."io.containerd.grpc.v1.cri".registry]
   config_path = "/etc/containerd/certs.d"

在你的 config.toml 中为 containerd设置 config_path = "/etc/containerd/certs.d"。在配置路径中创建一个目录树，该目录树包含host-namespace/hosts.toml一个表示要配置的主机命名空间的目录。然后在 中添加一个hosts.toml文件主机来配置主机命名空间。它应该如下所示：

完成路径：/etc/containerd/certs.d//hosts.toml

支持 Docker 的证书文件模式

如果主机目录中不存在 hosts.toml 配置，它将回退到根据Docker 的证书文件模式检查证书文件 （“.crt”文件用于 CA 证书，“.cert”/“.key”文件用于客户端证书）。

 2.1)第一种情况https方式

2.1.1.1) 忽略证书skip_verify = true

[root@jettoloader ~]#  mkdir -p /etc/containerd/certs.d/nginx.cclinux.cn:5043


[root@jettoloader nginx.cclinux.cn:5043]# cat hosts.toml 
#server = "https://docker.io"
#server = "http://nginx.cclinux.cn:5000"
#server = "https://nginx.cclinux.cn:5043"

#[host."http://nginx.cclinux.cn:5000"]
[host."https://nginx.cclinux.cn:5043"]
  capabilities = ["pull", "resolve","push"]
  skip_verify = true

2.1.1.2)不需重启测试注意不用重启systemctl restart containerd。pull和push

[root@jettoloader ~]#  ctr images pull   --user admin:123456aA  --hosts-dir "/etc/containerd/certs.d"  nginx.cclinux.cn:5043/library/registry:latest
nginx.cclinux.cn:5043/library/registry:latest:                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4: exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3790aef225b922bc97aaba099fe762f7b115aec55a0083824b548a6a1e610719:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0d96da54f60b86a4d869d44b44cfca69d71c4776b81d361bc057d6666ec0d878:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5b27040df4a23c90c3837d926f633fb327fb3af9ac4fa5d5bc3520ad578acb10:    exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e2ead8259a04d39492c25c9548078200c5ec429f628dcf7b7535137954cc2df0:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:79e9f2f55bf5465a02ee6a6170e66005b20c7aa6b115af6fcd04fad706ea651a:    exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.4 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4...
done: 21.297093ms

ctr 不使用 CRI 配置。因此，您在 config.toml 的 CRI 部分中指定的身份验证不会被 ctr 客户端读取。

如果需要auth[用户名和密码] 需要加参数：-u user:password 

如果需要证书需要指定： --hosts-dir "/etc/containerd/certs.d"

2.1.2.1) 不忽略证书skip_verify = true，此时需要制定ca证书文件

[root@jettoloader nginx.cclinux.cn:5043]# cat hosts.toml 
#server = "https://docker.io"
#server = "http://nginx.cclinux.cn:5000"

[host."https://nginx.cclinux.cn:5043"]
  capabilities = ["pull", "resolve","push"]
  #skip_verify = true
  #ca = "ca.crt"   #相对路径
  #ca = "/opt/auth/ca.crt"  #绝对路径
  #ca = ["/opt/auth/ca.crt"]
  ca = ["ca.crt"]
  client = [["/opt/auth/nginx.cclinux.cn.crt", "/opt/auth/nginx.cclinux.cn.key"]]
  #client = [["/opt/auth/nginx.cclinux.cn.crt", "/opt/auth/nginx.cclinux.cn.key"],["",""]]

注意：

此处的相对路径是：

[root@jettoloader nginx.cclinux.cn:5043]# pwd
/etc/containerd/certs.d/nginx.cclinux.cn:5043
[root@jettoloader nginx.cclinux.cn:5043]# ls
ca.crt  hosts.toml

此处的绝对路径是：

[root@jettoloader nginx.cclinux.cn:5043]# ls /opt/auth/
ca.crt  nginx.cclinux.cn.crt  nginx.cclinux.cn.key  nginx.conf  nginx.htpasswd

2.1.2.2)不需重启测试注意不用重启systemctl restart containerd。pull和push  

[root@jettoloader ~]#  ctr images pull   --user admin:123456aA  --hosts-dir "/etc/containerd/certs.d"  nginx.cclinux.cn:5043/library/registry:latest
nginx.cclinux.cn:5043/library/registry:latest:                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4: exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3790aef225b922bc97aaba099fe762f7b115aec55a0083824b548a6a1e610719:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0d96da54f60b86a4d869d44b44cfca69d71c4776b81d361bc057d6666ec0d878:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5b27040df4a23c90c3837d926f633fb327fb3af9ac4fa5d5bc3520ad578acb10:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e2ead8259a04d39492c25c9548078200c5ec429f628dcf7b7535137954cc2df0:    exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:79e9f2f55bf5465a02ee6a6170e66005b20c7aa6b115af6fcd04fad706ea651a:    exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.4 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4...
done: 36.400849ms

 2.2)第一种情况http方式

2.2.1)创建镜像仓库域名或ip目录,然后添加hosts.toml文件，

[root@jettoloader ~]# mkdir -p /etc/containerd/certs.d/nginx.cclinux.cn:5000
[root@jettoloader nginx.cclinux.cn:5000]# pwd
/etc/containerd/certs.d/nginx.cclinux.cn:5000
[root@jettoloader nginx.cclinux.cn:5000]# cat hosts.toml 
#server = "https://docker.io"
server = "http://nginx.cclinux.cn:5000"

[host."http://nginx.cclinux.cn:5000"]
  capabilities = ["pull", "resolve","push"]
  skip_verify = true

2.2.2) 测试pull、push，【无认证就是无用户名和密码】 注意不用重启systemctl restart containerd

[root@jettoloader ~]#  ctr images pull --hosts-dir "/etc/containerd/certs.d"  nginx.cclinux.cn:5000/library/registry:latest
nginx.cclinux.cn:5000/library/registry:latest:                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4: exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3790aef225b922bc97aaba099fe762f7b115aec55a0083824b548a6a1e610719:    exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:79e9f2f55bf5465a02ee6a6170e66005b20c7aa6b115af6fcd04fad706ea651a:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5b27040df4a23c90c3837d926f633fb327fb3af9ac4fa5d5bc3520ad578acb10:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0d96da54f60b86a4d869d44b44cfca69d71c4776b81d361bc057d6666ec0d878:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e2ead8259a04d39492c25c9548078200c5ec429f628dcf7b7535137954cc2df0:    exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.4 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4...
done: 20.402958ms

 2.2.3) 测试pull、push，【有认证就是用户名和密码】 注意不用重启systemctl restart containerd

[root@jettoloader ~]#  ctr images pull   --user admin:123456aA  --hosts-dir "/etc/containerd/certs.d"  nginx.cclinux.cn:5000/library/registry:latest
nginx.cclinux.cn:5000/library/registry:latest:                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4: exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3790aef225b922bc97aaba099fe762f7b115aec55a0083824b548a6a1e610719:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0d96da54f60b86a4d869d44b44cfca69d71c4776b81d361bc057d6666ec0d878:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5b27040df4a23c90c3837d926f633fb327fb3af9ac4fa5d5bc3520ad578acb10:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e2ead8259a04d39492c25c9548078200c5ec429f628dcf7b7535137954cc2df0:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:79e9f2f55bf5465a02ee6a6170e66005b20c7aa6b115af6fcd04fad706ea651a:    exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.5 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4...
done: 52.012024ms

                                                                        Q/A 环节

Q:我应该在其他地方指定这个配置吗？我不想-u user:password每次必须使用 ctr pull

A:目前，我们没有其他方法可以为该ctr工具指定身份验证

Crictl can pull images but ctr gives unauthorized, private registry with basic auth · Issue #5586 · containerd/containerd · GitHub​

作为一个团队，尚未就主机身份验证配置改进的路径做出决定..注意：

https://github.com/containerd/containerd/blob/main/docs/hosts.md

containerd/config.md at main · containerd/containerd · GitHub

该 文档涵盖了最近为支持除主机身份验证之外的所有主机配置所做的更改。

nerdctl (another containerd CLI) supports .docker/config.json for authentication

GitHub - containerd/nerdctl: contaiNERD CTL - Docker-compatible CLI for containerd, with support for Compose, Rootless, eStargz, OCIcrypt, IPFS, ...

​看 第三章节nerdctl工具

3. nerdctl 工具介绍

3.1 nerdctl，介绍

            2021年8月31日，Docker公司更改了收费策略，简而言之就是对个人和小公司不收费，对大公司收费,是时候开始试一下 containerd + lima[ 这对mac系统的]这个开源、免费的Docker替代品

           这可能已经是Docker公司第二次提醒我们 Docker 不等于容器了

          就像Kubernetes需要一个中立的容器标准，所以OCI出现了；容器客户端也需要一个中立的标准和参考实现，目前一下是一个比较合适的方案，值得关注这两个项目的长期发展。

           mac:containerd+nerdctl+lima

           linux:containerd+nerdctl

            Kubernetes 虽然制定了容器运行时接口（CRI）标准，但早期能用的容器运行时只有 Docker，而 Docker 又不适配这个标准，于是给 Docker 开了后门，花了大量的精力去适配它。后来有了更多的容器运行时可以选择后，Kubernetes 就不得不重新考量要不要继续适配 Docker 了，因为每次更新 Kubelet 都要考虑与 Docker 的适配问题。
          标准这个东西就是这样，我定好标准，你兼容了就一起玩，不兼容就拜拜，它就像两个人在一起的底线，你可以重，你可以丑，你也可以不完善，但是你不兼容标准就真的不能一起玩了，于是 Kubernetes 就把 Docker 踢出了群聊。

          最终 Kubernetes 选择了 Containerd，时至今日 Containerd 已经变成一个工业级的容器运行时了，它足够简单、健壮，可移植性也很强。

          现有 CLI 的不足

          但是在containerd的目标聚焦在和其他系统集成，所以它的默认命令行工具（crictl）也不是很好用，和docker也不兼容。

          虽然 Docker 能干的事情，现在 Containerd 都能干，但 Containerd 还有一个非常明显的缺陷：CLI 不够友好。它无法像 Docker 和 Podman 一样通过一条简单的命令启动一个容器，它的两个 CLI 工具 ctr 和 crictl 都无法实现这么一件非常简单的需求，而这个需求是大多数人都需要的，我总不能为了在本地测试容器而专门部署一个 Kubernetes 集群吧。可以认为ctr 和 crictl两个工具是针对k8s/k3s集群而生的。

nerdctl不仅与docker兼容，而且还支持了更多的功能：

支持containerd的命名空间查看，nerdctl不仅可以管理Docker容器，也可以直接管理本地的的Kubernetes pod支持将Docker Image Manifest镜像转换为OCI镜像、estargz镜像支持OCIcrypt（镜像加密） 什么是lima？

containerd要在macOS上使用，需要安装虚拟机、然后在虚拟机配置containerd，这个过程很浪费时间。

所以，lima项目简化了这个过程。

lima会启动一个虚拟机，然后在虚拟机中安装containerd，并自动配置虚拟机的文件共享，网络等。

lima相比Docker for Desktop Mac，也有一些优势：

内置lazy pulling支持（基于stargz）额外支持ARM on Intel，Intel on ARM（基于QEMU）可以自定义虚拟机发行版（默认是Ubuntu）


  3.1.1)nerdctl  (containerd CLI)，支持 .docker/config.json 进行身份验证   nerdctl

nerdctl 是containerd 的cli客户端，与docker cli大部分兼容。
nerdctl是containerd的非核心子项目

3.1.2)ctr 的设计对人类不太友好，例如缺少以下这些和 Docker 类似的功能：

docker run -p docker run --restart=always通过凭证文件 ~/.docker/config.json 来拉取镜像docker logs

3.1.3)除此之外还有一个 CLI 工具叫 crictl，和 ctr 一样不太友好。

3.1.4)为了解决这个痛点，Containerd 官方推出了一个新的 CLI 叫 nerdctl。nerdctl 的使用体验和 docker 一样顺滑，例如：

erdctl run -d -p 8080:80 --name=nginx --restart=always nginx

3.1.5)nerdctl 只是 docker 的复制品？

       nerdctl 的目标并不是单纯地复制 docker 的功能，它还实现了很多 docker 不具备的功能，例如延迟拉取镜像（lazy-pulling）、镜像加密（imgcrypt）等。具体看nerdctl

延迟拉取镜像功能可以参考这篇文章：Containerd 使用 Stargz Snapshotter 延迟拉取镜像

虽然这些功能预计最终也会在 Docker 中实现，可能需要几个月甚至几年的时间，因为 Docker 目前的设计只使用一小部分 Containerd 子系统。将来 Docker 有可能重构代码以使用完整的 Containerd，但目前还没看到什么实质性进展。所以 Containerd 社区决定创建一个新的 CLI 来更友好地使用 Containerd。

3.2 安装

下载地址：Releases · containerd/nerdctl,中下载最新的可执行文件，每一个版本都有两种可用的发行版

精简 (nerdctl--linux-amd64.tar.gz): 只包含nerdctl完整 (nerdctl-full--linux-amd64.tar.gz): 包含 containerd, runc, and CNI等依赖

如果你已经安装了 Containerd，只需要选择前一个发行版，否则就选择完整版。

精简Minimal

将归档文件解压到 /usr/local/bin 或 ~/bin

[root@jettoloader nerdctl]# tar Cxzvvf /usr/local/bin nerdctl-0.16.0-linux-amd64.tar.gz
[root@jettoloader nerdctl]# ls -l /usr/local/bin/
-rwxr-xr-x 2 root root        52 1月  31 12:26 nerdctl
-rwxr-xr-x 2 root root        52 1月  31 12:26 containerd-rootless-setuptool.sh
-rwxr-xr-x 2 root root        52 1月  31 12:26 containerd-rootless.sh
完整Full 
解压文件到/usr/local/

[root@jettoloader nerdctl]# tar -xf nerdctl-full-0.16.0-linux-amd64.tar.gz -C /usr/local/ 
启动服务 

[root@jettoloader nerdctl]# ls /usr/local/lib//systemd/system/
buildkit.service  containerd.service  stargz-snapshotter.service

[root@jettoloader nerdctl]# cp /usr/local/lib/systemd/system/*.service /etc/systemd/system/


[root@jettoloader nerdctl]# systemctl enable  buildkit containerd --now
[root@jettoloader nerdctl]# systemctl status buildkit containerd
● buildkit.service
   Loaded: loaded (/etc/systemd/system/buildkit.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2022-01-31 12:34:12 CST; 12min ago
 Main PID: 22978 (buildkitd)
   CGroup: /system.slice/buildkit.service
           └─22978 /usr/local/bin/buildkitd

1月 31 12:34:12 jettoloader systemd[1]: Starting buildkit.service...
1月 31 12:34:12 jettoloader buildkitd[22978]: time="2022-01-31T12:34:12+08:00" level=info msg="auto snapshotter: using overlayfs"
1月 31 12:34:12 jettoloader buildkitd[22978]: time="2022-01-31T12:34:12+08:00" level=warning msg="using host network as the default"
1月 31 12:34:12 jettoloader buildkitd[22978]: time="2022-01-31T12:34:12+08:00" level=info msg="found worker "89o6fwn2m85nergdyc3vxh9ri", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproje...md64 linux/386]"
1月 31 12:34:12 jettoloader buildkitd[22978]: time="2022-01-31T12:34:12+08:00" level=warning msg="using host network as the default"
1月 31 12:34:12 jettoloader buildkitd[22978]: time="2022-01-31T12:34:12+08:00" level=info msg="found worker "xsw05ykrexuxsb51r9ky8ozd1", labels=map[org.mobyproject.buildkit.worker.containerd.namespace:buildkit org.mobyproj...
1月 31 12:34:12 jettoloader buildkitd[22978]: time="2022-01-31T12:34:12+08:00" level=info msg="found 2 workers, default="89o6fwn2m85nergdyc3vxh9ri""
1月 31 12:34:12 jettoloader buildkitd[22978]: time="2022-01-31T12:34:12+08:00" level=warning msg="currently, only the default worker can be used."
1月 31 12:34:12 jettoloader systemd[1]: Started buildkit.service.
1月 31 12:34:12 jettoloader buildkitd[22978]: time="2022-01-31T12:34:12+08:00" level=info msg="running server on /run/buildkit/buildkitd.sock"

● containerd.service - containerd container runtime
   Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2022-01-31 12:36:52 CST; 10min ago
     Docs: https://containerd.io
 Main PID: 29686 (containerd)
   CGroup: /system.slice/containerd.service
           └─29686 /usr/local/bin/containerd

1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.055182710+08:00" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.055262035+08:00" level=info msg="Start subscribing containerd event"
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.055414180+08:00" level=info msg="Start recovering state"
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.055452341+08:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.055493510+08:00" level=info msg=serving... address=/run/containerd/containerd.sock
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.055557796+08:00" level=info msg="containerd successfully booted in 0.044853s"
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.080950138+08:00" level=info msg="Start event monitor"
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.081005740+08:00" level=info msg="Start snapshots syncer"
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.081021647+08:00" level=info msg="Start cni network conf syncer"
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.081030575+08:00" level=info msg="Start streaming server"
Hint: Some lines were ellipsized, use -l to show in full.
[root@jettoloader nerdctl]#

对于非root用户以上命令需要加上sudo，而且还要额外运行

1

containerd-rootless-setuptool.sh installz

注意，这个rootless-setuptool目前不支持centOS 7及其以下。

3.3 设置nerdctl子命令可以使用tab键 
编辑文件
[root@jettoloader nerdctl]# vim /etc/profile
source <(nerdctl completion bash)

#时期生效
[root@jettoloader nerdctl]#  source /etc/profile
3.4 修改containerd配置文件并配置加速 
[root@jettoloader nerdctl]# cat /etc/containerd/config.toml 
disabled_plugins = ["restart"]
[plugins]
   [plugins.cri.registry.mirrors."docker.io"]
     endpoint = ["https://frz7i079.mirror.aliyuncs.com"]
   #[plugins.cri.registry.mirrors."nginx.cclinux.cn:5000"]
   #  endpoint = ["http://nginx.cclinux.cn:5000"]



重启containerd

 [root@jettoloader nerdctl]# systemctl restart containerd
3.5  基本使用使用默认CNI网络（10.4.0.0/24）启动一个容器 测试运行 
[root@jettoloader nerdctl]# nerdctl run -it --rm alpine
/ # ls
bin    dev    etc    home   lib    media  mnt    opt    proc   root   run    sbin   srv    sys    tmp    usr    var


/ # ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if108:  mtu 1500 qdisc noqueue state UP 
    link/ether c2:5e:b8:43:b4:f4 brd ff:ff:ff:ff:ff:ff
    inet 10.4.0.3/24 brd 10.4.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::c05e:b8ff:fe43:b4f4/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
3.6 构建镜像,nerdctl build：从 Dockerfile 构建镜像

镜像构建是平时我们非常重要的一个需求，我们知道 ctr 并没有构建镜像的命令，而现在我们又不使用 Docker 了，那么如何进行镜像构建了，幸运的是 nerdctl 就提供了 nerdctl build 这样的镜像构建命令。

3.6.1) 比如现在我们定制一个 nginx 镜像，新建一个如下所示的 Dockerfile 文件：

[root@jettoloader ~]# mkdir demo 
[root@jettoloader ~]# cat demo/Dockerfile 
FROM nginx:alpine
RUN echo 'Hello Nerdctl From Containerd' > /usr/share/nginx/html/index.html

然后在文件所在目录执行镜像构建命令：

[root@jettoloader ~]# nerdctl build -t nginx:nerctl -f demo/Dockerfile demo
ERRO[0000] error: failed to list workers: Unavailable: connection error: desc = "transport: error while dialing: dial unix /run/buildkit/buildkitd.sock: connect: no such file or directory" 
FATA[0000] `buildctl` needs to be installed and `buildkitd` needs to be running, see https://github.com/moby/buildkit: exit status 1

FATA[0000] `buildctl` needs to be installed and `buildkitd` needs to be running, see https://github.com/moby/buildkit: exec: "buildctl": executable file not found in $PATH

 注意：也可以加上这个–no-cache选项

可以看到有一个错误提示，需要我们安装 buildctl 并运行 buildkitd，这是因为 nerdctl build 需要依赖 buildkit 工具。

buildkit 项目也是 Docker 公司开源的一个构建工具包，支持 OCI 标准的镜像构建。它主要包含以下部分:

服务端 buildkitd：当前支持 runc 和 containerd 作为 worker，默认是 runc，我们这里使用 containerd
客户端 buildctl：负责解析 Dockerfile，并向服务端 buildkitd 发出构建请求
buildkit 是典型的 C/S 架构，客户端和服务端是可以不在一台服务器上，而 nerdctl 在构建镜像的.

https://github.com/moby/buildkit

nerdctl 也可以和 buildkit 结合使用来构建容器镜像，需要先下载 buildkit 的可执行文件：

下载
[root@jettoloader nerdctl]#  wget https://github.com/moby/buildkit/releases/download/v0.9.3/buildkit-v0.9.3.linux-amd64.tar.gz

[root@containerd ~]#tar tf buildkit-v0.9.1.linux-amd64.tar.gz
bin/
bin/buildctl              #客户端
bin/buildkit-qemu-aarch64
bin/buildkit-qemu-arm
bin/buildkit-qemu-i386
bin/buildkit-qemu-mips64
bin/buildkit-qemu-mips64el
bin/buildkit-qemu-ppc64le
bin/buildkit-qemu-riscv64
bin/buildkit-qemu-s390x
bin/buildkit-runc
bin/buildkitd              #服务端


解压到PATH
[root@jettoloader nerdctl]#  tar -C /usr/local/ -zxvf buildkit-v0.9.3.linux-amd64.tar.gz


服务端:服务文件
[root@jettoloader nerdctl]#  /etc/systemd/system/buildkit.service
[Unit]
Description=BuildKit
documentation=https://github.com/moby/buildkit

[Service]
ExecStart=/usr/local/bin/buildkitd --oci-worker=false --containerd-worker=true

[Install]
WantedBy=multi-user.target


#启动服务
[root@jettoloader nerdctl]#  systemctl enable --now buildkit.service

#看下日志
[root@jettoloader nerdctl]#  journalctl -u buildkit

#案例下载：
[root@jettoloader nerdctl]#  git clone --depth=1 https://github.com.cnpmjs.org/kubesphere/kubesphere.git

#进入仓库目录，编译二进制文件：
[root@jettoloader nerdctl]# cd kubesphere
[root@jettoloader nerdctl]# make ks-apiserver


#将二进制文件拷贝到 Dockerfile 目录：
[root@jettoloader nerdctl]#  cp bin/cmd/ks-apiserver build/ks-apiserver


#进入 Dockerfile 目录，修改 Dockerfile：

# Copyright 2020 The KubeSphere Authors. All rights reserved.
# Use of this source code is governed by an Apache license
# that can be found in the LICENSE file.
FROM alpine:3.11

ARG HELM_VERSION=v3.5.2

RUN apk add --no-cache ca-certificates
# install helm
RUN wget https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz && 
    tar xvf helm-${HELM_VERSION}-linux-amd64.tar.gz && 
    rm helm-${HELM_VERSION}-linux-amd64.tar.gz && 
    mv linux-amd64/helm /usr/bin/ && 
    rm -rf linux-amd64
# To speed up building process, we copy binary directly from make
# result instead of building it again, so make sure you run the
# following command first before building docker image
#   make ks-apiserver
#
COPY  ks-apiserver /usr/local/bin/

EXPOSE 9090
CMD ["sh"]







#构建镜像：
[root@jettoloader nerdctl]# cd build/ks-apiserver
[root@jettoloader nerdctl]# nerdctl build -t ks-apiserver .

现在我们再来重新构建镜像：

[root@jettoloader ~]# nerdctl build -t harbor.jettech.com/nginx:nerctl -f demo/Dockerfile demo
[+] Building 4.4s (6/6) FINISHED                                                                                                                                                                                                                                              
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                     0.1s
 => => transferring dockerfile: 131B                                                                                                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                        0.1s
 => => transferring context: 2B                                                                                                                                                                                                                                          0.0s
 => [internal] load metadata for docker.io/library/nginx:alpine                                                                                                                                                                                                          3.2s
 => [1/2] FROM docker.io/library/nginx:alpine@sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3                                                                                                                                                    0.1s
 => => resolve docker.io/library/nginx:alpine@sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3                                                                                                                                                    0.1s
 => CACHED [2/2] RUN echo 'Hello Nerdctl From Containerd' > /usr/share/nginx/html/index.html                                                                                                                                                                             0.0s
 => exporting to oci image format                                                                                                                                                                                                                                        0.8s
 => => exporting layers                                                                                                                                                                                                                                                  0.0s
 => => exporting manifest sha256:0f36e8f9982c9f3157256f8db4272bef1895476be588216a9a5dd512abdbbd4d                                                                                                                                                                        0.0s
 => => exporting config sha256:94dacf915be33a342b7b40607bc476b07a39e608605b6182c835b54a2747ad58                                                                                                                                                                          0.0s
 => => sending tarball                                                                                                                                                                                                                                                   0.6s
unpacking harbor.jettech.com/nginx:nerctl (sha256:0f36e8f9982c9f3157256f8db4272bef1895476be588216a9a5dd512abdbbd4d)...done
[root@jettoloader ~]# nerdctl images
REPOSITORY                                TAG       IMAGE ID        CREATED           PLATFORM       SIZE
nginx                                     latest    2834dc507516    2 days ago        linux/amd64    146.2 MiB
harbor.jettech.com/nginx                  nerctl    0f36e8f9982c    6 seconds ago

这样我们就使用 nerdctl + buildkitd 轻松完成了容器镜像的构建

C/S架构

nerdctl或buildctl  【客户端】  +buildkitd【服务端】

docker cli【客户端】   + dockerd【服务端】

关于 nerdctl 的更多用法，可以参考官方仓库的 README

3.7 containerd网络，它对接的是cni

总结
从行业趋势来看，Docker 已经和 Kubernetes 社区渐行渐远，以 Containerd 为代表的实现了 CRI 接口的容器运行时将会受到 Kubernetes 的青睐。但纯粹使用 Containerd 还是有诸多困扰，比如不方便通过 CLI 来创建管理容器，有了 nerdctl 这个 CLI 工具，就就可以填补 Containerd 易用性的空缺，让你在单机上也能愉快地使用 Containerd。Containerd高级命令行工具

欢迎分享，转载请注明来源：内存溢出

原文地址:https://www.54852.com/zaji/5717162.html

赞 (0)
打赏 微信扫一扫微信扫一扫 支付宝扫一扫支付宝扫一扫
防结块剂防结块剂一级用户组
00
上一篇 2022-12-17
下一篇2022-12-17

发表评论

登录后才能评论

评论列表（0条）

    保存