2021-06-01

prowでPRのsquashしてもlgtmが外れないロジックの実装箇所

ここに書かれてる内容に関して。
マージされたPRはkubernetes/community#5779。

GitHubのPRの各種イベントが発生したときにLGTMプラグインがラベルを外すかどうかを判断する部分は、prow のこの辺りに実際のロジックがある様子。
opts.StoreTreeHash の説明はここ。
一方、store_tree_hash がTrueになってるのは website くらいなので、通常であれば、ユーザが手動でSquashして、その結果コミットハッシュが変わると、その瞬間にLGTMラベルは外れる。
つまり、LGTMラベルの方で、tide/merge-method-squash などのラベルを判断してLGTMをそのまま保持するというロジックは特に存在しない。

では、冒頭の記述はなにか？
tide/merge-method-squash などのタグを打っておくと、Squashは手動ではなくて、Tideの方でマージのタイミングでやってくれるので、ProwのLGTMプラグインがラベルを外すという処理が走ることもないため、結果としてSquash操作によるLGTMラベルの削除は実行されない。
このことを、and will not result in removal of theLGTMlabel (if already applied) と言ってるようだ。

つまり、tide/merge-method-squash などのラベルを設定したPRに関して、ユーザによる手動のSquashは実施しなくてよい。Tideがやってくれるので。ただ、これ、追加のコミットをPRに追加したりした時はたとえ tide/merge-method-squash がついててもLGTMが外れるのではないだろうか。

というか表現的には、TideがSquashしてくれるのでユーザによる手動Squashをする必要がないよ。でも手動SquashするとLGTM外れるよ。の方が正しいのではないか。

2021-02-16

ディスク使用率を見るためにdustをインストールする

Prerequisites

asdfを使う

Install

asdf plugin add rust
asdf list all rust
asdf install rust stable
asdf global rust stable

cargo install du-dust
asdf reshim rust

Usage

dust

Ref

2021-01-28

go get k8s.io/kubectl のリポジトリURLはどこから取得しているか

TL;DR

go get k8s.io/kubectl の場合 k8s.io/kubectl?go-get=1 なURLを参照して、中に書いてあるmetaタグの情報からgithubへの参照URLを得ている。
動作の詳細は go help importpath に書いてある。

経緯

go get k8s.io/kubectl とかやったときにちゃんと github.com/kubernetes/kubectl のリポジトリからコードを取得してくれると思います。
一方で、Webブラウザで k8s.io/kubectl にアクセスすると kubernetes.io/kubectl を参照すると思います。
この挙動の違いはどこから来るのだろうと思ったわけですね。

わかったこと

go help importpath に動作の詳細が書いてある。
Githubなど有名なところはgoコマンドがよろしく判断してくれる。

一方、自分たちでホストするようなモジュールの場合は、返却に含まれる<meta name="go-import" ......> の内容から参照先のリポジトリ情報を得ている。
例えば logのモジュール go.uber.org/zap とかは以下のような感じ。
タグからたどっていける。
なるほど。

$ curl https://go.uber.org/zap
<!DOCTYPE html>
<html>
    <head>
        <meta name="go-import" content="go.uber.org/zap git https://github.com/uber-go/zap">
        <meta name="go-source" content="go.uber.org/zap https://github.com/uber-go/zap https://github.com/uber-go/zap/tree/master{/dir} https://github.com/uber-go/zap/tree/master{/dir}/{file}#L{line}">
        <meta http-equiv="refresh" content="0; url=https://pkg.go.dev/go.uber.org/zap">
    </head>
    <body>
        Nothing to see here. Please <a href="https://pkg.go.dev/go.uber.org/zap">move along</a>.
    </body>
</html>

じゃあ k8s.io/kubectl も同じだろうと思ったらそれだけだと出てこない。

$ curl https://k8s.io/kubectl
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.10.3</center>
</body>
</html>

ここで go help importpath にかかれている内容を詳しく見ると go-get=1 というクエリを付加して呼び出す形式が見つかる。

$ curl https://k8s.io/kubectl?go-get=1
<html><head>
      <meta name="go-import"
            content="k8s.io/kubectl
                     git https://github.com/kubernetes/kubectl">
      <meta name="go-source"
            content="k8s.io/kubectl
                     https://github.com/kubernetes/kubectl
                     https://github.com/kubernetes/kubectl/tree/master{/dir}
                     https://github.com/kubernetes/kubectl/blob/master{/dir}/{file}#L{line}">
</head></html>

出てきた。
こうやってリポジトリへのURLを取得しているのか。
ちなみに go-get=1 が付いていない場合は最終的に kubernetes.io/kubectl へのリダイレクトになっている。(ページ自体存在しないという404になる。)

2014-01-25

Convert iptables info to graph with graph-easy and graphviz

OpenStack Nova Neutron iptables graphviz

Result

create iptables chain graph. result is below.

f:id:atoato88:20140125140915p:plain

Prerequisite

Install require commands

#sudo cpan Graph::Easy # for graph-easy command
sudo apt-get install libgraph-easy-perl # or use this command
sudo apt-get install graphviz # for dot command

Sample iptables chains

iptables-grizzly.txt

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
quantum-openvswi-INPUT  all  --  anywhere             anywhere
ACCEPT     gre  --  anywhere             anywhere
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:bootps

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
neutron-filter-top  all  --  anywhere             anywhere
quantum-openvswi-FORWARD  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             192.168.122.0/24     state RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     anywhere
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
neutron-filter-top  all  --  anywhere             anywhere
quantum-openvswi-OUTPUT  all  --  anywhere             anywhere

Chain neutron-filter-top (2 references)
target     prot opt source               destination
quantum-openvswi-local  all  --  anywhere             anywhere

Chain quantum-openvswi-FORWARD (1 references)
target     prot opt source               destination
quantum-openvswi-sg-chain  all  --  anywhere             anywhere             PHYSDEV match --physdev-out tap40fae852-e1 --physdev-is-bridged
quantum-openvswi-sg-chain  all  --  anywhere             anywhere             PHYSDEV match --physdev-in tap40fae852-e1 --physdev-is-bridged

Chain quantum-openvswi-INPUT (1 references)
target     prot opt source               destination
quantum-openvswi-o40fae852-e  all  --  anywhere             anywhere             PHYSDEV match --physdev-in tap40fae852-e1 --physdev-is-bridged

Chain quantum-openvswi-OUTPUT (1 references)
target     prot opt source               destination

Chain quantum-openvswi-i40fae852-e (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             state INVALID
RETURN     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
RETURN     tcp  --  anywhere             anywhere             tcp dpt:ssh
RETURN     icmp --  anywhere             anywhere
RETURN     udp  --  L100.BSTNMA-VFTTP-109.verizon-gni.net  anywhere             udp spt:bootps dpt:bootpc
quantum-openvswi-sg-fallback  all  --  anywhere             anywhere

Chain quantum-openvswi-local (1 references)
target     prot opt source               destination

Chain quantum-openvswi-o40fae852-e (2 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             MAC ! FA:16:3E:20:ED:FD
RETURN     udp  --  anywhere             anywhere             udp spt:bootpc dpt:bootps
DROP       all  -- !pool-100-0-0-2.bstnma.fios.verizon.net  anywhere
DROP       udp  --  anywhere             anywhere             udp spt:bootps dpt:bootpc
DROP       all  --  anywhere             anywhere             state INVALID
RETURN     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
RETURN     all  --  anywhere             anywhere
quantum-openvswi-sg-fallback  all  --  anywhere             anywhere

Chain quantum-openvswi-sg-chain (2 references)
target     prot opt source               destination
quantum-openvswi-i40fae852-e  all  --  anywhere             anywhere             PHYSDEV match --physdev-out tap40fae852-e1 --physdev-is-bridged
quantum-openvswi-o40fae852-e  all  --  anywhere             anywhere             PHYSDEV match --physdev-in tap40fae852-e1 --physdev-is-bridged
ACCEPT     all  --  anywhere             anywhere

Chain quantum-openvswi-sg-fallback (2 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywher

Convert to ascii graph

ubuntu@wan1-01:~$ cat iptables-grizzly.txt | sed -e "s/Chain \(.*\) (.*)/[Chain\1]/g" | sed "/\(target\|DNAT\|ACCEPT\|MASQUERADE\|SNAT\|DROP\|RETURN\|REJECT\).*/d" | sed -e "s/^\([^ ]*\) .*/[\1]/g" | sed "/^$/d" | awk -v ORS=, '{print $1}' | sed -e "s/\],\[Chain/] [Chain/g" | sed -e "s/\[Chain\([^ ,]*\),/[\1---->/g" | sed -e "s/\(.*\),/\1/g" | sed -e "s/Chain//g" | sed -e "s/\(.*\)---->/\1/g" | graph-easy

f:id:atoato88:20140125140925p:plain


                                  +--------------------------------+
                                  |                                v
+-------------------------+     +--------------------------+     +------------------------------+     +------------------------------+     +------------------------------+
|         FORWARD         | --> | quantum-openvswi-FORWARD | --> |  quantum-openvswi-sg-chain   | --> | quantum-openvswi-i40fae852-e | --> | quantum-openvswi-sg-fallback |
+-------------------------+     +--------------------------+     +------------------------------+     +------------------------------+     +------------------------------+
  |                                                                                                                                          ^
  +-------------------------------+                                                                                                          |
                                  v                                                                                                          |
+-------------------------+     +--------------------------+     +------------------------------+                                            |
|         OUTPUT          | --> |    neutron-filter-top    | --> |    quantum-openvswi-local    |                                            |
+-------------------------+     +--------------------------+     +------------------------------+                                            |
  |                                                                                                                                          |
  |                                                                                                                                          |
  v                                                                                                                                          |
+-------------------------+                                                                                                                  |
| quantum-openvswi-OUTPUT |                                                                                                                  |
+-------------------------+                                                                                                                  |
+-------------------------+     +--------------------------+     +------------------------------+                                            |
|          INPUT          | --> |  quantum-openvswi-INPUT  | --> | quantum-openvswi-o40fae852-e | -------------------------------------------+
+-------------------------+     +--------------------------+     +------------------------------+
ubuntu@wan1-01:~$

Convert to png file with graphviz(dot command)

cat iptables-grizzly.txt | sed -e "s/Chain \(.*\) (.*)/[Chain\1]/g" | sed "/\(target\|DNAT\|ACCEPT\|MASQUERADE\|SNAT\|DROP\|RETURN\|REJECT\).*/d" | sed -e "s/^\([^ ]*\) .*/[\1]/g" | sed "/^$/d" | awk -v ORS=, '{print $1}' | sed -e "s/\],\[Chain/] [Chain/g" | sed -e "s/\[Chain\([^ ,]*\),/[\1---->/g" | sed -e "s/\(.*\),/\1/g" | sed -e "s/Chain//g" | sed -e "s/\(.*\)---->/\1/g" | graph-easy -as dot | dot -Tpng -o iptables-grizzly.png

f:id:atoato88:20140125140915p:plain

Other iptables chain

VM count:3
Neutron Plugin:Open vSwitch

f:id:atoato88:20140125140921p:plain

Other iptables chain 2

OpenStack Essex

f:id:atoato88:20140125140856p:plain

2013-05-02

Quantum multinode settings on DevStack

OpenStack DevStack Quantum Grizzly

localrc

compute-node host settings.

NOVA_BRANCH=stable/grizzly
CINDER_BRANCH=stable/grizzly
KEYSTONE_BRANCH=stable/grizzly
GLANCE_BRANCH=stable/grizzly
QUANTUM_BRANCH=stable/grizzly
SWIFT_BRANCH=stable/grizzly
CEILOMETER_BRANCH=stable/grizzly

ENABLED_SERVICES=n-cpu,rabbit,g-api,quantum,q-agt
DATABASE_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_TOKEN=password
SERVICE_PASSWORD=password
ADMIN_PASSWORD=password

SERVICE_HOST=192.168.1.201
DATABASE_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
Q_HOST=$SERVICE_HOST

ENABLE_TENANT_TUNNELS=True
PIP_USE_MIRRORS=False

Additional settings

/etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini

Add mysql setting. Somehow, DevStack create no mysql settings.

sql_connection = mysql://root:password@192.168.1.201/ovs_quantum?charset=utf8

/etc/quantum/quantum.conf

change keystone auth_host from 127.0.0.1 to <controller host IP address>.

[keystone_authtoken]
auth_host = 192.168.1.201
auth_port = 35357
auth_protocol = http
admin_tenant_name = %SERVICE_TENANT_NAME%
admin_user = %SERVICE_USER%
admin_password = %SERVICE_PASSWORD%
signing_dir = /var/lib/quantum/keystone-signing

Add debug and verbose value.

debug = True
verbose = True

2013-04-29

Meter volume usage in ceilometer

Cinder OpenStack DevStack Ceilometer

Prerequisite

version : stable/grizzly

Problem

Ceilometer can meter volume usage, but it can't cooperate with cinder in default settings.

Cause

Ceilometer use cinder exchange for cinder notifications. Similarly, nova exchange for nova, quantum exchange for quantum. Nova and quantum usage are metered with ceilometer as expected. But, cinder doesn't work successfully, because it uses openstack exhange on its proccess, not cinder exhange.

Solution

You need to set parameters for exchange that cinder use in ceilometer.conf below.

cinder_control_exchange=openstack

Following table shows exchanges that ceilometer use for each component of openstack.

Component	key name	default value of exchange	changed value
Nova	nova_control_exchange	nova	nova(default)
Glance	glance_control_exchange	glance_notifications	glance_notifications(default)
Quantum	quantum_control_exchange	quantum	quantum(default)
Cinder	cinder_control_exchange	cinder	openstack

You can change it in ceilometer.conf.

Here is my setting.

[DEFAULT]
mysql://root:password@localhost/nova?charset=utf8 =
os_auth_url = http://192.168.1.201:35357/v2.0
os_tenant_name = service
os_password = password
os_username = ceilometer
policy_file = /etc/ceilometer/policy.json
verbose = True
notification_topics = notifications,glance_notifications
rabbit_password = password
rabbit_host = localhost
rpc_backend = ceilometer.openstack.common.rpc.impl_kombu

cinder_control_exchange=openstack
#change exchange name for cinder from default value *cinder*

[keystone_authtoken]
signing_dir = /var/cache/ceilometer
admin_tenant_name = service
admin_password = password
admin_user = ceilometer
auth_protocol = http

2013-04-29

Version of python-keystoneclient in horizon at grizzly

OpenStack DevStack Horizon

It is current version of python-keystoneclient below

commit 5c37d85944d9eed73ec6dd6254842108386bcc4f

Currently, horizon in stable/grizzly branch using it can't work successfully. When you try changing tenant on horizon, following error occurs.

[Sun Apr 28 17:05:39 2013] [error] DEBUG:openstack_auth.views:Switching to tenant 156c5573a4064caa9d6cd28dda26cc2b for user "admin".
[Sun Apr 28 17:05:39 2013] [error] ERROR:django.request:Internal Server Error: /auth/switch/156c5573a4064caa9d6cd28dda26cc2b/
[Sun Apr 28 17:05:39 2013] [error] Traceback (most recent call last):
[Sun Apr 28 17:05:39 2013] [error]   File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
[Sun Apr 28 17:05:39 2013] [error]     response = callback(request, *callback_args, **callback_kwargs)
[Sun Apr 28 17:05:39 2013] [error]   File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/decorators.py", line 20, in _wrapped_view
[Sun Apr 28 17:05:39 2013] [error]     return view_func(request, *args, **kwargs)
[Sun Apr 28 17:05:39 2013] [error]   File "/usr/local/lib/python2.7/dist-packages/openstack_auth/views.py", line 99, in switch
[Sun Apr 28 17:05:39 2013] [error]     token=request.user.token.id)
[Sun Apr 28 17:05:39 2013] [error]   File "/opt/stack/python-keystoneclient/keystoneclient/v2_0/tokens.py", line 38, in authenticate
[Sun Apr 28 17:05:39 2013] [error]     if self.api.management_url is None:
[Sun Apr 28 17:05:39 2013] [error]   File "/opt/stack/python-keystoneclient/keystoneclient/base.py", line 89, in _create
[Sun Apr 28 17:05:39 2013] [error]     if return_raw:
[Sun Apr 28 17:05:39 2013] [error]   File "/opt/stack/python-keystoneclient/keystoneclient/client.py", line 420, in post
[Sun Apr 28 17:05:39 2013] [error]     return self._cs_request(url, 'PATCH', **kwargs)
[Sun Apr 28 17:05:39 2013] [error]   File "/opt/stack/python-keystoneclient/keystoneclient/client.py", line 409, in _cs_request
[Sun Apr 28 17:05:39 2013] [error]
[Sun Apr 28 17:05:39 2013] [error] TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

You need to use old version of python-keystoneclient. Using git reset, it backs codes to date about grizzly release(roughly 2013/4/4).

cd /opt/stack/python-keystoneclient
git reset --hard 5c37d85944d9eed73ec6dd6254842108386bcc4f

Notes a info for commit 5c37d85944d9eed73ec6dd6254842108386bcc4f below.

commit 5c37d85944d9eed73ec6dd6254842108386bcc4f
Author: Steven Hardy 
Date:   Wed Apr 3 17:14:30 2013 +0100

    Ec2Signer: Initial support for v4 signature verification

    Adds initial support for verifying AWS v4 signatures, tested with
    the latest boto trunk (which now uses v4 signatures by default)

    Change-Id: Id163363e259cf08aa251a7a00ff4293b742cbef6
    blueprint: ec2signer-v4signatures

At last, you need to restart apache2 daemon.

sudo service apache2 restart