【ICLR 2022】RegionViT：从区域到局部的ViT
RegionViT提出从区域到局部的视觉Transformer结构，以金字塔结构引入区域到局部注意替代全局自注意。先生成不同贴片大小的区域和局部令牌，经区域自注意提取全局信息，再通过局部自注意传递给局部令牌，结合相对位置编码。在多视觉任务上表现优异，实现高效且兼具全局感受野与局部性。
RegionViT：从区域到局部的ViT
摘要
近年来，视觉Transformer(VIT)在图像分类方面显示出了与卷积神经网络(CNNs)相当的强大能力。 然而，原始ViT只是直接从自然语言处理中继承了相同的体系结构，而自然语言处理通常没有针对视觉应用进行优化。 基于此，本文提出了一种新的视觉Transformer结构，该结构采用金字塔结构，在视觉Transformer中引入了新的区域到局部的注意而不是全局的自注意。 更具体地说，我们的模型首先从具有不同贴片大小的图像中生成区域令牌和局部令牌，其中每个区域令牌与基于空间位置的一组局部令牌相关联。 区域到局部注意包括两个步骤：首先，区域自注意在所有区域令牌之间提取全局信息，然后局部自注意通过自注意在一个区域令牌和相关的局部令牌之间交换信息。 因此，即使局部自我注意的范围局限于局部区域，但它仍然可以接收到全局信息。 在图像分类、目标和关键点检测、语义分割和动作识别等四个视觉任务上的大量实验表明，我们的方法优于或与包括许多并行工作在内的现有ViT变体相当。
1. RegionViT
由于全局自注意力计算太贵，很多工作提出使用局部自注意力，即在一个小区域内进行全局自注意力，但是局部自注意力又会带来另外一个问题，即感受野过小。为此，本文提出了一种新的从粗到细的Transformer——RegionViT。通过区域令牌进行全局交互，并将区域令牌包含的全局信息通过局部自注意力传递给对应的局部Token。本文方法的整体架构如图2所示：
本文的核心模块是区域到局部的Transformer编码器，主要思想就是通过区域令牌进行全局交互，并将区域令牌包含的全局信息通过局部自注意力传递给对应的局部Token，具体操作如下公式所示：
yrd=xrd−1+RSA(LN(xrd−1)),yi,jd=[yri,jd∥{xli,j,m,nd−1}m,n∈M]zi,jd=yi,jd+LSA(LN(yi,jd)),xi,jd=zi,jd+FFN(LN(zi,jd))yrd=xrd−1+RSA(LN(xrd−1)),yi,jd=[yri,jd∥{xli,j,m,nd−1}m,n∈M]zi,jd=yi,jd+LSA(LN(yi,jd)),xi,jd=zi,jd+FFN(LN(zi,jd))
局部性是理解视觉内容的重要线索。因此，本文提出使用相对位置编码，值得注意的是，该位置编码只添加到局部Token中，不添加区域Token到局部Token的位置编码。具体公式如下：
a(xm,ym),(xn,yn)=softmax(q(xm,ym)k(xn,yn)T+b(xm−xn,ym−yn)),a(xm,ym),(xn,yn)=softmax(q(xm,ym)k(xn,yn)T+b(xm−xn,ym−yn)),
2. 代码复现
2.1 下载并导入所需的库
In [ ]
%matplotlib inlineimport paddleimport numpy as npimport matplotlib.pyplot as pltfrom paddle.vision.datasets import Cifar10from paddle.vision.transforms import Transposefrom paddle.io import Dataset, DataLoaderfrom paddle import nnimport paddle.nn.functional as Fimport paddle.vision.transforms as transformsimport osimport matplotlib.pyplot as pltfrom matplotlib.pyplot import figureimport itertoolsfrom functools import partial
2.2 创建数据集
In [3]
train_tfm = transforms.Compose([
transforms.RandomResizedCrop(224, scale=(0.6, 1.0)),
transforms.ColorJitter(brightness=0.2,contrast=0.2, saturation=0.2),
transforms.RandomHorizontalFlip(0.5),
transforms.RandomRotation(20),
transforms.ToTensor(),
transforms.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
])
test_tfm = transforms.Compose([
transforms.Resize((224, 224)),
transforms.ToTensor(),
transforms.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
])
In [4]
paddle.vision.set_image_backend('cv2')# 使用Cifar10数据集train_dataset = Cifar10(data_file='data/data152754/cifar-10-python.tar.gz', mode='train', transform = train_tfm, )
val_dataset = Cifar10(data_file='data/data152754/cifar-10-python.tar.gz', mode='test',transform = test_tfm)print("train_dataset: %d" % len(train_dataset))print("val_dataset: %d" % len(val_dataset))
train_dataset: 50000 val_dataset: 10000
In [5]
batch_size=64
In [6]
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, drop_last=True, num_workers=4) val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False, drop_last=False, num_workers=4)
2.3 模型的创建
2.3.1 标签平滑
In [7]
class LabelSmoothingCrossEntropy(nn.Layer):
def __init__(self, smoothing=0.1):
super().__init__()
self.smoothing = smoothing def forward(self, pred, target):
confidence = 1. - self.smoothing
log_probs = F.log_softmax(pred, axis=-1)
idx = paddle.stack([paddle.arange(log_probs.shape[0]), target], axis=1)
nll_loss = paddle.gather_nd(-log_probs, index=idx)
smooth_loss = paddle.mean(-log_probs, axis=-1)
loss = confidence * nll_loss + self.smoothing * smooth_loss return loss.mean()
2.3.2 DropPath
In [8]
def drop_path(x, drop_prob=0.0, training=False):
"""
Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).
the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper...
See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ...
"""
if drop_prob == 0.0 or not training: return x
keep_prob = paddle.to_tensor(1 - drop_prob)
shape = (paddle.shape(x)[0],) + (1,) * (x.ndim - 1)
random_tensor = keep_prob + paddle.rand(shape, dtype=x.dtype)
random_tensor = paddle.floor(random_tensor) # binarize
output = x.divide(keep_prob) * random_tensor return outputclass DropPath(nn.Layer):
def __init__(self, drop_prob=None):
super(DropPath, self).__init__()
self.drop_prob = drop_prob def forward(self, x):
return drop_path(x, self.drop_prob, self.training)
2.3.3 RegionViT模型的创建
In [9]
class LayerNorm2D(nn.Layer):
def __init__(self, channels, eps=1e-5, elementwise_affine=True):
super().__init__()
self.channels = channels
self.eps = paddle.to_tensor(eps)
self.elementwise_affine = elementwise_affine if self.elementwise_affine:
self.weight = self.create_parameter(shape=(1, channels, 1, 1), default_initializer=nn.initializer.Constant(1.0))
self.bias = self.create_parameter(shape=(1, channels, 1, 1), default_initializer=nn.initializer.Constant(0.0)) else:
self.register_buffer('weight', None)
self.register_buffer('bias', None) def forward(self, input):
mean = input.mean(1, keepdim=True)
std = paddle.sqrt(input.var(1, unbiased=False, keepdim=True) + self.eps)
out = (input - mean) / std if self.elementwise_affine:
out = out * self.weight + self.bias return out
In [10]
class AttentionWithRelPos(nn.Layer):
def __init__(self, dim, num_heads=8, qkv_bias=False, qk_scale=None, attn_drop=0., proj_drop=0.,
attn_map_dim=None, num_cls_tokens=1):
super().__init__()
self.num_heads = num_heads
head_dim = dim // num_heads
self.scale = qk_scale or head_dim ** -0.5
self.qkv = nn.Linear(dim, dim * 3, bias_attr=qkv_bias)
self.attn_drop = nn.Dropout(attn_drop)
self.proj = nn.Linear(dim, dim)
self.proj_drop = nn.Dropout(proj_drop)
self.num_cls_tokens = num_cls_tokens if attn_map_dim is not None:
one_dim = attn_map_dim[0]
rel_pos_dim = (2 * one_dim - 1)
self.rel_pos = self.create_parameter(shape=(num_heads, rel_pos_dim ** 2), default_initializer=nn.initializer.Constant(0.0))
tmp = paddle.arange(rel_pos_dim ** 2).reshape((rel_pos_dim, rel_pos_dim))
out = []
offset_x = offset_y = one_dim // 2
for y in range(one_dim): for x in range(one_dim): for dy in range(one_dim): for dx in range(one_dim):
out.append(tmp[dy - y + offset_y, dx - x + offset_x])
self.rel_pos_index = paddle.to_tensor(out, dtype=paddle.int32)
tn = nn.initializer.TruncatedNormal(std=.02)
tn(self.rel_pos) else:
self.rel_pos = None
def forward(self, x, patch_attn=False, mask=None):
B, N, C = x.shape
qkv = self.qkv(x).reshape((B, N, 3, self.num_heads, C // self.num_heads)).transpose([2, 0, 3, 1, 4])
q, k, v = qkv[0], qkv[1], qkv[2]
attn = (q @ k.transpose([0, 1, 3, 2])) * self.scale if self.rel_pos is not None and patch_attn: # use for the indicating patch + cls:
rel_pos = self.rel_pos[:, self.rel_pos_index].reshape((self.num_heads, N - self.num_cls_tokens, N - self.num_cls_tokens))
attn[:, :, self.num_cls_tokens:, self.num_cls_tokens:] = attn[:, :, self.num_cls_tokens:, self.num_cls_tokens:] + rel_pos if mask is not None: ## mask is only (BH_sW_s)(ksks)(ksks), need to expand it
mask = mask.unsqueeze(1).expand((-1, self.num_heads, -1, -1))
attn = attn.masked_fill(mask == 0, paddle.finfo(attn.dtype).min)
attn = F.softmax(attn, axis=-1)
attn = self.attn_drop(attn)
x = (attn @ v).transpose([0, 2, 1, 3]).reshape((B, N, C))
x = self.proj(x)
x = self.proj_drop(x) return x
In [11]
def to_2tuple(x):
return (x, x)class PatchEmbed(nn.Layer):
""" Image to Patch Embedding
"""
def __init__(self, img_size=224, patch_size=16, in_chans=3, embed_dim=768, patch_conv_type='linear'):
super().__init__()
img_size = to_2tuple(img_size)
patch_size = to_2tuple(patch_size)
num_patches = (img_size[1] // patch_size[1]) * (img_size[0] // patch_size[0])
self.img_size = img_size
self.patch_size = patch_size
self.num_patches = num_patches if patch_conv_type == '3conv': if patch_size[0] == 4:
tmp = [
nn.Conv2D(in_chans, embed_dim // 4, kernel_size=3, stride=2, padding=1),
LayerNorm2D(embed_dim // 4),
nn.GELU(),
nn.Conv2D(embed_dim // 4, embed_dim // 2, kernel_size=3, stride=2, padding=1),
LayerNorm2D(embed_dim // 2),
nn.GELU(),
nn.Conv2D(embed_dim // 2, embed_dim, kernel_size=3, stride=1, padding=1),
] else: raise ValueError(f"Unknown patch size {patch_size[0]}")
self.proj = nn.Sequential(*tmp) else: if patch_conv_type == '1conv':
kernel_size = (2 * patch_size[0], 2 * patch_size[1])
stride = (patch_size[0], patch_size[1])
padding = (patch_size[0] - 1, patch_size[1] - 1) else:
kernel_size = patch_size
stride = patch_size
padding = 0
self.proj = nn.Conv2D(in_chans, embed_dim, kernel_size=kernel_size,
stride=stride, padding=padding) def forward(self, x, extra_padding=False):
B, C, H, W = x.shape # FIXME look at relaxing size constraints
# assert H == self.img_size[0] and W == self.img_size[1], \
# f"Input image size ({H}*{W}) doesn't match model ({self.img_size[0]}*{self.img_size[1]})."
if extra_padding and (H % self.patch_size[0] != 0 or W % self.patch_size[1] != 0):
p_l = (self.patch_size[1] - W % self.patch_size[1]) // 2
p_r = (self.patch_size[1] - W % self.patch_size[1]) - p_l
p_t = (self.patch_size[0] - H % self.patch_size[0]) // 2
p_b = (self.patch_size[0] - H % self.patch_size[0]) - p_t
x = F.pad(x, (p_l, p_r, p_t, p_b))
x = self.proj(x) return x
In [12]
class Mlp(nn.Layer):
""" MLP as used in Vision Transformer, MLP-Mixer and related networks
"""
def __init__(self, in_features, hidden_features=None, out_features=None, act_layer=nn.GELU, bias=True, drop=0.):
super().__init__()
out_features = out_features or in_features
hidden_features = hidden_features or in_features
bias = to_2tuple(bias)
drop_probs = to_2tuple(drop)
self.fc1 = nn.Linear(in_features, hidden_features, bias_attr=bias[0])
self.act = act_layer()
self.drop1 = nn.Dropout(drop_probs[0])
self.fc2 = nn.Linear(hidden_features, out_features, bias_attr=bias[1])
self.drop2 = nn.Dropout(drop_probs[1]) def forward(self, x):
x = self.fc1(x)
x = self.act(x)
x = self.drop1(x)
x = self.fc2(x)
x = self.drop2(x) return x
In [13]
class R2LAttentionPlusFFN(nn.Layer):
def __init__(self, input_channels, output_channels, kernel_size, num_heads, mlp_ratio=1., qkv_bias=False, qk_scale=None,
act_layer=nn.GELU, norm_layer=nn.LayerNorm, drop_path=0., attn_drop=0., drop=0.,
cls_attn=True):
super().__init__() if not isinstance(kernel_size, (tuple, list)):
kernel_size = [(kernel_size, kernel_size), (kernel_size, kernel_size), 0]
self.kernel_size = kernel_size if cls_attn:
self.norm0 = norm_layer(input_channels) else:
self.norm0 = None
self.norm1 = norm_layer(input_channels)
self.attn = AttentionWithRelPos(
input_channels, num_heads=num_heads, qkv_bias=qkv_bias, qk_scale=qk_scale, attn_drop=attn_drop, proj_drop=drop,
attn_map_dim=(kernel_size[0][0], kernel_size[0][1]), num_cls_tokens=1) # NOTE: drop path for stochastic depth, we shall see if this is better than dropout here
self.drop_path = DropPath(drop_path) if drop_path > 0. else nn.Identity()
self.norm2 = norm_layer(input_channels)
self.mlp = Mlp(in_features=input_channels, hidden_features=int(output_channels * mlp_ratio), out_features=output_channels, act_layer=act_layer, drop=drop)
self.expand = nn.Sequential(
norm_layer(input_channels),
act_layer(),
nn.Linear(input_channels, output_channels)
) if input_channels != output_channels else None
self.output_channels = output_channels
self.input_channels = input_channels def forward(self, xs):
out, B, H, W, mask = xs
cls_tokens = out[:, 0:1, ...]
C = cls_tokens.shape[-1]
cls_tokens = cls_tokens.reshape((B, -1, C)) # (N)x(H/sxW/s)xC
if self.norm0 is not None:
cls_tokens = cls_tokens + self.drop_path(self.attn(self.norm0(cls_tokens))) # (N)x(H/sxK/s)xC
# ks, stride, padding = self.kernel_size
cls_tokens = cls_tokens.reshape((-1, 1, C)) # (NxH/sxK/s)x1xC
out = paddle.concat((cls_tokens, out[:, 1:, ...]), axis=1)
tmp = out
tmp = tmp + self.drop_path(self.attn(self.norm1(tmp), patch_attn=True, mask=mask))
identity = self.expand(tmp) if self.expand is not None else tmp
tmp = identity + self.drop_path(self.mlp(self.norm2(tmp))) return tmp
In [14]
class Projection(nn.Layer):
def __init__(self, input_channels, output_channels, act_layer, mode='sc'):
super().__init__()
tmp = [] if 'c' in mode:
ks = 2 if 's' in mode else 1
if ks == 2:
stride = ks
ks = ks + 1
padding = ks // 2
else:
stride = ks
padding = 0
if input_channels == output_channels and ks == 1:
tmp.append(nn.Identity()) else:
tmp.extend([
LayerNorm2D(input_channels),
act_layer(),
])
tmp.append(nn.Conv2D(in_channels=input_channels, out_channels=output_channels, kernel_size=ks, stride=stride, padding=padding, groups=input_channels))
self.proj = nn.Sequential(*tmp)
self.proj_cls = self.proj def forward(self, xs):
cls_tokens, patch_tokens = xs # x: BxCxHxW
cls_tokens = self.proj_cls(cls_tokens)
patch_tokens = self.proj(patch_tokens) return cls_tokens, patch_tokens
In [15]
def convert_to_flatten_layout(cls_tokens, patch_tokens, ws):
"""
Convert the token layer in a flatten form, it will speed up the model.
Furthermore, it also handle the case that if the size between regional tokens and local tokens are not consistent.
"""
# padding if needed, and all paddings are happened at bottom and right.
B, C, H, W = patch_tokens.shape
_, _, H_ks, W_ks = cls_tokens.shape
need_mask = False
p_l, p_r, p_t, p_b = 0, 0, 0, 0
if H % (H_ks * ws) != 0 or W % (W_ks * ws) != 0:
p_l, p_r = 0, W_ks * ws - W
p_t, p_b = 0, H_ks * ws - H
patch_tokens = F.pad(patch_tokens, (p_l, p_r, p_t, p_b))
need_mask = True
B, C, H, W = patch_tokens.shape
kernel_size = [H // H_ks, W // W_ks]
tmp = F.unfold(patch_tokens, kernel_sizes=kernel_size, strides=kernel_size, paddings=[0, 0]) # Nx(Cxksxks)x(H/sxK/s)
patch_tokens = tmp.transpose([0, 2, 1]).reshape((-1, C, kernel_size[0] * kernel_size[1])).transpose([0, 2, 1]) # (NxH/sxK/s)x(ksxks)xC
if need_mask:
BH_sK_s, ksks, C = patch_tokens.shape
H_s, W_s = H // ws, W // ws
mask = paddle.ones(BH_sK_s // B, 1 + ksks, 1 + ksks, dtype='float32')
right = paddle.zeros(1 + ksks, 1 + ksks, dtype='float32')
tmp = paddle.zeros(ws, ws, dtype='float32')
tmp[0:(ws - p_r), 0:(ws - p_r)] = 1.
tmp = tmp.repeat(ws, ws)
right[1:, 1:] = tmp
right[0, 0] = 1
right[0, 1:] = paddle.to_tensor([1.] * (ws - p_r) + [0.] * p_r).repeat(ws)
right[1:, 0] = paddle.to_tensor([1.] * (ws - p_r) + [0.] * p_r).repeat(ws)
bottom = paddle.zeros_like(right)
bottom[0:ws * (ws - p_b) + 1, 0:ws * (ws - p_b) + 1] = 1.
bottom_right = copy.deepcopy(right)
bottom_right[0:ws * (ws - p_b) + 1, 0:ws * (ws - p_b) + 1] = 1.
mask[W_s - 1:(H_s - 1) * W_s:W_s, ...] = right
mask[(H_s - 1) * W_s:, ...] = bottom
mask[-1, ...] = bottom_right
mask = mask.repeat(B, 1, 1) else:
mask = None
cls_tokens = cls_tokens.flatten(2).transpose([0, 2, 1]) # (N)x(H/sxK/s)xC
cls_tokens = cls_tokens.reshape((-1, 1, cls_tokens.shape[-1])) # (NxH/sxK/s)x1xC
out = paddle.concat((cls_tokens, patch_tokens), axis=1) return out, mask, p_l, p_r, p_t, p_b, B, C, H, Wdef convert_to_spatial_layout(out, output_channels, B, H, W, kernel_size, mask, p_l, p_r, p_t, p_b):
"""
Convert the token layer from flatten into 2-D, will be used to downsample the spatial dimension.
"""
cls_tokens = out[:, 0:1, ...]
patch_tokens = out[:, 1:, ...] # cls_tokens: (BxH/sxW/s)x(1)xC, patch_tokens: (BxH/sxW/s)x(ksxks)xC
C = output_channels
kernel_size = kernel_size[0]
H_ks = H // kernel_size[0]
W_ks = W // kernel_size[1] # reorganize data, need to convert back to cls_tokens: BxCxH/sxW/s, patch_tokens: BxCxHxW
cls_tokens = cls_tokens.reshape((B, -1, C)).transpose([0, 2, 1]).reshape((B, C, H_ks, W_ks))
patch_tokens = patch_tokens.transpose([0, 2, 1]).reshape((B, -1, kernel_size[0] * kernel_size[1] * C)).transpose([0, 2, 1])
patch_tokens = F.fold(patch_tokens, [H, W], kernel_sizes=kernel_size, strides=kernel_size, paddings=[0, 0]) if mask is not None: if p_b > 0:
patch_tokens = patch_tokens[:, :, :-p_b, :] if p_r > 0:
patch_tokens = patch_tokens[:, :, :, :-p_r] return cls_tokens, patch_tokens
In [16]
class ConvAttBlock(nn.Layer):
def __init__(self, input_channels, output_channels, kernel_size, num_blocks, num_heads, mlp_ratio=1., qkv_bias=False, qk_scale=None, pool='sc',
act_layer=nn.GELU, norm_layer=nn.LayerNorm, drop_path_rate=(0.,), attn_drop_rate=0., drop_rate=0.,
cls_attn=True, peg=False):
super().__init__()
tmp = [] if pool:
tmp.append(Projection(input_channels, output_channels, act_layer=act_layer, mode=pool)) for i in range(num_blocks):
kernel_size_ = kernel_size
tmp.append(R2LAttentionPlusFFN(output_channels, output_channels, kernel_size_, num_heads, mlp_ratio, qkv_bias, qk_scale,
act_layer=act_layer, norm_layer=norm_layer, drop_path=drop_path_rate[i], attn_drop=attn_drop_rate, drop=drop_rate,
cls_attn=cls_attn))
self.block = nn.LayerList(tmp)
self.output_channels = output_channels
self.ws = kernel_size if not isinstance(kernel_size, (tuple, list)):
kernel_size = [[kernel_size, kernel_size], [kernel_size, kernel_size], 0]
self.kernel_size = kernel_size
self.peg = nn.Conv2D(output_channels, output_channels, kernel_size=3, padding=1, groups=output_channels, bias=False) if peg else None
def forward(self, xs):
cls_tokens, patch_tokens = xs
cls_tokens, patch_tokens = self.block[0]((cls_tokens, patch_tokens))
out, mask, p_l, p_r, p_t, p_b, B, C, H, W = convert_to_flatten_layout(cls_tokens, patch_tokens, self.ws) for i in range(1, len(self.block)):
blk = self.block[i]
out = blk((out, B, H, W, mask)) if self.peg is not None and i == 1:
cls_tokens, patch_tokens = convert_to_spatial_layout(out, self.output_channels, B, H, W, self.kernel_size, mask, p_l, p_r, p_t, p_b)
cls_tokens = cls_tokens + self.peg(cls_tokens)
patch_tokens = patch_tokens + self.peg(patch_tokens)
out, mask, p_l, p_r, p_t, p_b, B, C, H, W = convert_to_flatten_layout(cls_tokens, patch_tokens, self.ws)
cls_tokens, patch_tokens = convert_to_spatial_layout(out, self.output_channels, B, H, W, self.kernel_size, mask, p_l, p_r, p_t, p_b) return cls_tokens, patch_tokens
In [17]
class RegionViT(nn.Layer):
"""
Note:
The variable naming mapping between codes and papers:
- cls_tokens -> regional tokens
- patch_tokens -> local tokens
"""
def __init__(self, img_size=224, patch_size=16, in_chans=3, num_classes=1000, embed_dim=(768,), depth=(12,),
num_heads=(12,), mlp_ratio=4., qkv_bias=True, qk_scale=None, drop_rate=0., attn_drop_rate=0.,
drop_path_rate=0., norm_layer=partial(nn.LayerNorm, epsilon=1e-6), # regionvit parameters
kernel_sizes=None, downsampling=None,
patch_conv_type='3conv',
computed_cls_token=True, peg=False,
det_norm=False):
super().__init__()
self.num_classes = num_classes
self.kernel_sizes = kernel_sizes
self.num_features = embed_dim[-1] # num_features for consistency with other models
self.embed_dim = embed_dim
self.patch_size = patch_size
self.img_size = img_size
self.patch_embed = PatchEmbed(
img_size=img_size, patch_size=patch_size, in_chans=in_chans, embed_dim=embed_dim[0],
patch_conv_type=patch_conv_type) if not isinstance(mlp_ratio, (list, tuple)):
mlp_ratio = [mlp_ratio] * len(depth)
self.computed_cls_token = computed_cls_token
self.cls_token = PatchEmbed(
img_size=img_size, patch_size=patch_size * kernel_sizes[0], in_chans=in_chans, embed_dim=embed_dim[0],
patch_conv_type='linear'
)
self.pos_drop = nn.Dropout(p=drop_rate)
total_depth = sum(depth)
dpr = [x.item() for x in paddle.linspace(0, drop_path_rate, total_depth)] # stochastic depth decay rule
dpr_ptr = 0
self.layers = nn.LayerList() for i in range(len(embed_dim) - 1):
curr_depth = depth[i]
dpr_ = dpr[dpr_ptr: dpr_ptr + curr_depth]
self.layers.append(
ConvAttBlock(embed_dim[i], embed_dim[i + 1], kernel_size=kernel_sizes[i], num_blocks=depth[i], drop_path_rate=dpr_,
num_heads=num_heads[i], mlp_ratio=mlp_ratio[i], qkv_bias=qkv_bias, qk_scale=qk_scale,
pool=downsampling[i], norm_layer=norm_layer, attn_drop_rate=attn_drop_rate, drop_rate=drop_rate,
cls_attn=True, peg=peg)
)
dpr_ptr += curr_depth
self.norm = norm_layer(embed_dim[-1]) # Classifier head
self.head = nn.Linear(embed_dim[-1], num_classes) if num_classes > 0 else nn.Identity() if not computed_cls_token:
tn = nn.initializer.TruncatedNormal(std=.02)
tn(self.cls_token)
self.det_norm = det_norm if self.det_norm: # add a norm layer for the outputs at each stage, for detection
for i in range(4):
layer = LayerNorm2D(embed_dim[1 + i])
layer_name = f'norm{i}'
self.add_module(layer_name, layer)
self.apply(self._init_weights) def _init_weights(self, m):
tn = nn.initializer.TruncatedNormal(std=.02)
ones = nn.initializer.Constant(1.0)
zeros = nn.initializer.Constant(0.0) if isinstance(m, nn.Linear):
tn(m.weight) if isinstance(m, nn.Linear) and m.bias is not None:
zeros(m.bias) elif isinstance(m, nn.LayerNorm):
zeros(m.bias)
ones(m.weight) def forward_features(self, x, detection=False):
o_x = x
x = self.patch_embed(x) # B x branches x classes
cls_tokens = self.cls_token(o_x, extra_padding=True)
x = self.pos_drop(x) # N C H W
tmp_out = [] for idx, layer in enumerate(self.layers):
cls_tokens, x = layer((cls_tokens, x)) if self.det_norm:
norm_layer = getattr(self, f'norm{idx}')
x = norm_layer(x)
tmp_out.append(x) if detection: return tmp_out
N, C, H, W = cls_tokens.shape
cls_tokens = cls_tokens.reshape((N, C, -1)).transpose([0, 2, 1])
cls_tokens = self.norm(cls_tokens)
out = paddle.mean(cls_tokens, axis=1) return out def forward(self, x):
x = self.forward_features(x)
x = self.head(x) return x
In [18]
_model_cfg = { 'tiny': { 'img_size': 224, 'patch_conv_type': '3conv', 'patch_size': 4, 'embed_dim': [64, 64, 128, 256, 512], 'num_heads': [2, 4, 8, 16], 'mlp_ratio': 4., 'depth': [2, 2, 8, 2], 'kernel_sizes': [7, 7, 7, 7], # 8x8, 4x4, 2x2, 1x1,
'downsampling': ['c', 'sc', 'sc', 'sc'],
}, 'small': { 'img_size': 224, 'patch_conv_type': '3conv', 'patch_size': 4, 'embed_dim': [96, 96, 192, 384, 768], 'num_heads': [3, 6, 12, 24], 'mlp_ratio': 4., 'depth': [2, 2, 8, 2], 'kernel_sizes': [7, 7, 7, 7], # 8x8, 4x4, 2x2, 1x1,
'downsampling': ['c', 'sc', 'sc', 'sc'],
}, 'medium': { 'img_size': 224, 'patch_conv_type': '1conv', 'patch_size': 4, 'embed_dim': [96] + [96 * (2 ** i) for i in range(4)], 'num_heads': [3, 6, 12, 24], 'mlp_ratio': 4., 'depth': [2, 2, 14, 2], 'kernel_sizes': [7, 7, 7, 7], # 8x8, 4x4, 2x2, 1x1,
'downsampling': ['c', 'sc', 'sc', 'sc'],
}, 'base': { 'img_size': 224, 'patch_conv_type': '1conv', 'patch_size': 4, 'embed_dim': [128, 128, 256, 512, 1024], 'num_heads': [4, 8, 16, 32], 'mlp_ratio': 4., 'depth': [2, 2, 14, 2], 'kernel_sizes': [7, 7, 7, 7], # 8x8, 4x4, 2x2, 1x1,
'downsampling': ['c', 'sc', 'sc', 'sc'],
}, 'small_w14': { 'img_size': 224, 'patch_conv_type': '3conv', 'patch_size': 4, 'embed_dim': [96, 96, 192, 384, 768], 'num_heads': [3, 6, 12, 24], 'mlp_ratio': 4., 'depth': [2, 2, 8, 2], 'kernel_sizes': [14, 14, 14, 14], # 8x8, 4x4, 2x2, 1x1,
'downsampling': ['c', 'sc', 'sc', 'sc'],
}, 'small_w14_peg': { 'img_size': 224, 'patch_conv_type': '3conv', 'patch_size': 4, 'embed_dim': [96, 96, 192, 384, 768], 'num_heads': [3, 6, 12, 24], 'mlp_ratio': 4., 'depth': [2, 2, 8, 2], 'kernel_sizes': [14, 14, 14, 14], # 8x8, 4x4, 2x2, 1x1,
'downsampling': ['c', 'sc', 'sc', 'sc'], 'peg': True
}, 'base_w14': { 'img_size': 224, 'patch_conv_type': '1conv', 'patch_size': 4, 'embed_dim': [128, 128, 256, 512, 1024], 'num_heads': [4, 8, 16, 32], 'mlp_ratio': 4., 'depth': [2, 2, 14, 2], 'kernel_sizes': [14, 14, 14, 14], # 8x8, 4x4, 2x2, 1x1,
'downsampling': ['c', 'sc', 'sc', 'sc'],
}, 'base_w14_peg': { 'img_size': 224, 'patch_conv_type': '1conv', 'patch_size': 4, 'embed_dim': [128, 128, 256, 512, 1024], 'num_heads': [4, 8, 16, 32], 'mlp_ratio': 4., 'depth': [2, 2, 14, 2], 'kernel_sizes': [14, 14, 14, 14], # 8x8, 4x4, 2x2, 1x1,
'downsampling': ['c', 'sc', 'sc', 'sc'], 'peg': True
},
}
In [19]
num_classes = 10def regionvit_tiny_224():
model_cfg = _model_cfg['tiny']
model = RegionViT(**model_cfg, num_classes=num_classes) return modeldef regionvit_small_224():
model_cfg = _model_cfg['small']
model = RegionViT(**model_cfg, num_classes=num_classes) return modeldef regionvit_small_w14_224():
model_cfg = _model_cfg['small_w14']
model = RegionViT(**model_cfg, num_classes=num_classes) return modeldef regionvit_small_w14_peg_224():
model_cfg = _model_cfg['small_w14_peg']
model = RegionViT(**model_cfg, num_classes=num_classes) return modeldef regionvit_medium_224():
model_cfg = _model_cfg['medium']
model = RegionViT(**model_cfg, num_classes=num_classes) return modeldef regionvit_base_224():
model_cfg = _model_cfg['base']
model = RegionViT(**model_cfg, num_classes=num_classes) return modeldef regionvit_base_w14_224():
model_cfg = _model_cfg['base_w14']
model = RegionViT(**model_cfg, num_classes=num_classes) return modeldef regionvit_base_w14_peg_224():
model_cfg = _model_cfg['base_w14_peg']
model = RegionViT(**model_cfg, num_classes=num_classes) return model
2.3.4 模型的参数
In [ ]
model = regionvit_tiny_224() paddle.summary(model, (1, 3, 224, 224))
2.4 训练
In [22]
learning_rate = 0.0001n_epochs = 100paddle.seed(42) np.random.seed(42)
In [ ]
work_path = 'work/model'# RegionViT-Tinymodel = regionvit_tiny_224()
criterion = LabelSmoothingCrossEntropy()
scheduler = paddle.optimizer.lr.CosineAnnealingDecay(learning_rate=learning_rate, T_max=50000 // batch_size * n_epochs, verbose=False)
optimizer = paddle.optimizer.Adam(parameters=model.parameters(), learning_rate=scheduler, weight_decay=1e-5)
gate = 0.0threshold = 0.0best_acc = 0.0val_acc = 0.0loss_record = {'train': {'loss': [], 'iter': []}, 'val': {'loss': [], 'iter': []}} # for recording lossacc_record = {'train': {'acc': [], 'iter': []}, 'val': {'acc': [], 'iter': []}} # for recording accuracyloss_iter = 0acc_iter = 0for epoch in range(n_epochs): # ---------- Training ----------
model.train()
train_num = 0.0
train_loss = 0.0
val_num = 0.0
val_loss = 0.0
accuracy_manager = paddle.metric.Accuracy()
val_accuracy_manager = paddle.metric.Accuracy() print("#===epoch: {}, lr={:.10f}===#".format(epoch, optimizer.get_lr())) for batch_id, data in enumerate(train_loader):
x_data, y_data = data
labels = paddle.unsqueeze(y_data, axis=1)
logits = model(x_data)
loss = criterion(logits, y_data)
acc = accuracy_manager.compute(logits, labels)
accuracy_manager.update(acc) if batch_id % 10 == 0:
loss_record['train']['loss'].append(loss.numpy())
loss_record['train']['iter'].append(loss_iter)
loss_iter += 1
loss.backward()
optimizer.step()
scheduler.step()
optimizer.clear_grad()
train_loss += loss
train_num += len(y_data)
total_train_loss = (train_loss / train_num) * batch_size
train_acc = accuracy_manager.accumulate()
acc_record['train']['acc'].append(train_acc)
acc_record['train']['iter'].append(acc_iter)
acc_iter += 1
# Print the information.
print("#===epoch: {}, train loss is: {}, train acc is: {:2.2f}%===#".format(epoch, total_train_loss.numpy(), train_acc*100)) # ---------- Validation ----------
model.eval() for batch_id, data in enumerate(val_loader):
x_data, y_data = data
labels = paddle.unsqueeze(y_data, axis=1) with paddle.no_grad():
logits = model(x_data)
loss = criterion(logits, y_data)
acc = val_accuracy_manager.compute(logits, labels)
val_accuracy_manager.update(acc)
val_loss += loss
val_num += len(y_data)
total_val_loss = (val_loss / val_num) * batch_size
loss_record['val']['loss'].append(total_val_loss.numpy())
loss_record['val']['iter'].append(loss_iter)
val_acc = val_accuracy_manager.accumulate()
acc_record['val']['acc'].append(val_acc)
acc_record['val']['iter'].append(acc_iter) print("#===epoch: {}, val loss is: {}, val acc is: {:2.2f}%===#".format(epoch, total_val_loss.numpy(), val_acc*100)) # ===================save====================
if val_acc > best_acc:
best_acc = val_acc
paddle.save(model.state_dict(), os.path.join(work_path, 'best_model.pdparams'))
paddle.save(optimizer.state_dict(), os.path.join(work_path, 'best_optimizer.pdopt'))print(best_acc)
paddle.save(model.state_dict(), os.path.join(work_path, 'final_model.pdparams'))
paddle.save(optimizer.state_dict(), os.path.join(work_path, 'final_optimizer.pdopt'))
2.5 结果分析
In [24]
def plot_learning_curve(record, title='loss', ylabel='CE Loss'):
''' Plot learning curve of your CNN '''
maxtrain = max(map(float, record['train'][title]))
maxval = max(map(float, record['val'][title]))
ymax = max(maxtrain, maxval) * 1.1
mintrain = min(map(float, record['train'][title]))
minval = min(map(float, record['val'][title]))
ymin = min(mintrain, minval) * 0.9
total_steps = len(record['train'][title])
x_1 = list(map(int, record['train']['iter']))
x_2 = list(map(int, record['val']['iter']))
figure(figsize=(10, 6))
plt.plot(x_1, record['train'][title], c='tab:red', label='train')
plt.plot(x_2, record['val'][title], c='tab:cyan', label='val')
plt.ylim(ymin, ymax)
plt.xlabel('Training steps')
plt.ylabel(ylabel)
plt.title('Learning curve of {}'.format(title))
plt.legend()
plt.show()
In [25]
plot_learning_curve(loss_record, title='loss', ylabel='CE Loss')
In [26]
plot_learning_curve(acc_record, title='acc', ylabel='Accuracy')
In [27]
import time
work_path = 'work/model'model = regionvit_tiny_224()
model_state_dict = paddle.load(os.path.join(work_path, 'best_model.pdparams'))
model.set_state_dict(model_state_dict)
model.eval()
aa = time.time()for batch_id, data in enumerate(val_loader):
x_data, y_data = data
labels = paddle.unsqueeze(y_data, axis=1) with paddle.no_grad():
logits = model(x_data)
bb = time.time()print("Throughout:{}".format(int(len(val_dataset)//(bb - aa))))
Throughout:678
In [28]
def get_cifar10_labels(labels):
"""返回CIFAR10数据集的文本标签。"""
text_labels = [ 'airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'] return [text_labels[int(i)] for i in labels]
In [29]
def show_images(imgs, num_rows, num_cols, pred=None, gt=None, scale=1.5):
"""Plot a list of images."""
figsize = (num_cols * scale, num_rows * scale)
_, axes = plt.subplots(num_rows, num_cols, figsize=figsize)
axes = axes.flatten() for i, (ax, img) in enumerate(zip(axes, imgs)): if paddle.is_tensor(img):
ax.imshow(img.numpy()) else:
ax.imshow(img)
ax.axes.get_xaxis().set_visible(False)
ax.axes.get_yaxis().set_visible(False) if pred or gt:
ax.set_title("pt: " + pred[i] + "\ngt: " + gt[i]) return axes
In [30]
work_path = 'work/model'X, y = next(iter(DataLoader(val_dataset, batch_size=18))) model = regionvit_tiny_224() model_state_dict = paddle.load(os.path.join(work_path, 'best_model.pdparams')) model.set_state_dict(model_state_dict) model.eval() logits = model(X) y_pred = paddle.argmax(logits, -1) X = paddle.transpose(X, [0, 2, 3, 1]) axes = show_images(X.reshape((18, 224, 224, 3)), 1, 18, pred=get_cifar10_labels(y_pred), gt=get_cifar10_labels(y)) plt.show()
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
总结
本文提出了一种从区域到局部的一种从粗到细的Transformer，既具有全局的感受野，又具有局部性，实现简单高效。
以上就是【ICLR 2022】RegionViT：从区域到局部的ViT的详细内容，更多请关注启程网【www.vszh.cn】AI 人工智能栏目。
本文地址：https://www.vszh.cn/article/1160839.html
如非特殊说明，本站内容均来自于网友自主分享，如有任何问题均请联系我们进行处理！
猜您喜欢
《二重螺旋》全新区域介绍
2026-06-18二重螺旋全新区域怎么样？本次《二重螺旋》手游迎来重大版本更新，全新区域“芙罗蒂娅百花车站”正式上线。该区域位于冰湖城内城，是科赛托斯省核心交通节点，兼具战略地位与叙事深度。建筑外观以粗犷钢铁结构为主，呈现典型的工业美学风格；而其地下空间则隐藏着持续扩建中的大型锻铁厂，形成地上车站与地下工业体共生的独特空间结构。 《二重螺旋》全新区域介绍 步入百花车站内部，节奏感强烈的环境氛围扑面而来：人流穿梭于站...
Cinema 4D如何使用区域光
2026-06-02在cinema4d这款功能强大的三维建模与动画软件中，区域光是一项极具表现力的照明工具，能够精准塑造空间氛围并实现细腻自然的光影层次。接下来我们就来系统讲解区域光的使用技巧。 创建区域光 启动Cinema4D后，进入目标项目场景。点击顶部菜单栏中的“创建”→“灯光”→“区域光”，即可在视图中生成一个默认的区域光源。 设置区域光核心参数 创建完成后，选中该灯光，在右侧属性面板中可对其关键参数进行调节...
紫鸟浏览器怎么管理语言区域_紫鸟浏览器管理语言区域【指南】
2026-05-30紫鸟浏览器语言区域设置有五种方法：一、账号管理后台配置；二、全局设置声明默认语言；三、工作区绑定强制模拟；四、右键菜单实时翻译；五、生态中心安装多语言插件。 如果您在使用紫鸟浏览器时发现界面语言非简体中文，或访问境外平台时网页内容未按中文环境加载、本地化资源缺失，则可能是语言区域设置未正确配置。以下是管理紫鸟浏览器语言区域的具体操作步骤： 一、通过账号管理后台配置语言区域 该方法直接作用于指定账号...
Lightroom怎么局部调整色彩 LR局部调整方法【指南】
2026-05-29Lightroom局部调色有五种核心方法：调整画笔精准涂抹单色区域；颜色范围蒙版锁定指定色相；HSL点选调色快速响应；渐变滤镜统一压暗天空；径向滤镜反向营造胶片暗角。 你想只让照片里的一片晚霞更红、让模特的嘴唇更饱满、让玻璃反光处的蓝色更通透，又不想牵动整张图的色调——Lightroom局部调整色彩正是解决这类需求的核心能力。 用调整画笔精准涂抹单色区域 这是最自由也最常用的方法，适合不规则形状、...
-
异环新增了哪些地图区域
2026-05-26神秘森林 这是《异环》中全新开放的一处洋溢着魔法气息的自然秘境。林木参天，枝干虬曲，不少古树表皮缠绕着散发幽蓝微光的奇异藤蔓，随风轻颤，仿佛拥有生命。步入其间，耳畔萦绕清越鸟啼与若隐若现的灵兽低吟，光影在层层叠叠的叶隙间流转，营造出如梦似幻的沉浸氛围。区域内散落大量高价值采集点——稀有草药、活性苔藓及元素凝露等资源密集分布，是采集流玩家不可错过的丰饶之地。但需格外提防潜伏于灌木丛、树冠阴影或藤蔓垂...
通义万象的Inpainting局部重绘功能在修改生成图片中局部细节时精确度怎么样？
2026-05-22通义万象Inpainting局部重绘功能具备高精度蒙版控制、强语义理解、结构保真、多尺度稳定及跨模型协同五大维度精度：支持亚像素蒙版、CLIP+MMDiT语义驱动、VAE潜变量结构保留、denoise全梯度可控、ControlNet多约束增强，实现自然融合与上下文一致。 ☞☞☞AI智能聊天,问答助手,AI智能搜索,多模态理解力帮你轻松跨越从0到1的创作门槛☜☜☜ 如果您在使用通义万象生成图像后，发...
自然奇观冰间湖主要分布在哪些区域
2026-05-21自然奇观“冰间湖”主要分布在哪些区域？【神奇海洋】2026年5月20日新题目已开答！正确答案现已公布，点击领取拼图，一起为修复海洋生态贡献力量！今日正确答案是：【两极地区】。自然奇观“冰间湖”主要分布在两极地区。 >>>>2026支付宝：蚂蚁庄园今日课堂答题大全 >>>>2026支付宝：蚂蚁新村今天正确答案大全 >>>>20...
通义万象生成的图片如何进行局部修改？涂抹重绘功能使用教程
2026-05-20通义万相“图像局部重绘”功能支持精准修改图片特定区域：一、上传原图进入局部重绘模块；二、用中等画笔精确涂抹目标区域；三、输入明确无冲突的中文提示词并设置重绘强度60%–75%；四、生成4张候选图，可微调优化；五、多区域修改需分步进行。 ☞☞☞AI智能聊天,问答助手,AI智能搜索,多模态理解力帮你轻松跨越从0到1的创作门槛☜☜☜ 如果您已生成一张通义万相图片，但需调整其中特定区域（如更换服装、修改配...