今年参加了不少活动，前两天去了 WayToAGI 的线下，发现很多非技术背景的"一人公司"，已经用 AI 工具做出了非常好的产品，而且取得了不错的收益。他们不懂代码，不懂模型原理，但就是能把工具用好，把产品做出来，把钱赚到。视频生成现在简单到离谱——一句话或者一个简单的分镜，就能生成效果很好的视频。主要用的是 Google 的工具，还有 MiniMax。以前觉得视频生成还早，现在发现已经能用了。
Operational Info ---------------- State : Active Physical state : LinkUp Speed : IB-NDR Width : 4x FEC : Ethernet_Consortium_LL_50G_RS_FEC_PLR -(272,257+1) Loopback Mode : No Loopback Auto Negotiation : ON
Supported Info -------------- Enabled Link Speed : 0x00000080 (NDR) Supported Cable Speed : 0x00000080 (NDR)
Troubleshooting Info -------------------- Status Opcode : 0 Group Opcode : N/A Recommendation : No issue was observed
Tool Information ---------------- Firmware Version : 28.39.3004 amBER Version : 2.22 MFT Version : mft 4.26.1-3
Physical Counters and BER Info ------------------------------ Time Since Last Clear [Min] : 1744.3 Symbol Errors : 0 Symbol BER : 15E-255 Effective Physical Errors : 16 Effective Physical BER : 3E-16 Raw Physical Errors Per Lane : 185219462,14538251,3096793152,798004665 Raw Physical BER : 1E-7 Link Down Counter : 0 Link Error Recovery Counter : 0
Hierarchical Load Balancing When the number of server nodes divides the number of expert groups, we use the hierarchical load balancing policy to harness the group-limited expert routing. We first pack the expert groups to nodes evenly, ensuring the loads of different nodes are balanced. Then, we replicate the experts within each node. Finally, we pack the replicated experts to individual GPUs to ensure different GPUs are load-balanced. The hierarchical load balancing policy can be used in prefilling stage with a smaller expert-parallel size.
Global Load Balancing In other cases, we use the global load balancing policy that replicates the experts globally regardless of expert groups, and pack the replicated experts to individual GPUs. This policy can be adopted in decoding stage with a larger expert-parallel size.
Here, dispatch uses FP8 (1 byte), while combine uses BF16 (2 bytes), and the hidden size of each token is approximately 7K. The factor 9 indicates that each token is ransferred to 8 routed experts and 1 shared expert.
# 遍历所有容器 for container_id in $(docker ps -aq) do # 使用docker inspect命令获取容器信息，并查找文件系统层ID if docker inspect $container_id | grep -q "$fslayer_id" then # 如果找到了，打印容器ID并退出脚本 echo"File system layer $fslayer_id is in container: $container_id" exit 0 fi done
# 逐个检查所有的镜像 for image_id in $(docker images -q) do # 使用docker history命令检查镜像的每一层 if docker history$image_id | grep -q "$fslayer_id" then # 如果找到了，打印镜像ID并退出脚本 echo"OverlayFS layer $fslayer_id is in image: $image_id" exit 0 fi done
# 如果遍历所有容器后还没有找到，打印错误信息 echo"File system layer $fslayer_id was not found in any container or image" exit 1
from components.frontend import Frontend from components.kv_router import Router from components.prefill_worker import PrefillWorker from components.processor import Processor from components.worker import VllmWorker
We observe a performance improvement in the CUTLASS FP8 kernel between NVCC 12.2 and 12.3. By comparing the compiled SASS, we discover that one bit in a series of FADD instructions is flipped in an interleaving pattern. After referencing some open-source CUDA assembler implementations, we identified that this bit controls yield, which may enhance warp-level parallelism (just a guess, yielding the current warp and let other warps work).
No plans atm. Squash was never taken out of experimental and is only supported in moby outside the builder component. You should squash layers with a multi-stage dockerfile(if you are sure you know what you are doing and need to squash at all).
#!/bin/sh ################################################################################ # The setups in this file belong to https://code.shin.company/docker-squash # I appreciate you respecting my intellectual efforts in creating them. # If you intend to copy or use ideas from this project, please credit properly. # Author: SHIN Company <shin@shin.company> # License: https://code.shin.company/docker-squash/blob/main/LICENSE ################################################################################
# Check if Docker is installed, if not exit if [ ! -x "$(command -v docker)" ]; then echo"Docker is not installed. Please install Docker and try again." >&2 exit 1 fi
usage() { cat <<EOF Combines Docker image layers into a single layer, reducing storage space and improving runtime performance by decreasing mount points. Usage: ${0##*/} <source_image> [build_options] Arguments: source_image The original image ID or name:tag to be squashed. An absolute path to your Dockerfile can also be used. build_options Optional Docker build options like --build-arg, --label, etc. Author: SHIN Company <shin@shin.company> License: https://code.shin.company/docker-squash/blob/main/LICENSE EOF }
if [ -z "$id" ]; then echo"Invalid docker image: $1. Please build or pull the image first." >&2 exit 1 fi
local base="scratch" localalias="temp-$id" local tag="$(get_name "$id")" local labels="$(get_labels "$id")" local shell="$(get_shell "$id")" local envs="$(get_envs "$id")" local onbuilds="$(get_onbuilds "$id")" local exposes="$(get_exposes "$id")" local workdir="$(get_workdir "$id")" local user="$(get_user "$id")" local volumes="$(get_volumes "$id")" local entrypoint="$(get_entrypoint "$id")" local cmd="$(get_cmd "$id")" local stopsignal="$(get_stopsignal "$id")" local healthcheck="$(get_healthcheck "$id")"
awk NF <<Dockerfile ################################################################################ # Base image: $tag (Image ID: $id) # Created at: $(date -u +"%Y-%m-%dT%H:%M:%SZ") # Created by: https://code.shin.company/docker-squash ################################################################################ # CLEANING UP THE SOURCE IMAGE. ################################################ # Enable SBOM attestations # See: https://docs.docker.com/build/attestations/sbom/ ARG BUILDKIT_SBOM_SCAN_CONTEXT=true FROM $tag AS $alias ARG BUILDKIT_SBOM_SCAN_STAGE=true # Pre-squash scripts may be useful to clean the source image before squashing. # Use build argument to add your pre-squash scripts, and run them in this stage. # Example: # ${0##*/} $tag --build-arg PRESQUASH_SCRIPTS="rm -rf /tmp/*" # ${0##*/} $tag --build-arg PRESQUASH_SCRIPTS="/path/to/script.sh" ARG PRESQUASH_SCRIPTS="\${PRESQUASH_SCRIPTS:-rm -rf /tmp/* /usr/share/doc/* /var/cache/* /var/lib/apt/lists/* /var/log/*}" RUN [ ! -z "\$PRESQUASH_SCRIPTS" ] && sh -c "\$PRESQUASH_SCRIPTS" || true ################################################################################ # BUILDING SQUASHED IMAGE FROM SCRATCH. ######################################## FROM $base AS squashed-$id ARG BUILDKIT_SBOM_SCAN_STAGE=true COPY --link --from=$alias / / $(if [ -n "$labels" ]; then echo "$labels"; fi) $(if [ -n "$shell" ]; then echo "SHELL $shell"; fi) $(if [ -n "$envs" ]; then echo "$envs"; fi) $(if [ -n "$onbuilds" ]; then echo "$onbuilds"; fi) $(if [ -n "$exposes" ]; then echo "$exposes"; fi) $(if [ -n "$workdir" ]; then echo "WORKDIR $workdir"; fi) $(if [ -n "$user" ]; then echo "USER $user"; fi) $(if [ -n "$volumes" ]; then echo "$volumes"; fi) $(if [ -n "$entrypoint" ]; then echo "ENTRYPOINT $entrypoint"; fi) $(if [ -n "$cmd" ]; then echo "CMD $cmd"; fi) $(if [ -n "$stopsignal" ]; then echo "STOPSIGNAL $stopsignal"; fi) $(if [ -n "$healthcheck" ]; then echo "HEALTHCHECK $healthcheck"; fi) # FINISH. ###################################################################### ################################################################################ Dockerfile }
# Check Source Image / Build From Dockerfile ################################################################################
# Enable debug mode if [ ! -z "$DEBUG" ]; thenset -ex; fi
# Build tempoprary image from Dockerfile if [ -f "$dockerfile" ]; then hash="$(sha256sum "$dockerfile")/$(echo $@ | sha256sum)" temptag="docker-squash-build-$(echo $hash | sha256sum | head -c 8)" context="$(dirname "$dockerfile")"
if ! docker build -f "$dockerfile""$context""$@" -t "$temptag"; then echo"Failed to build image from $dockerfile." >&2 exit 1 fi
source="$temptag" cleanup="${CLEAR_TEMP_BUILD:-1}" else # try pulling the docker image from Docker Hub if [ -z "$(get_name "$dockerfile")" ]; then docker pull "$dockerfile" 2>&1 fi
source="$(get_name "$dockerfile")" fi
# Check if source exists if [ -z "$source" ]; then echo"Invalid image data from '$dockerfile'. Please build or pull the image first." >&2 exit 1 fi
# Squash / Output to New Dockerfile ################################################################################
# Print Dockerfile to stdout if [ ! -z "$print" ]; then generate "$source" exit 0 fi
# Use Docker buildx if available BUILD_CMD="docker build" if docker buildx version &>/dev/null; then BUILD_CMD="docker buildx build" fi
FROM $tag AS $alias ARG BUILDKIT_SBOM_SCAN_STAGE=true # Pre-squash scripts may be useful to clean the source image before squashing. # Use build argument to add your pre-squash scripts, and run them in this stage. # Example: # ${0##*/} $tag --build-arg PRESQUASH_SCRIPTS="rm -rf /tmp/*" # ${0##*/} $tag --build-arg PRESQUASH_SCRIPTS="/path/to/script.sh" ARG PRESQUASH_SCRIPTS="\${PRESQUASH_SCRIPTS:-rm -rf /tmp/* /usr/share/doc/* /var/cache/* /var/lib/apt/lists/* /var/log/*}" RUN [ ! -z "\$PRESQUASH_SCRIPTS" ] && sh -c "\$PRESQUASH_SCRIPTS" || true
################################################################################ # BUILDING SQUASHED IMAGE FROM SCRATCH. ######################################## FROM $base AS squashed-$id ARG BUILDKIT_SBOM_SCAN_STAGE=true COPY --link --from=$alias / / $(if [ -n "$labels" ]; then echo "$labels"; fi) $(if [ -n "$shell" ]; then echo "SHELL $shell"; fi) $(if [ -n "$envs" ]; then echo "$envs"; fi) $(if [ -n "$onbuilds" ]; then echo "$onbuilds"; fi) $(if [ -n "$exposes" ]; then echo "$exposes"; fi) $(if [ -n "$workdir" ]; then echo "WORKDIR $workdir"; fi) $(if [ -n "$user" ]; then echo "USER $user"; fi) $(if [ -n "$volumes" ]; then echo "$volumes"; fi) $(if [ -n "$entrypoint" ]; then echo "ENTRYPOINT $entrypoint"; fi) $(if [ -n "$cmd" ]; then echo "CMD $cmd"; fi) $(if [ -n "$stopsignal" ]; then echo "STOPSIGNAL $stopsignal"; fi) $(if [ -n "$healthcheck" ]; then echo "HEALTHCHECK $healthcheck"; fi)
Create a minimal base image using scratch The reserved, minimal scratch image serves as a starting point for building containers. Using the scratch image signals to the build process that you want the next command in the Dockerfile to be the first filesystem layer in your image.
