[BUG] CI pipe line was failed

[sio-funmatsu]

Description

I noticed that the CI pipeline for the iotedgedev project seems to be failing, and as a result, the latest version of the package is not being updated on PyPI. I wanted to bring this to your attention as it is affecting our ability to use the latest features and fixes.

Expected behavior

The CI pipeline should succeed, and the latest version of iotedgedev should be published on PyPI.

Actual behavior

The CI pipeline is failing, and the PyPI version is not being updated to reflect the latest changes.

Steps to Reproduce

Rerun the CI

Environment

v3.3.8

[maggiojoe2]

Adding more context as I am having the same issue as of a few weeks ago. This is the error I am seeing when I enable debugging:

##[debug]found: '/usr/bin/sudo'
##[debug]/usr/bin/sudo arg: pip3 install iotedgedev==3.3.3
##[debug]/usr/bin/sudo arg: pip3 install iotedgedev==3.3.3
##[debug]   install
##[debug]   iotedgedev==3.3.3
##[debug]   iotedgedev==3.3.3
##[debug]
##[debug]ERROR: Could not find a version that satisfies the requirement iotedgedev==3.3.3 (from versions: 0.79.0, 0.80.0, 0.81.0, 0.82.0, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 2.0.0, 2.0.1, 2.0.2, 2.1.0)
ERROR: No matching distribution found for iotedgedev==3.3.3

##[debug]which 'iotedgedev'
##[debug]not found
##[debug]task result: Failed
##[error]Error: Unable to locate executable file: 'iotedgedev'. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.

I am using ubuntu-latest for my vmImage. I have tried with other versions of ubuntu with no luck.

Previously, I had manually installed iotedgedev with pip, but that is also throwing errors.

[ko16a46]

I ran into this issue as well. This is actually a bug observed when using the AzureIoTEdge@2 task in the Azure Pipeline Tasks repo, but it seems the root cause for the issue is the python dependency logic in this project

Like maggiojoe2 I tried prior workarounds like using other versions of ubuntu, manually installing iotedgedev with pip, or setting the IOTEDGEDEV_VERSION to 2.1.0. All of these failed for me. Not to mention, the solution isn't ideal because we end up using an older version of iotedgedev

In the end, I got it working by switching to windows-2022 as the vmImage, but this now causes issues with building edge modules on Windows. Some form of python dependency pinning needs to be done in this project perhaps, at least according to this guy's suggestions.

[ko16a46]

Update: I got it working by using the UsePythonVersion task to set the python version to 3.9 (which is apparently the latest version supported by iotedgehubdev which is a dependency of iotedgedev).

Then I manually installed iotedgedev with a bash script pip3 install iotedgedev.
The AzureIoTEdge task uses sudo pip3 install iotedgedev under the hood, but the sudo version of pip3 isn't affected by UsePythonVersion so the install fails. By manually installing iotedgedev after lowering the python version, the CI pipeline now works

[xjsxujingsong]

Hi Brian Robles (@ko16a46) We used to use Ubuntu 20.04 to install iotedgedev for CI/CD pipeline, but yesterday, Microsoft mentioned that they will not support 20.04. We switched to 22.04 or 24.04 in which python version is higher than 3.9. We tried a few ways to use python 3.9 in 22.04 but failed. Anyone has a solution here?

[ko16a46]

Hi @ko16a46 We used to use Ubuntu 20.04 to install iotedgedev for CI/CD pipeline, but yesterday, Microsoft mentioned that they will not support 20.04. We switched to 22.04 or 24.04 in which python version is higher than 3.9. We tried a few ways to use python 3.9 in 22.04 but failed. Anyone has a solution here?

In our case, we use ubuntu-22.04, and we use the UsePythonVersion@0 task. Something like below

steps:
- task: UsePythonVersion@0
  displayName: 'Setup python'
  inputs:
    versionSpec: '3.9'
    architecture: 'x64'

- bash: |
    echo "Install iotedgedev"
    pip3 install iotedgedev
    iotedgedev --version
  displayName: 'Setup IoT Edge'

Additionally, we couldn't use iotedgedev to build the modules, although in our case it was mainly due to trying to build cross-platform (the pipeline agent is amd64 but our edge modules target arm). We had to build our edge modules using docker build. For reference, we had something like the following:

steps:
  // ...
  - bash: |
      # Based on answers in this Q&A link
      # https://learn.microsoft.com/en-gb/answers/questions/1387720/azure-iot-edge-image-build-for-amd64-arm64v8-in-az

      echo "Install binfmt (cross-platform emulator collection)"
      docker run --privileged --rm tonistiigi/binfmt --install all
      docker buildx create --name multiarchbuilder
      docker buildx use multiarchbuilder
      docker buildx inspect --bootstrap
      docker buildx install
      docker buildx ls

      # https://github.com/docker/buildx/issues/1170
      docker run --rm --privileged multiarch/qemu-user-static --reset -p yes -c yes
    displayName: 'Setup Docker'

  - task: Bash@3
    displayName: 'Build modules'
    inputs:
      filePath: scripts/build_modules.sh
      arguments: '-p arm64v8 -t $(Build.SourceVersion)'

where build_modules.sh is a script we wrote to build modules similar to iotedgedev

#!/bin/bash

# Builds Azure IoT Edge modules in the modules folder
#
# Unlike `iotedgedev build`, this script correctly sets the target platform
# so that modules can be cross-compiled for other architectures
#
# Usage:
#   ./scripts/build_modules.sh [-p PLATFORM] [-t TAG]
#
# Options:
#   -p  IoT Edge platform to build the image for. Defaults to arm64v8
#   -t  Additional suffix tag to use for the built image

set -e

# Parse command line arguments
PLATFORM="arm64v8"
DOCKER_PLATFORM="linux/arm64"

while getopts "p:t:" opt; do
  case $opt in
    t)
        TAG+=("$OPTARG")
        ;;
    p)
        PLATFORM=$OPTARG
        PLATFORM_BASE=${PLATFORM%%.*}
        case $PLATFORM_BASE in
          arm64v8)
            DOCKER_PLATFORM="linux/arm64"
            ;;
          amd64)
            DOCKER_PLATFORM="linux/amd64"
            ;;
          *)
            echo "Invalid platform: $PLATFORM_BASE" >&2
            exit 1
            ;;
        esac
        ;;
    \?)
      echo "Invalid option: -$OPTARG" >&2
      ;;
  esac
done
shift $((OPTIND -1))

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
ROOT_DIR="$SCRIPT_DIR/.."
pushd $ROOT_DIR

echo "=================================================="
echo "Building for platform: $PLATFORM ($DOCKER_PLATFORM)"
echo "=================================================="
echo ""

for module in modules/*/module.json; do
    module_dir=$(dirname "$module")
    module_image=$(cat "$module_dir/module.json" | jq -r '.image.repository')
    module_version=$(cat "$module_dir/module.json" | jq -r '.image.tag.version')
    module_dockerfile=$(cat "$module_dir/module.json" | jq -r ".image.tag.platforms.\"$PLATFORM\"")
    module_file="$module_dir/$module_dockerfile"

    echo "----------------------------------------------------------------------------------------------------"
    echo "Building module: $module_image from $module_file"
    echo "----------------------------------------------------------------------------------------------------"
    echo ""

    if [[ ! -f $module_file ]]; then
        echo "No Dockerfile found: $module_file"
        continue
    fi

    docker build -f "$module_file" -t "$module_image" --platform "$DOCKER_PLATFORM" --load .

    for container_tag in "${TAG[@]}"; do
        module_tag="$module_image:$module_version-$container_tag-$PLATFORM"
        echo "Tagging image: $module_tag"
        docker tag "$module_image" "$module_tag"
    done

    echo ""
    echo ""
done


echo "=================================================="
docker image ls

popd

After that, we can push modules, generate manifest, and deploy using the standard AzureIoTEdge@2 pipeline task

[xjsxujingsong]

Brian Robles (@ko16a46) Thank you very much.