dkuerten
Posts: 14
Joined: Tue Jan 14, 2025 1:56 pm

TeamCity Agent getting stuck building Installer

I am using a TeamCity Server to build my installer but the Agent always gets stuck without any log messages.

Both the server and agent of TeamCity are running inside a Docker Container. The build completes without issues until the Installer build Step is reached. Then the agent just runs indefinitely, never completing the task and not providing any console output.
I also tested this on a local instance running directly on my System and there it runs to completion with the same configuration.

I am using the newest TeamCity Version and Advanced Installer version 23.3
Catalin
Posts: 7794
Joined: Wed Jun 13, 2018 7:49 am

Re: TeamCity Agent getting stuck building Installer

Hello,

Without a log file, I'm afraid I can't say for sure what is going wrong in your specific case.

Could you pleaase try writing the Advanced Installer build output to a txt file and see if there is anything of use there?

Code: Select all

AdvancedInstaller.com /build YourProject.aip > "ThePathWhereLogWillBeCreated\buildlog.txt"
Best regards,
Catalin
Catalin Gheorghe - Advanced Installer Team
Follow us: Twitter - Facebook - YouTube
dkuerten
Posts: 14
Joined: Tue Jan 14, 2025 1:56 pm

Re: TeamCity Agent getting stuck building Installer

I am not sure how to redirect the output as there is no option in the plugin. Also the whole agent just gets stuck and I cannot even stop the build. I need to forcefully restart the whole agent because nothing else works.

Her is the configuration I am using for the Build:
Screenshot 2026-07-14 134128.png
Screenshot 2026-07-14 134128.png (69.36 KiB) Viewed 16981 times
This is the console of the most recent build on the locally running server using the same configration and project:

Code: Select all

...
[13:05:10] : Step 4/5: Installer (Advanced Installer) (58s)
[13:05:13] :	 [Step 4/5] advancedinstaller.aip.path=<-app-name-placeholder->\Installer\Installer.aip
[13:05:13] :	 [Step 4/5] advancedinstaller.aip.build=DefaultBuild
[13:05:13] :	 [Step 4/5] advancedinstaller.aip.setup.file=null
[13:05:13] :	 [Step 4/5] advancedinstaller.aip.setup.folder=Publish
[13:05:13] :	 [Step 4/5] advancedinstaller.aip.extra.commands=null
[13:05:13] :	 [Step 4/5] advancedinstaller.aip.path=<-app-name-placeholder->\Installer\Installer.aip
[13:05:13] :	 [Step 4/5] advancedinstaller.aip.build=DefaultBuild
[13:05:13] :	 [Step 4/5] advancedinstaller.aip.setup.file=null
[13:05:13] :	 [Step 4/5] advancedinstaller.aip.setup.folder=Publish
[13:05:13] :	 [Step 4/5] advancedinstaller.aip.extra.commands=null
[13:05:13] :	 [Step 4/5] Starting: C:\BuildAgent\tools\advancedinstaller-23.3\.unpacked\bin\x86\AdvancedInstaller.com /execute C:\BuildAgent\work\1c74a930859aff1b\<-app-name-placeholder->\Installer\Installer.aip C:\Users\<-user-placeholder->\AppData\Local\Temp\aic15768200129973722018.tmp
[13:05:13] :	 [Step 4/5] in directory: C:\BuildAgent\work\1c74a930859aff1b
[13:05:13] :	 [Step 4/5] Notification: Starting synchronization of "APPDIR" with "C:\BuildAgent\work\1c74a930859aff1b\<-app-name-placeholder->\bin\Publish\net8.0-windows10.0.17763.0"
[13:05:13] :	 [Step 4/5] Notification: File added:"APPDIR\BitFaster.Caching.dll"
[13:05:13] :	 [Step 4/5] Notification: File added:"APPDIR\Interfaces.OPC.dll"
[13:05:13] :	 [Step 4/5] Notification: File added:"APPDIR\Interfaces.OPC.pdb"
[13:05:13] :	 [Step 4/5] Notification: File added:"APPDIR\libHarfBuzzSharp.pdb"
[13:05:13] :	 [Step 4/5] Notification: File added:"APPDIR\libSkiaSharp.pdb"
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Diagnostics.PerformanceCounter.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\Microsoft.Bcl.Cryptography.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Text.Encodings.Web.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\OxyPlot.SkiaSharp.Wpf.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\Microsoft.Win32.Registry.AccessControl.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Security.Cryptography.Pkcs.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\OpenTK.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Security.Cryptography.ProtectedData.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\Microsoft.Win32.SystemEvents.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Drawing.Common.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.CodeDom.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Configuration.ConfigurationManager.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Diagnostics.EventLog.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Diagnostics.EventLog.Messages.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.DirectoryServices.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Formats.Asn1.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.IO.Packaging.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.IO.Pipelines.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Private.Windows.Core.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Security.Cryptography.Xml.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Text.Json.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Security.Permissions.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Text.Encoding.CodePages.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Threading.AccessControl.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Windows.Extensions.dll
[13:05:13] :	 [Step 4/5] Notification: File deleted:APPDIR\System.Linq.AsyncEnumerable.dll
[13:05:13] :	 [Step 4/5] Notification: Empty Folder deleted:APPDIR\cs
[13:05:13] :	 [Step 4/5] Notification: Empty Folder deleted:APPDIR\ko
[13:05:13] :	 [Step 4/5] Notification: Empty Folder deleted:APPDIR\ja
[13:05:13] :	 [Step 4/5] Notification: Empty Folder deleted:APPDIR\pl
[13:05:13] :	 [Step 4/5] Notification: Empty Folder deleted:APPDIR\zh-Hant
[13:05:13] :	 [Step 4/5] Notification: Empty Folder deleted:APPDIR\pt-BR
[13:05:13] :	 [Step 4/5] Notification: Ending synchronization of "APPDIR"
[13:05:13] :	 [Step 4/5] Notification: Value of property "ProductVersion" was updated to 0.4.1.4  from file: "APPDIR\<-app-name-placeholder->.exe"
[13:05:13] :	 [Step 4/5] Notification: "ProductCode" property  changed as a result of "ProductVersion" property change.
[13:05:13] :	 [Step 4/5] SetOutputLocation -buildname "DefaultBuild" -path "C:\BuildAgent\work\1c74a930859aff1b\Publish"
[13:05:13] :	 [Step 4/5] Build -buildslist "DefaultBuild"
[13:05:13] :	 [Step 4/5] Notification: Synchronization of "APPDIR" with "C:\BuildAgent\work\1c74a930859aff1b\<-app-name-placeholder->\<-app-name-placeholder->\bin\Publish\net8.0-windows10.0.17763.0" did not produce any changes to project 
[13:05:13] :	 [Step 4/5] Notification: Value of property "ProductVersion" remained unchanged after update from file: "APPDIR\<-app-name-placeholder->.exe"
[13:05:13] :	 [Step 4/5]  
[13:05:13] :	 [Step 4/5] Checking builds status 
[13:05:13] :	 [Step 4/5] Build required. 
[13:05:13] :	 [Step 4/5]  
[13:05:13] :	 [Step 4/5] 
[13:05:13] :	 [Step 4/5] [ DefaultBuild ] 
[13:05:13] :	 [Step 4/5] Creating MSI database (en) 
[13:05:13] :	 [Step 4/5] Writing Summary Information 
[13:05:14] :	 [Step 4/5] Creating MSI database (de) 
[13:05:14] :	 [Step 4/5] Writing Summary Information 
[13:05:14] :	 [Step 4/5] Building package: C:\BuildAgent\work\1c74a930859aff1b\Publish\<-app-name-placeholder->_0.4.1.4.exe 
[13:05:14] :	 [Step 4/5] Prepare build 
[13:05:14] :	 [Step 4/5] Preparing files 
[13:06:07] :	 [Step 4/5] Creating CAB file(s) 
[13:06:07] :	 [Step 4/5] Preparing binaries 
[13:06:07] :	 [Step 4/5] Creating transforms for additional languages 
[13:06:07] :	 [Step 4/5] Creating MSI database 
[13:06:07] :	 [Step 4/5] Writing Summary Information 
[13:06:08] :	 [Step 4/5] Creating EXE file 
[13:06:08] :	 [Step 4/5]  
[13:06:08] :	 [Step 4/5] Total build time: 55 sec.
[13:06:08] :	 [Step 4/5]  
[13:06:08] :	 [Step 4/5] Build finished successfully.
[13:06:08] :	 [Step 4/5] Process exited with code 0
[13:06:09] : Step 5/5: Deploy (PowerShell) (1s)
...
Catalin
Posts: 7794
Joined: Wed Jun 13, 2018 7:49 am

Re: TeamCity Agent getting stuck building Installer

Hello,

Thank you very much for your followup on this and for sharing the screenshots with me.

I might be wrong here, but I do not see any commands passed to the Advanced Installer in your screenshot.

For example, if you want to build the project, you should be passing the /build command in the "Commands" field.

Building from command line

Could you please try that and let me know if it helps?

Best regards,
Catalin
Catalin Gheorghe - Advanced Installer Team
Follow us: Twitter - Facebook - YouTube
dkuerten
Posts: 14
Joined: Tue Jan 14, 2025 1:56 pm

Re: TeamCity Agent getting stuck building Installer

I am using this official guide to setup the build step. There is no mention of explicitly adding a /build argument. As I mentioned before, this exact configuration worked without issues on my local instance.

Based on the protocol from the local instance I posted the plugin seems to call /execute to start the build.
Catalin
Posts: 7794
Joined: Wed Jun 13, 2018 7:49 am

Re: TeamCity Agent getting stuck building Installer

Hello,

I apologize for the quite delayed reply on this one - I had to take some days off due to catching a cold and also I faced some problems when trying to configure the exact environment on my end (mostly skill issue, I admit it).

Please allow me a bit more time to test this tomorrow (hopefully I will get the environment working by then) and I will followup as soon as I will have more details.

Thank you very much for your understanding on this and I am really sorry this takes quite some time, I know this might be a blocker for you.

Best regards,
Catalin
Catalin Gheorghe - Advanced Installer Team
Follow us: Twitter - Facebook - YouTube
dkuerten
Posts: 14
Joined: Tue Jan 14, 2025 1:56 pm

Re: TeamCity Agent getting stuck building Installer

Thank you for the reply,

Here is the compose file I used to setup my containers. Please note, that you have to replace the <Server-IP-Address> with the address of the server. In my case using the address of the host machine worked.

Code: Select all

services:
  teamcity-server:
    image: jetbrains/teamcity-server:latest
    user: 'NT Authority\System'
    ports:
      - 8112:8111
      - 1433:1433
    volumes:
      - C:/TeamCity/teamcity-server/data:C:/ProgramData/JetBrains/TeamCity:rw
      - C:/TeamCity/teamcity-server/logs:C:/TeamCity/logs:rw
    mem_limit: 3g
  agent:
    # Since TeamCity only supports LTS versions of .NET, we need to build a container with its own version.
    # The checksum must be generated from the .zip file (not the .exe). This can be done in PowerShell using the command "(Get-FileHash .\dotnet-sdk-9.0.306-win-x64.exe -Algorithm SHA512).Hash"
    # For more information, see https://github.com/JetBrains/teamcity-docker-images/blob/master/custom/README.md
    build:
      context: ./Teamcity-Agent
      dockerfile: Dockerfile
      args:
        - teamCityAgentImage=jetbrains/teamcity-agent:latest-windowsservercore
        - dotnetSdkVersion=10.0.301
        - dotnetSdkChecksum=38456e992c4df0ff0ac9fc5f28ff09a88543c0fc4e4deedffda9c4ebaf852c4519addacf28814ea77ea42ce2d37db812fae5ba1fe25f06364ca5a6027036387f
        - dotnetSdkVersion2=9.0.314
        - dotnetSdkChecksum2=060896e42db258e7f609cfd7cf074eaf822d6cf0fec80a52cb278178390e5ce23b0b32e042c6ea6cddfff9c375ab7d36c4102006e500b65bb0378c208048ce7d
    volumes:
      - C:/TeamCity/teamcity-agent/conf:C:/BuildAgent/conf:rw
      - C:/TeamCity/teamcity-agent/system:C:/BuildAgent/system:rw
      - C:/TeamCity/teamcity-agent/work:C:/BuildAgent/work:rw
      - C:/TeamCity/teamcity-agent/output:C:/BuildAgent/output:rw
    environment:
      - SERVER_URL=http://<Server-IP-Address>:8112
      - TEAMCITY_SERVER_MEM_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=350m
And here is the Dockerfile I use to build my custom agent container, which is based on this template. I modified it to install two .Net versions because some of our projects use different ones. The file is located in a subfolder called "Teamcity-Agent" relative to the compose file.

Code: Select all

ARG dotnetSdkChecksum
ARG dotnetSdkVersion2
ARG dotnetSdkChecksum2

# PowerShell
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

# Modify .NET & other paths
USER ContainerAdministrator
RUN setx /M PATH ('{0};{1}\bin;C:\Program Files\Git\cmd;C:\Program Files\Mercurial' -f $env:PATH, $env:JAVA_HOME)

RUN [Net.ServicePointManager]::SecurityProtocol = 'tls12, tls11, tls' ; \
    $code = Get-Content -Path "scripts/Web.cs" -Raw ; \
    Add-Type -IgnoreWarnings -TypeDefinition "$code" -Language CSharp ; \
    $url = 'https://builds.dotnet.microsoft.com/dotnet/Sdk/' + $Env:dotnetSdkVersion + '/dotnet-sdk-' + $Env:dotnetSdkVersion + '-win-x64.zip'; \
    $downloadScript = [Scripts.Web]::DownloadFiles($url + '#SHA512#' + $Env:dotnetSdkChecksum, 'dotnet.zip') ; \
    # Remove to successfully expand archive
    Remove-Item -Force -Recurse $Env:ProgramFiles\dotnet; \
    # 2. Extract .NET version
    Expand-Archive dotnet.zip -Force -DestinationPath $Env:ProgramFiles\dotnet; \
    Remove-Item -Force dotnet.zip; \
    Get-ChildItem -Path $Env:ProgramFiles\dotnet -Include *.lzma -File -Recurse | foreach { $_.Delete()};

RUN [Net.ServicePointManager]::SecurityProtocol = 'tls12, tls11, tls' ; \
    $code = Get-Content -Path "scripts/Web.cs" -Raw ; \
    Add-Type -IgnoreWarnings -TypeDefinition "$code" -Language CSharp ; \
    $url = 'https://builds.dotnet.microsoft.com/dotnet/Sdk/' + $Env:dotnetSdkVersion2 + '/dotnet-sdk-' + $Env:dotnetSdkVersion2 + '-win-x64.zip'; \
    $downloadScript = [Scripts.Web]::DownloadFiles($url + '#SHA512#' + $Env:dotnetSdkChecksum2, 'dotnet2.zip') ; \
    # Remove to successfully expand archive
    #Remove-Item -Force -Recurse $Env:ProgramFiles\dotnet; \
    # 2. Extract .NET version
    Expand-Archive dotnet2.zip -Force -DestinationPath $Env:ProgramFiles\dotnet; \
    Remove-Item -Force dotnet2.zip; \
    Get-ChildItem -Path $Env:ProgramFiles\dotnet -Include *.lzma -File -Recurse | foreach { $_.Delete()};


# Switch back to regular user
USER ContainerUser
I hope this helps you reproducing my scenario.
Catalin
Posts: 7794
Joined: Wed Jun 13, 2018 7:49 am

Re: TeamCity Agent getting stuck building Installer

Hello,

First of all, thank you so much for sharing the dockerfile with me.

After some more tedious work, I managed to get it working now.

The build is working just fine on my end, though.

Here is the build log:
Screenshot_150.png
Screenshot_150.png (81.12 KiB) Viewed 14146 times

Here is how I have the build step configured:
Screenshot_151.png
Screenshot_151.png (116.88 KiB) Viewed 14146 times

And here is the built MSI after the build took place:
Screenshot_152.png
Screenshot_152.png (25.25 KiB) Viewed 14146 times

The only difference I can think of would be the fact that I removed, for testing purposes, the "USER ContainerUser" from the dockerfile, leaving the user as admin.

With that in mind, I think that most likely this is an issue with something on your end, which unfortunately I can not tell what.

If no build log appears at all, it makes me think that the process simply hangs before it even gets the chance to start Advanced Installer.

Best regards,
Catalin
Catalin Gheorghe - Advanced Installer Team
Follow us: Twitter - Facebook - YouTube
dkuerten
Posts: 14
Joined: Tue Jan 14, 2025 1:56 pm

Re: TeamCity Agent getting stuck building Installer

Unfortunately using a different user in the Dockerfile did not made a difference.

Based on your results I created a fresh installer project and added it to a new TeamCity Build configuration. The configuration only contains the build step for the installer. I tried even switching from my custom build agent container to the default one (jetbrains/teamcity-agent:latest-windowsservercore). The result was that the build produces the same result and hangs indefinetly, requireing me to forcefully restart the whole container.

When initially selected the wrong agent contanier I also discovered that at least something happens. With the wrong container it produced the following output:

Code: Select all

Step 1/1: Installer (Advanced Installer)
13:30:42   Failed to deploy Advanced Installer tool to agent ip_<agent-ip>-2. Error: Cannot run program "msiexec": CreateProcess error=2, The system cannot find the file specified
13:30:42   Step Installer (Advanced Installer) failed with unexpected error
13:30:51 Advanced Installer tool cleanup
This at least shows that the build step starts.

Could you provide your docker configuration so I can compare it to my own?
Catalin
Posts: 7794
Joined: Wed Jun 13, 2018 7:49 am

Re: TeamCity Agent getting stuck building Installer

Hello,

I am really sorry to hear that this still does not work. :(
Could you provide your docker configuration so I can compare it to my own?
Sure thing, here is the docker compose file:

Code: Select all

services:
  teamcity-server:
    build:
      context: ./Teamcity-Server
      dockerfile: dockerfile
    image: teamcity-server-advinst:latest
    user: 'NT Authority\System'
    ports:
      - 8112:8111
      - 1433:1433
    volumes:
      - C:/TeamCity/teamcity-server/data:C:/ProgramData/JetBrains/TeamCity:rw
      - C:/TeamCity/teamcity-server/logs:C:/TeamCity/logs:rw
    mem_limit: 3g

  agent:
    build:
      context: ./Teamcity-Agent
      dockerfile: Dockerfile
      args:
        - teamCityAgentImage=jetbrains/teamcity-agent:latest-windowsservercore
        - dotnetSdkVersion=10.0.301
        - dotnetSdkChecksum=38456e992c4df0ff0ac9fc5f28ff09a88543c0fc4e4deedffda9c4ebaf852c4519addacf28814ea77ea42ce2d37db812fae5ba1fe25f06364ca5a6027036387f
        - dotnetSdkVersion2=9.0.314
        - dotnetSdkChecksum2=060896e42db258e7f609cfd7cf074eaf822d6cf0fec80a52cb278178390e5ce23b0b32e042c6ea6cddfff9c375ab7d36c4102006e500b65bb0378c208048ce7d
    volumes:
      - C:/TeamCity/teamcity-agent/projects:C:/BuildAgent/projects:rw
      - C:/TeamCity/teamcity-agent/conf:C:/BuildAgent/conf:rw
      - C:/TeamCity/teamcity-agent/system:C:/BuildAgent/system:rw
      - C:/TeamCity/teamcity-agent/work:C:/BuildAgent/work:rw
      - C:/TeamCity/teamcity-agent/output:C:/BuildAgent/output:rw
    environment:
      - SERVER_URL=http://11.0.1.157:8112
      - TEAMCITY_SERVER_MEM_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=350m

and here is the dockerfile:
FROM jetbrains/teamcity-server:latest

USER ContainerAdministrator

SHELL ["pwsh", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

COPY advinst.msi C:/temp/advinst.msi

RUN ["pwsh", "-Command", "$p = Start-Process -FilePath 'msiexec.exe' -ArgumentList '/i','C:\\temp\\advinst.msi','/qn','/norestart' -Wait -PassThru; Write-Host \"MSIEXEC EXIT CODE: $($p.ExitCode)\""]

RUN Remove-Item C:/temp/advinst.msi -Force

Would there be anything else that you require? If so, please let me know and I will gladly provide it.

Best regards,
Catalin
Catalin Gheorghe - Advanced Installer Team
Follow us: Twitter - Facebook - YouTube
dkuerten
Posts: 14
Joined: Tue Jan 14, 2025 1:56 pm

Re: TeamCity Agent getting stuck building Installer

After a lot of testing around and asking in the TeamCity forum I found some new clues.

I tested running AdvancedInstaller without TeamCity. I did a fresh installation on the container and even tested it with a blank container without TeamCity (mcr.microsoft.com/windows/servercore:ltsc2022) and found out that even creating a project through the command line interface produces the same result as observed before. There is no further console output and I have to cancel the process manually.

After that I tested the same on the host. I installed the same AdvancedInstaller version, launched a console and used the same command. It worked without issues.

Confused by that I installed docker on a completley different windows PC, launched a new container and installed a fresh version of AdvancedInstaller again. And there it also blocked the process when trying to create a new project.

To summarize: I was able to reproduce this issue on two separate computers, with completley fresh docker containers, without TeamCity, and in both cases the process just hangs without further output.

One Idea I had in the meantime: Could this be a license issue? I know that you can test an installer inside a docker container and need an Enterprise license to do so, but does this also extends to building an installer?
Catalin
Posts: 7794
Joined: Wed Jun 13, 2018 7:49 am

Re: TeamCity Agent getting stuck building Installer

Hello,

Thank you for your followup on this.
I tested running AdvancedInstaller without TeamCity. I did a fresh installation on the container and even tested it with a blank container without TeamCity (mcr.microsoft.com/windows/servercore:ltsc2022) and found out that even creating a project through the command line interface produces the same result as observed before. There is no further console output and I have to cancel the process manually.
I think this is a big discovery, although it's really strange.

At least, I think that it will allow us to do some more testing.

First of all, did you manually run the /register command? Can you confirm it worked? For example, if you then open Advanced Installer GUI and check the "File" --> "Help" --> "About", does the program appear to be licensed?
Screenshot_168.png
Screenshot_168.png (8.78 KiB) Viewed 9609 times

If we confirm this works, we can then try to check Event Viewer to see if we can find something in there.

Or we can try and use Process Monitor (from sysinternals) and see if we can also find anything in there.

I am thinking we can also try to use ProcDump in that container.

First, we need to get the PID of the process. We can do so either via task manager or via PowerShell:

Code: Select all

Get-Process AdvancedInstaller* | Select-Object Id, ProcessName
and then run procdump, like this:

Code: Select all

procdump -ma <PID> C:\temp\hang.dmp
After generating the dump file, we can open with using WinDbg and running:

Code: Select all

~*kb
k, kb, kc, kd, kp, kP, kv (Display Stack Backtrace)

This should list every thread's call stack. We can look specifically at what the stuck thread is blocked inside.

For example:

- WaitForSingleObject/GetMessage near a window-related module points to a hidden dialog

- a wait inside ws2_32/wininet/winhttp points to a networking related problem

The last suggestion is honestly maybe a bit overkill, as I've only used it once, and it didn't give me much, but at this point I'm thinking of trying everything that crosess my mind, as I really want to manage to assist you here to get the entire setup up and running.

Best regards,
Catalin
Catalin Gheorghe - Advanced Installer Team
Follow us: Twitter - Facebook - YouTube
dkuerten
Posts: 14
Joined: Tue Jan 14, 2025 1:56 pm

Re: TeamCity Agent getting stuck building Installer

First things first, I checked the license and it is set correctly (at least on the host). I also tried a build without defining a license. During that I discovered that I can dump the current threads of the teamcity agent program via the server UI and did some digging in there. I discovered that it hangs the exact moment it tries to execute any advanced installer command using the "AdvancedInstaller.com" tool. This leads back to the root cause of not being able to use the tool inside a container (which works fine on the host).

So next I used ProcDump in different secnarios and it always resultet in this:

Code: Select all

.  0  Id: 2c20.2148 Suspend: 0 Teb: 006bb000 Unfrozen
 # ChildEBP RetAddr      Args to Child              
00 008ff5e0 743c2873     00000002 00e7c524 00000000 ntdll!NtWaitForMultipleObjects+0xc
01 008ff774 743c2728     00000002 00e7c524 00000001 KERNELBASE!WaitForMultipleObjectsEx+0x133
02 008ff790 00e730da     00000002 00e7c524 00000001 KERNELBASE!WaitForMultipleObjects+0x18
WARNING: Stack unwind information not available. Following frames may be wrong.
03 008ff84c 00e751df     00000003 00c7f160 00c845d8 AdvancedInstaller+0x30da
04 008ff894 758367f9     006b8000 758367e0 008ff8fc AdvancedInstaller+0x51df
05 008ff8a4 77c2811d     006b8000 94e11578 00000000 kernel32!BaseThreadInitThunk+0x19
06 008ff8fc 77c280eb     ffffffff 77c5767c 00000000 ntdll!__RtlUserThreadStart+0x2b
07 008ff90c 00000000     00e75270 006b8000 00000000 ntdll!_RtlUserThreadStart+0x1b

   1  Id: 2c20.240c Suspend: 0 Teb: 006cb000 Unfrozen
 # ChildEBP RetAddr      Args to Child              
00 050ff360 743e277f     00000114 00000000 00000000 ntdll!NtFsControlFile+0xc
01 050ff3a4 00e728f3     00000114 00000000 d3140406 KERNELBASE!ConnectNamedPipe+0x3f
WARNING: Stack unwind information not available. Following frames may be wrong.
02 050ff7f8 758367f9     00c7fb18 758367e0 050ff860 AdvancedInstaller+0x28f3
03 050ff808 77c2811d     00c7fb18 916115e4 00000000 kernel32!BaseThreadInitThunk+0x19
04 050ff860 77c280eb     ffffffff 77c5767c 00000000 ntdll!__RtlUserThreadStart+0x2b
05 050ff870 00000000     00e727f0 00c7fb18 00000000 ntdll!_RtlUserThreadStart+0x1b
My guess is that it tries to access a file and blocks there. So my first idea was to unmount all volumens, so that the container should have full access. Sadly this did not work.
What I still find confusing that you stated it did work on your setup, which should technically be exactly the same based on how docker works.
Do you perhaps know if the "AdvancedInstaller.com" tool tries to access any specific Files that could be inacessable for some reason?

I will setup a blank container and again manually install AdvancedInstaller to see if this makes any difference, but this has to wait until next week.
Catalin
Posts: 7794
Joined: Wed Jun 13, 2018 7:49 am

Re: TeamCity Agent getting stuck building Installer

Hello,

Just wanted to do a quick followup on this to let you know that I'm still investigating this issue.
I will setup a blank container and again manually install AdvancedInstaller to see if this makes any difference, but this has to wait until next week.
How did this go on your end?

I did the same on my end and I can reproduce the issue, somehow.

I am running the /register command and it is hanging, just like you mentioned.

And when I try to create a new object, I get the following:
Screenshot_169.png
Screenshot_169.png (28.86 KiB) Viewed 3814 times
I have two other customers that are reporting the same isse as in my screenshot above.

Although it's not the same issue as yours, I feel like somehow they might be related.

In your case, however, you mentioned that the registering of the product worked - can you please confirm this?

The issue that I have with the other two customers stems from the fact that both run the process under session 0, under the Local Service account.

In our specific case, we do not run the process under that account. However, if I did understand correctly, all processes in a docker container run in session 0 (without UI).

This made me think that the two issues might be somehow related.
Do you perhaps know if the "AdvancedInstaller.com" tool tries to access any specific Files that could be inacessable for some reason?
The only thing that it does, behind the scenes, is that it creates a pipe to communicate with advinst.exe (our executable).
What I still find confusing that you stated it did work on your setup, which should technically be exactly the same based on how docker works.
Yes, I do find it confusing a bit now as well. It did really work, as I've proven with the screenshots previously (the MSI was built, everything worked just fine).

Best regards,
Catalin
Catalin Gheorghe - Advanced Installer Team
Follow us: Twitter - Facebook - YouTube
dkuerten
Posts: 14
Joined: Tue Jan 14, 2025 1:56 pm

Re: TeamCity Agent getting stuck building Installer

How did this go on your end?
I tested it on a third PC, again with a servercore:ltsc2022 image. Sadly the result was the same as before. All commands executed using AdvancedInstaller.com hang indefinitely. This also includes the /register command.
I have two other customers that are reporting the same isse as in my screenshot above.
I can confirm the exact same output in my container.
In your case, however, you mentioned that the registering of the product worked - can you please confirm this?
I think there is a misconception. I verified it on the host system, where running the agent worked without issues. I did not register the license myself, the TeamCity Agent must have done this automatically when it executes a build using the license key stored inside the build step. Verifying it on the container is not possible when I am unable to execute any commands.
I have two other customers that are reporting the same isse as in my screenshot above.

Although it's not the same issue as yours, I feel like somehow they might be related.
I think it sounds quite plausable that these issues could be related or even share the same root cause. I must admit I am no expert when it comes to Docker, there could be some fine detail we missed that makes the difference.

Return to “Common Problems”