- Closes#787.
- The `blockSizeInBytes` parameter on `EXT4Unpacker`
was misleading — it does not represent an ext4 filesystem
block size.
- This change renames `blockSizeInBytes` to
`capacityInBytes`, updates the doc comment to accurately
describe the parameter and updates the name in the call
sites.
- Closes#774.
- Adds a default journal configuration parameter to the EXT4Unpacker
init function, defaulting to nil (no journal) to preserve source
compatibility.
- Closes#749.
- ExportOperation hardcoded the pushed index descriptor's
mediaType to the OCI image index type. RegistryClient.push
uses that descriptor's mediaType as the HTTP Content-Type
header. When the source index was in
Docker manifest.list.v2+json format (the common case for
images pulled from Docker Hub and other public registries),
the body's embedded mediaType field disagreed with the
header, and OCI registries rejected the index PUT with
HTTP 400 MANIFEST_INVALID.
- Use the source index's mediaType for the pushed descriptor
so the header always matches the body. Per-architecture
child manifests are unaffected because they were already
pushed with their actual mediaType.
- Add a parameterized unit test for ExportOperation.export
covering both Docker manifest.list
To be able to test vminitd/vmexec/linux specific packages on ci it'd be
a heck of a lot easier if `make` just worked. This should be the last
bit needed. The default goal currently compiles just fine after the
linux specific `make deps` is ran. Next in line would be adding decent
unit tests/actually getting ci setup for the linux bits.
I've added an optional progress handler for rootfs unpacking so that
consumers can show progress. The Ubuntu image takes about 8 seconds to
unpack on my machine, and I'm developing an app where it would be useful
to show progress in the UI.
Total size is determined in an optional first pass that scans archive
headers. Bytes written are then reported during unpacking. The optional
first pass adds 15 (Alpine) to 115 (Ubuntu) ms to unpacking duration on
my machine, depending on image size.
"clean up" is a verb, and "cleanup" is a noun. There are several
examples of the incorrect usage of "cleanup" as a verb in this
repository, including one in the public API (`cleanupOrphanedBlobs`).
For this public method, I've added a deprecation notice to the name with
the incorrect usage.
- Closesapple/container#988.
- macOS libarchive is not built with zstd support, so the workaround in
ArchiveReader is to attempt to decompress every archive as zstd. If
decompression fails, we pass the original archive to libarchive. If it
succeeds, we pass the uncompressed archive.
- Adds blob media type recognition for zstd to EXT4Unpacker. Tested zstd
blob unpack using `image pull tonistiigi/hello-world:zstd-docker`.
Co-authored-by: Aditya Ramani <a_ramani@apple.com>
Closely related to https://github.com/apple/container/issues/305 I would
like to override the used SSL TrustRoots via standard env variables.
This here would add this configuration and would give an entrypoint for
an implementation of 305 to provide CLI flags or similar.
This has no tests yet, as this would require setting up something like a
MITM proxy when testing against a registry.
As I am unfamiliar with the codebase, I would be willing to do this, but
would require a first nudge on where to best implement this.
To actually use this, we would need to add the allowed env variables to
the `container system start` command env filter.
Adds `calculateOrphanedBlobsSize()` to calculate the size of orphaned
blobs, will need this to include them under size and reclaimable space
for images in the `container system df` command so it matches up with
what `container image prune` frees up on disk.
- Fixes#417.
Rename `_prune()` to `cleanupOrphanedBlobs()` to clarify what it
actually does, and remove `prune()` method as we'll do all that logic in
container directly.
Fixes#369
Per the OCI Image Spec, manifest descriptor annotations are optional.
Previously, archives without annotations would fail to import with
"Failed to import image".
**Changes:**
- Modified `getImageReferencefromDescriptor` to return digest-based
references (`untagged@sha256:...`) when annotations are missing
- Removed guard that skipped manifests without annotations
- Added test case with `scratch_no_annotations.tar`
**Testing:**
All 167 tests pass, including new test for images without annotations.
This refactors the rootfs command to produce an image and have the
ability to create an ext4 formatted block of the init filesystem
directly from the command.
closes#220
Signed-off-by: crosbymichael <michael_crosby@apple.com>
A ContainerManager is a type that handles more of the required resource
needed to create and run a container.
---------
Signed-off-by: crosbymichael <michael_crosby@apple.com>
In general I believe it's clever to synthesize an index for
single-platform image manifest, but we still need a way to distinguish
it. Add a dedicated annotation is the slightest change I've come up
with, and it's also OCI compliant. With this change come in, we can work
around https://github.com/apple/container/issues/212 and imitate the
behavior of other runtime with `container`.
Note that since `cctl` is meant to be a dedicated tool for inspecting
the Containerization framework itself, I didn't apply the indirection
for it, and it will be as-is with the genuine storage.
Creates a new Unpacker protocol that defines a single method
```
func unpack(_ image: Image, for platform: Platform, at path: URL, progress: ProgressHandler?) async throws -> Mount
```
This change also removes the `unpack(...)` method from the Image type.
Before
```
let mount = try await image.unpack(for: platform, at: path)
```
After
```
let unpacker = EXT4Unpacker(blockSizeInBytes: 2.gib())
let mount = try await unpacker.unpack(image, for: platform, at: path)
```
---------
Signed-off-by: Aditya Ramani <a_ramani@apple.com>
## Issue
When pulling images, the download speed appears to be slower compared to
Docker.
I found that parallel chunk generation was being performed during the
layer download process.
While individual chunks allow parallel download operations, the overall
process remains sequential between chunks.
This results less performance when chunks contain both small and large
layers mixed together.
## Changes
Discontinued chunk-based segmentation to enable more efficient parallel
downloads.
## Results(in my local env)
| image | layers | old | new |
| ---|---|---|---|
| node:latest | 12 | 1m15s | 1m10s |
| ghcr.io/norio-nomura/swift_discord_bot:main | 54 | 2m45s | 2m30s |
<details>
<summary>raw terminal log</summary>
```
❯ rm -rf ~/Library/Application\ Support/com.apple.containerization && time bin/cctl_old images pull docker.io/library/node:latest
image pulled
bin/cctl_old images pull docker.io/library/node:latest 42.37s user 6.60s system 64% cpu 1:15.94 total
❯ rm -rf ~/Library/Application\ Support/com.apple.containerization && time bin/cctl_new images pull docker.io/library/node:latest
image pulled
bin/cctl_new images pull docker.io/library/node:latest 42.27s user 6.72s system 69% cpu 1:10.58 total
❯ rm -rf ~/Library/Application\ Support/com.apple.containerization && time bin/cctl_old images pull docker.io/library/node:latest
image pulled
bin/cctl_old images pull docker.io/library/node:latest 45.65s user 7.36s system 70% cpu 1:15.64 total
❯ rm -rf ~/Library/Application\ Support/com.apple.containerization && time bin/cctl_new images pull docker.io/library/node:latest
image pulled
bin/cctl_new images pull docker.io/library/node:latest 39.76s user 6.32s system 65% cpu 1:10.50 total
❯ rm -rf ~/Library/Application\ Support/com.apple.containerization && time bin/cctl_old images pull docker.io/library/node:latest
image pulled
bin/cctl_old images pull docker.io/library/node:latest 42.47s user 6.75s system 65% cpu 1:14.72 total
❯ rm -rf ~/Library/Application\ Support/com.apple.containerization && time bin/cctl_new images pull docker.io/library/node:latest
image pulled
bin/cctl_new images pull docker.io/library/node:latest 42.28s user 6.65s system 69% cpu 1:09.93 total
❯ rm -rf ~/Library/Application\ Support/com.apple.containerization && time bin/cctl_new images pull ghcr.io/norio-nomura/swift_discord_bot:main
image pulled
bin/cctl_new images pull ghcr.io/norio-nomura/swift_discord_bot:main 103.83s user 18.71s system 81% cpu 2:30.02 total
❯ rm -rf ~/Library/Application\ Support/com.apple.containerization && time bin/cctl_old images pull ghcr.io/norio-nomura/swift_discord_bot:main
image pulled
bin/cctl_old images pull ghcr.io/norio-nomura/swift_discord_bot:main 120.79s user 20.70s system 85% cpu 2:45.26 total
```
</details>
<details>
<summary>patch for download only</summary>
```diff
diff --git a/Sources/cctl/ImageCommand.swift b/Sources/cctl/ImageCommand.swift
index 84c5218..4aa4bb8 100644
--- a/Sources/cctl/ImageCommand.swift
+++ b/Sources/cctl/ImageCommand.swift
@@ -127,6 +127,7 @@ extension Application {
}
print("image pulled")
+ return
let tempDir = FileManager.default.uniqueTemporaryDirectory(create: true)
if let platform {
```
</details>
This change adds documentation to quite a few existing public types that
didn't have a blurb before.
Additionally, this fixes a couple things that I think either didn't make
sense when going to document them:
- Rename ConnectionStream to VsockConnectionStream. This type only
functions for vsock connections.
- Deletes NsLock+Closure. This was not used anywhere.
- Rename ContainerizationOCI/Config.swift to ImageConfig.swift.
Signed-off-by: Danny Canter <danny_canter@apple.com>
Arnav Reddy