Deterministic builds work! (for the most part) #317

Manually merged
jwinterm merged 4 commits from :master into master 4 years ago
qvqc commented 4 years ago

This gives us reproducable binaries like monero and bitcoin! This means those with access to a ubuntu 18.04 machine with 2+ cores, 8gb ram, 25gb+ of space can compile wownero with a few commands and recieve the exact same fingerprint binaries as everyone else. This will help clear up confusion about where and how binaries were produced and whether they contain an up to date backdoor.

Sample Output:

f6056e45d57471ad1f9b1b47dbd4932763d65fe33ab17596f5036d2d059194c4  wownero-aarch64-linux-gnu-45142e0ec.tar.bz2
a1071d1ceed7d614d4e7551c31e8f3a669c033b176467bde1541bdce24d38b22  wownero-arm-linux-gnueabihf-45142e0ec.tar.bz2
635953df82140d1da8169e1c61fd1d0533cbd02a5a6f42168eaf7ea1cda91e4c  wownero-x86_64-linux-gnu-45142e0ec.tar.bz2
961c23f4a28c1de37abe9d86ab31a055e80c41ecfc33ab50899d18df94853d10  wownero-x86_64-unknown-freebsd-45142e0ec.tar.bz2
2bb557abac58aac33b791dac3e732b794fc6bc307094869837d57a2c342690c2  wownero-x86_64-w64-mingw32-45142e0ec.zip
0d244540aa2395b409d71fe2198b9e612444e951222671e66aea4bdb73d05603  wownero-x86_64-apple-darwin11-45142e0ec.tar.bz2

The deterministic scripts are from monero-project/monero/contrib/gitian fork with only some small wowification. dropped a few arch that were causing issues(i686-linux-gnu, i686-w64-mingw32, arm-linux-android, aarch64-linux-android). The fail logs are attached here will make an issue as well.

It does have a chicken and an egg issue where a tag is needed that contains up to date gitian descriptor files(/contrib/gitian). So in order for this to work the wownero/wownero:master will need a new tag containing this pr.

edit: docs say that the tag that you build from must contain the gitian discriptor files. misread that as the build needs a tag to be built. tested and successfully built with a commit# only.

The docs are here(am using the docker method):
https://git.wownero.com/wownero/wownero/src/branch/master/contrib/gitian/README.md

havn't figured out the signing bit yet, but will take a look soon.

One script to build them all!

$ ./gitian-build.py --os lfwm --detach-sign --jobs 2 --memory 7000 --no-commit --docker --build qvqc deterministic-testing
deterministic-testing
HEAD is now at 45142e0ec depends: fix broken links for ds_store / mac_alias

Checking Depends Freshness

--2020-08-30 18:24:55--  https://downloads.sourceforge.net/project/osslsigncode/osslsigncode/osslsigncode-1.7.1.tar.gz
Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 216.105.38.13
Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|216.105.38.13|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://deac-fra.dl.sourceforge.net/project/osslsigncode/osslsigncode/osslsigncode-1.7.1.tar.gz [following]
--2020-08-30 18:24:57--  https://deac-fra.dl.sourceforge.net/project/osslsigncode/osslsigncode/osslsigncode-1.7.1.tar.gz
Resolving deac-fra.dl.sourceforge.net (deac-fra.dl.sourceforge.net)... 37.203.33.33
Connecting to deac-fra.dl.sourceforge.net (deac-fra.dl.sourceforge.net)|37.203.33.33|:443... connected.
HTTP request sent, awaiting response... 304 Not Modified
File ‘inputs/osslsigncode-1.7.1.tar.gz’ not modified on server. Omitting download.

--2020-08-30 18:24:59--  https://bitcoincore.org/cfields/osslsigncode-Backports-to-1.7.1.patch
Resolving bitcoincore.org (bitcoincore.org)... 198.251.83.116
Connecting to bitcoincore.org (bitcoincore.org)|198.251.83.116|:443... connected.
HTTP request sent, awaiting response... 304 Not Modified
File ‘inputs/osslsigncode-Backports-to-1.7.1.patch’ not modified on server. Omitting download.

make: Entering directory '/home/qvqc/builder/inputs/wownero/contrib/depends'
make[1]: Entering directory '/home/qvqc/builder/inputs/wownero/contrib/depends'
make[1]: Leaving directory '/home/qvqc/builder/inputs/wownero/contrib/depends'
make[1]: Entering directory '/home/qvqc/builder/inputs/wownero/contrib/depends'
make[1]: Leaving directory '/home/qvqc/builder/inputs/wownero/contrib/depends'
make[1]: Entering directory '/home/qvqc/builder/inputs/wownero/contrib/depends'
make[1]: Leaving directory '/home/qvqc/builder/inputs/wownero/contrib/depends'
make: Leaving directory '/home/qvqc/builder/inputs/wownero/contrib/depends'

Building Dependencies


Compiling deterministic-testing Linux
Submodule path 'external/RandomWOW': checked out '89b7c02bba9100f5ed60056b1e7a82b742af56ce'
Submodule path 'external/miniupnp': checked out '4c700e09526a7d546394e85628c57e9490feefa0'
Submodule path 'external/rapidjson': checked out '3729501db975b74d125f62f18d7bc7b236325f99'
Submodule path 'external/unbound': checked out '0f6c0579d66b65f86066e30e7876105ba2775ef4'
--- Building for bionic amd64 ---
Stopping target if it is up
Making a new image copy
Starting target
Checking if target is up.
Preparing build environment
Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Upgrading system, may take a while (log in var/install.log)
Creating package manifest
Creating build script (var/build-script)
Running build script (log in var/build.log)
Grabbing results from target
Grabbing cache from target
Generating report
f6056e45d57471ad1f9b1b47dbd4932763d65fe33ab17596f5036d2d059194c4  wownero-aarch64-linux-gnu-45142e0ec.tar.bz2
a1071d1ceed7d614d4e7551c31e8f3a669c033b176467bde1541bdce24d38b22  wownero-arm-linux-gnueabihf-45142e0ec.tar.bz2
635953df82140d1da8169e1c61fd1d0533cbd02a5a6f42168eaf7ea1cda91e4c  wownero-x86_64-linux-gnu-45142e0ec.tar.bz2
724db231959ef0262a6ba02b99b5a434d7dae39e719043784be33480a1118f2c  wownero-linux-0.8-res.yml
Done.
Moving var/install.log to var/install-linux.log
Moving var/build.log to var/build-linux.log

Compiling deterministic-testing FreeBSD
Submodule path 'external/RandomWOW': checked out '89b7c02bba9100f5ed60056b1e7a82b742af56ce'
Submodule path 'external/miniupnp': checked out '4c700e09526a7d546394e85628c57e9490feefa0'
Submodule path 'external/rapidjson': checked out '3729501db975b74d125f62f18d7bc7b236325f99'
Submodule path 'external/unbound': checked out '0f6c0579d66b65f86066e30e7876105ba2775ef4'
--- Building for bionic amd64 ---
Stopping target if it is up
Making a new image copy
Starting target
Checking if target is up.
Preparing build environment
Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Upgrading system, may take a while (log in var/install.log)
Creating package manifest
Creating build script (var/build-script)
Running build script (log in var/build.log)
Grabbing results from target
Grabbing cache from target
Generating report
961c23f4a28c1de37abe9d86ab31a055e80c41ecfc33ab50899d18df94853d10  wownero-x86_64-unknown-freebsd-45142e0ec.tar.bz2
4c3887b8f749b148669b2cf3e4aa41bfe457d62e9d399bf7e315d36e48271f5a  wownero-freebsd-0.8-res.yml
Done.
Moving var/install.log to var/install-freebsd.log
Moving var/build.log to var/build-freebsd.log

Compiling deterministic-testing Windows
Submodule path 'external/RandomWOW': checked out '89b7c02bba9100f5ed60056b1e7a82b742af56ce'
Submodule path 'external/miniupnp': checked out '4c700e09526a7d546394e85628c57e9490feefa0'
Submodule path 'external/rapidjson': checked out '3729501db975b74d125f62f18d7bc7b236325f99'
Submodule path 'external/unbound': checked out '0f6c0579d66b65f86066e30e7876105ba2775ef4'
--- Building for bionic amd64 ---
Stopping target if it is up
Making a new image copy
Starting target
Checking if target is up.
Preparing build environment
Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Set alternatives (log in var/install.log)
Upgrading system, may take a while (log in var/install.log)
Creating package manifest
Creating build script (var/build-script)
Running build script (log in var/build.log)
Grabbing results from target
Grabbing cache from target
Generating report
2bb557abac58aac33b791dac3e732b794fc6bc307094869837d57a2c342690c2  wownero-x86_64-w64-mingw32-45142e0ec.zip
c9d1ec9b2723ca0c2c63553a5bc0f84a6a6d7c55a70feadfb6dda566059db682  wownero-win-0.8-res.yml
Done.
Moving var/install.log to var/install-win.log
Moving var/build.log to var/build-win.log

Compiling deterministic-testing MacOS
Submodule path 'external/RandomWOW': checked out '89b7c02bba9100f5ed60056b1e7a82b742af56ce'
Submodule path 'external/miniupnp': checked out '4c700e09526a7d546394e85628c57e9490feefa0'
Submodule path 'external/rapidjson': checked out '3729501db975b74d125f62f18d7bc7b236325f99'
Submodule path 'external/unbound': checked out '0f6c0579d66b65f86066e30e7876105ba2775ef4'
--- Building for bionic amd64 ---
Stopping target if it is up
Making a new image copy
Starting target
Checking if target is up.
Preparing build environment
Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Upgrading system, may take a while (log in var/install.log)
Creating package manifest
Creating build script (var/build-script)
Running build script (log in var/build.log)
Grabbing results from target
Grabbing cache from target
Generating report
0d244540aa2395b409d71fe2198b9e612444e951222671e66aea4bdb73d05603  wownero-x86_64-apple-darwin11-45142e0ec.tar.bz2
05a35fb1e1e6047738cb9abae35097209839aaba5be5434690054cbf5bcce591  wownero-osx-0.8-res.yml
Done.
Moving var/install.log to var/install-osx.log
Moving var/build.log to var/build-osx.log
This gives us reproducable binaries like monero and bitcoin! This means those with access to a ubuntu 18.04 machine with 2+ cores, 8gb ram, 25gb+ of space can compile wownero with a few commands and recieve the exact same fingerprint binaries as everyone else. This will help clear up confusion about where and how binaries were produced and whether they contain an up to date backdoor. Sample Output: ``` f6056e45d57471ad1f9b1b47dbd4932763d65fe33ab17596f5036d2d059194c4 wownero-aarch64-linux-gnu-45142e0ec.tar.bz2 a1071d1ceed7d614d4e7551c31e8f3a669c033b176467bde1541bdce24d38b22 wownero-arm-linux-gnueabihf-45142e0ec.tar.bz2 635953df82140d1da8169e1c61fd1d0533cbd02a5a6f42168eaf7ea1cda91e4c wownero-x86_64-linux-gnu-45142e0ec.tar.bz2 961c23f4a28c1de37abe9d86ab31a055e80c41ecfc33ab50899d18df94853d10 wownero-x86_64-unknown-freebsd-45142e0ec.tar.bz2 2bb557abac58aac33b791dac3e732b794fc6bc307094869837d57a2c342690c2 wownero-x86_64-w64-mingw32-45142e0ec.zip 0d244540aa2395b409d71fe2198b9e612444e951222671e66aea4bdb73d05603 wownero-x86_64-apple-darwin11-45142e0ec.tar.bz2 ``` The deterministic scripts are from monero-project/monero/contrib/gitian fork with only some small wowification. dropped a few arch that were causing issues(i686-linux-gnu, i686-w64-mingw32, arm-linux-android, aarch64-linux-android). The fail logs are attached here will make an issue as well. ~~It does have a chicken and an egg issue where a tag is needed that contains up to date gitian descriptor files(/contrib/gitian). So in order for this to work the wownero/wownero:master will need a new tag containing this pr.~~ **edit:** docs say that the tag that you build from must contain the gitian discriptor files. misread that as the build needs a tag to be built. tested and successfully built with a commit# only. The docs are here(am using the docker method): https://git.wownero.com/wownero/wownero/src/branch/master/contrib/gitian/README.md havn't figured out the signing bit yet, but will take a look soon. One script to build them all! ``` $ ./gitian-build.py --os lfwm --detach-sign --jobs 2 --memory 7000 --no-commit --docker --build qvqc deterministic-testing deterministic-testing HEAD is now at 45142e0ec depends: fix broken links for ds_store / mac_alias Checking Depends Freshness --2020-08-30 18:24:55-- https://downloads.sourceforge.net/project/osslsigncode/osslsigncode/osslsigncode-1.7.1.tar.gz Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 216.105.38.13 Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|216.105.38.13|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://deac-fra.dl.sourceforge.net/project/osslsigncode/osslsigncode/osslsigncode-1.7.1.tar.gz [following] --2020-08-30 18:24:57-- https://deac-fra.dl.sourceforge.net/project/osslsigncode/osslsigncode/osslsigncode-1.7.1.tar.gz Resolving deac-fra.dl.sourceforge.net (deac-fra.dl.sourceforge.net)... 37.203.33.33 Connecting to deac-fra.dl.sourceforge.net (deac-fra.dl.sourceforge.net)|37.203.33.33|:443... connected. HTTP request sent, awaiting response... 304 Not Modified File ‘inputs/osslsigncode-1.7.1.tar.gz’ not modified on server. Omitting download. --2020-08-30 18:24:59-- https://bitcoincore.org/cfields/osslsigncode-Backports-to-1.7.1.patch Resolving bitcoincore.org (bitcoincore.org)... 198.251.83.116 Connecting to bitcoincore.org (bitcoincore.org)|198.251.83.116|:443... connected. HTTP request sent, awaiting response... 304 Not Modified File ‘inputs/osslsigncode-Backports-to-1.7.1.patch’ not modified on server. Omitting download. make: Entering directory '/home/qvqc/builder/inputs/wownero/contrib/depends' make[1]: Entering directory '/home/qvqc/builder/inputs/wownero/contrib/depends' make[1]: Leaving directory '/home/qvqc/builder/inputs/wownero/contrib/depends' make[1]: Entering directory '/home/qvqc/builder/inputs/wownero/contrib/depends' make[1]: Leaving directory '/home/qvqc/builder/inputs/wownero/contrib/depends' make[1]: Entering directory '/home/qvqc/builder/inputs/wownero/contrib/depends' make[1]: Leaving directory '/home/qvqc/builder/inputs/wownero/contrib/depends' make: Leaving directory '/home/qvqc/builder/inputs/wownero/contrib/depends' Building Dependencies Compiling deterministic-testing Linux Submodule path 'external/RandomWOW': checked out '89b7c02bba9100f5ed60056b1e7a82b742af56ce' Submodule path 'external/miniupnp': checked out '4c700e09526a7d546394e85628c57e9490feefa0' Submodule path 'external/rapidjson': checked out '3729501db975b74d125f62f18d7bc7b236325f99' Submodule path 'external/unbound': checked out '0f6c0579d66b65f86066e30e7876105ba2775ef4' --- Building for bionic amd64 --- Stopping target if it is up Making a new image copy Starting target Checking if target is up. Preparing build environment Updating apt-get repository (log in var/install.log) Installing additional packages (log in var/install.log) Upgrading system, may take a while (log in var/install.log) Creating package manifest Creating build script (var/build-script) Running build script (log in var/build.log) Grabbing results from target Grabbing cache from target Generating report f6056e45d57471ad1f9b1b47dbd4932763d65fe33ab17596f5036d2d059194c4 wownero-aarch64-linux-gnu-45142e0ec.tar.bz2 a1071d1ceed7d614d4e7551c31e8f3a669c033b176467bde1541bdce24d38b22 wownero-arm-linux-gnueabihf-45142e0ec.tar.bz2 635953df82140d1da8169e1c61fd1d0533cbd02a5a6f42168eaf7ea1cda91e4c wownero-x86_64-linux-gnu-45142e0ec.tar.bz2 724db231959ef0262a6ba02b99b5a434d7dae39e719043784be33480a1118f2c wownero-linux-0.8-res.yml Done. Moving var/install.log to var/install-linux.log Moving var/build.log to var/build-linux.log Compiling deterministic-testing FreeBSD Submodule path 'external/RandomWOW': checked out '89b7c02bba9100f5ed60056b1e7a82b742af56ce' Submodule path 'external/miniupnp': checked out '4c700e09526a7d546394e85628c57e9490feefa0' Submodule path 'external/rapidjson': checked out '3729501db975b74d125f62f18d7bc7b236325f99' Submodule path 'external/unbound': checked out '0f6c0579d66b65f86066e30e7876105ba2775ef4' --- Building for bionic amd64 --- Stopping target if it is up Making a new image copy Starting target Checking if target is up. Preparing build environment Updating apt-get repository (log in var/install.log) Installing additional packages (log in var/install.log) Upgrading system, may take a while (log in var/install.log) Creating package manifest Creating build script (var/build-script) Running build script (log in var/build.log) Grabbing results from target Grabbing cache from target Generating report 961c23f4a28c1de37abe9d86ab31a055e80c41ecfc33ab50899d18df94853d10 wownero-x86_64-unknown-freebsd-45142e0ec.tar.bz2 4c3887b8f749b148669b2cf3e4aa41bfe457d62e9d399bf7e315d36e48271f5a wownero-freebsd-0.8-res.yml Done. Moving var/install.log to var/install-freebsd.log Moving var/build.log to var/build-freebsd.log Compiling deterministic-testing Windows Submodule path 'external/RandomWOW': checked out '89b7c02bba9100f5ed60056b1e7a82b742af56ce' Submodule path 'external/miniupnp': checked out '4c700e09526a7d546394e85628c57e9490feefa0' Submodule path 'external/rapidjson': checked out '3729501db975b74d125f62f18d7bc7b236325f99' Submodule path 'external/unbound': checked out '0f6c0579d66b65f86066e30e7876105ba2775ef4' --- Building for bionic amd64 --- Stopping target if it is up Making a new image copy Starting target Checking if target is up. Preparing build environment Updating apt-get repository (log in var/install.log) Installing additional packages (log in var/install.log) Set alternatives (log in var/install.log) Upgrading system, may take a while (log in var/install.log) Creating package manifest Creating build script (var/build-script) Running build script (log in var/build.log) Grabbing results from target Grabbing cache from target Generating report 2bb557abac58aac33b791dac3e732b794fc6bc307094869837d57a2c342690c2 wownero-x86_64-w64-mingw32-45142e0ec.zip c9d1ec9b2723ca0c2c63553a5bc0f84a6a6d7c55a70feadfb6dda566059db682 wownero-win-0.8-res.yml Done. Moving var/install.log to var/install-win.log Moving var/build.log to var/build-win.log Compiling deterministic-testing MacOS Submodule path 'external/RandomWOW': checked out '89b7c02bba9100f5ed60056b1e7a82b742af56ce' Submodule path 'external/miniupnp': checked out '4c700e09526a7d546394e85628c57e9490feefa0' Submodule path 'external/rapidjson': checked out '3729501db975b74d125f62f18d7bc7b236325f99' Submodule path 'external/unbound': checked out '0f6c0579d66b65f86066e30e7876105ba2775ef4' --- Building for bionic amd64 --- Stopping target if it is up Making a new image copy Starting target Checking if target is up. Preparing build environment Updating apt-get repository (log in var/install.log) Installing additional packages (log in var/install.log) Upgrading system, may take a while (log in var/install.log) Creating package manifest Creating build script (var/build-script) Running build script (log in var/build.log) Grabbing results from target Grabbing cache from target Generating report 0d244540aa2395b409d71fe2198b9e612444e951222671e66aea4bdb73d05603 wownero-x86_64-apple-darwin11-45142e0ec.tar.bz2 05a35fb1e1e6047738cb9abae35097209839aaba5be5434690054cbf5bcce591 wownero-osx-0.8-res.yml Done. Moving var/install.log to var/install-osx.log Moving var/build.log to var/build-osx.log ```
qvqc added 4 commits 4 years ago
dsc commented 4 years ago
Owner

Holy shit! 👍

Holy shit! 👍
jwinterm merged commit 74a9ab0b03 into master manually 4 years ago
continuous-integration/drone/pr Build is passing
The pull request has been manually merged as 74a9ab0b03.
Sign in to join this conversation.
No reviewers
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

No dependencies set.

Reference: wownero/wownero#317
Loading…
There is no content yet.